Monitoring at trivago

Insights, experiences and learnings from trivago's tech teams.

Read How we scaled our Prometheus setup

How we scaled our Prometheus setup

In 2020 we started to migrate one of our most significant workloads, our Node.js based GraphQL API and many of its microservices, from our datacenter to Google Kubernetes Engine. We deploy it in three GCP regions, each having its Kubernetes cluster. Since then, our monitoring infrastructure has changed due to various periods of instability and pandemic induced scaling challenges.

Read the whole article ›
Read Better URL Search with Elasticsearch

Better URL Search with Elasticsearch

At trivago, we generate a huge amount of logs and we have our own custom setup for shipping logs using mostly Protocol Buffers. Eventually we end up with some fields in Elasticsearch (ES) that contain partial (or full) URLs. For instance, in our specific case we store the query component of the URL in a field called query and the path component in a field named url_path. Sample values for these fields could be:

Read the whole article ›
Read Nomad - our experiences and best practices

Nomad - our experiences and best practices

Hello from trivago's performance & monitoring team. One important part of our job is to ship more than a terabyte of logs and system metrics per day, from various data sources into elasticsearch, several time series databases and other data sinks. We do so by reading most of the data from multiple Kafka clusters and processing them with nearly 100 Logstashes. Our clusters currently consists of ~30 machines running Debian 7 with bare-metal installations of the aforementioned services. This summer we decided to migrate all of this to an on-premise [Nomad](https://www.nomadproject.io/ cluster) cluster.

Read the whole article ›
Read Splitting a Monitoring Monolith into Separate Components

Splitting a Monitoring Monolith into Separate Components

Ever heard about Microservices? Those tiny litte pieces of code that are used to split a big pile of magic into smaller pieces of magic? Well, they're not that tiny after all and require lots of preliminary work to use them properly. Have a look at this post to hear about my journey of splitting an existing monolith written in PHP up into several microservices written in Go.
Read the whole article ›