Backend at trivago

Insights, experiences and learnings from trivago's tech teams.

Read Introducing Protector - a Circuit Breaker for Time Series Databases

Introducing Protector - a Circuit Breaker for Time Series Databases

At trivago we store a subset of our realtime metric data in InfluxDB and we are quite impressed by the load it can handle. Despite all the joy, we had to learn some lessons the hard way. It is pretty easy to overload the database or the web browser by executing queries that return too many datapoints. To prevent that, we wrote Protector - a circuit breaker for Time series databases that blocks malicious queries.

Read the whole article ›
Read Better Log Parsing with Logstash and Google Protocol Buffers

Better Log Parsing with Logstash and Google Protocol Buffers

At trivago we rely heavily on the ELK stack for our log processing. We stream our webserver access logs, error logs, performance benchmarks and all kind of diagnostic data into Kafka and process it from there into Elasticsearch using Logstash. Our preferred encoding within this pipeline is Google's Protocol Buffers, short protobuf. In this blog post, we will explain with an example how to read protobuf encoded messages from Kafka using Logstash.

Read the whole article ›
Read Elasticsearch and Kibana for Selenium Automation

Elasticsearch and Kibana for Selenium Automation

The advances and growth of our Selenium based automated testing infrastructure generated an unexpected number of test results to evaluate. We had to rethink our reporting systems. Combining the power of Selenium with Kibana's graphing and filtering features totally changed our way of working. Now we have real-time testing feedback and the ability of filtering between thousands of tests, all in one Dashboard.

Read the whole article ›
Read tCache - Scalable data-aware Java Caching

tCache - Scalable data-aware Java Caching

Caching data is an essential part in many high-load scenarios. A local 1st-level cache can augment a shared 2nd-level cache like Redis and Memcached to further boost performance. An in-process cache involves no network overhead, so the cache speed is only limited by local resources like CPU, memory transfer speed and locking. tCache is a production-proof local in-process cache for the JVM, which is part of trivago's OpenSource Java library triava (Apache v2 license). This article outlines cutting-edge features like data-aware evictions that are operating near lock-free.

Read the whole article ›
Read Introducing Gollum: A NxM message multiplexer written in Go

Introducing Gollum: A NxM message multiplexer written in Go

Here at trivago we write a huge number of log messages every day that need to be stored and monitored. To handle all these messages we created Gollum, a tool that enables us to conveniently send messages from multiple sources to different services. While initially only covering log messages Gollum quickly evolved to a routing framework for all kinds of data. This blogpost is a short introduction to Gollum and how we use it at trivago.

Read the whole article ›