Open Source at trivago

Imagine a world without open source software. Pretty scary, isn't it?

There would be no free operating systems that let you take full control of your computer.

For a long time, we've mentioned that we're running our own JavaScript framework: Melody. Today, we are happy to share the framework we've been using since 2016 as an Open Source project with you. We designed Melody to be fast, memory efficient and to be flexible enough for the future.

On the 11th of November 2017 trivago hosted a Hackathon at NASDAQ's MarketSite in New York. The winners were awarded with a trip to Europe to visit trivago's offices as well as conference tickets for The Next Web, Amsterdam. We invited the winners to our office where we got the chance to interview them about their Hackathon experience.

Back in April 2015, I felt the need to do some work and earn money besides my studies in Computer Science at the University of Düsseldorf. After doing some research and crawling a few job platforms, I finally applied for a job in IT-Support at trivago. The job offer looked very appealing and life at trivago promised to be fun.

Hackathons! That’s where you feel super excited about creating something interesting, or useful, or just entertaining. They are a great way to encourage creativity and teamwork. Recently, trivago hosted a Python Hackathon in collaboration with the Python Usergroup Düsseldorf. The event was a full success!

At trivago, we use a Cucumber based framework for end-to-end tests of our most important web applications. Cucumber stores test result as JSON files which can be turned into human-readable test reports.

When I joined trivago a year ago, we had problems with our releases. The traffic was increasing each day. When we put the server back into the load balancer without warming up the OPcache it would die. From time to time the warmup failed silently. Our DCO (data center operations) crew had to log into the servers and restart a few processes manually. During this time every release was very intense.

You do not run a successful, stable software project over several years without some amount of automated testing. If several dozens of developers are working on the same code base, the need for test automation becomes even greater. After all, their changes might have unintended effects on other people's code, or on certain edge cases that will not be noticed until the changes go live — and maybe not even then.

For the past few years, Webpack has played a central and important role at trivago. We use it for handling SVG icons and to improve our startup time for the benefit of our users by loading resources on demand. We run a highly complicated build with plenty of custom plugins which perform all sorts of optimisations for us that no other tool would allow us to do. And because we truly love open source we’ve also open sourced our solution to speed up multi-compiler builds, which we rely on heavily to deliver ideal bundles to our users.

At trivago, we are using an in-house developed Selenium framework based on cucumber-jvm to run automated browser tests. As the test suite increased (the time exceeded 45 minutes for a full run), we were looking for ways to move away from sequential towards parallel execution. For Cucumber, there are actually not that many options available:

Concepts like separation of concerns, logic decoupling or dependency injection are things we developers have heard more than a couple of times. At trivago, the Android app is developed using the Model View ViewModel (MVVM) architecture, aiming for views as dumb as possible, leaving the decision making to the view models. This leads to an increased test coverage since testing logic in views is something we can’t do that easily.

At trivago we store a subset of our realtime metric data in InfluxDB and we are quite impressed by the load it can handle. Despite all the joy, we had to learn some lessons the hard way. It is pretty easy to overload the database or the web browser by executing queries that return too many datapoints. To prevent that, we wrote Protector - a circuit breaker for Time series databases that blocks malicious queries.

At trivago we rely heavily on the ELK stack for our log processing. We stream our webserver access logs, error logs, performance benchmarks and all kind of diagnostic data into Kafka and process it from there into Elasticsearch using Logstash. Our preferred encoding within this pipeline is Google's Protocol Buffers, short protobuf. In this blog post, we will explain with an example how to read protobuf encoded messages from Kafka using Logstash.

When using webpack to build your assets, it's only a matter of time until you wish for targeted builds. Whether it's the output of the library you're working on (CJS, UMD, AMD, Var, etc.), or the specific feature set (IE8 support, no IE8 support). parallel-webpack can run those builds in parallel, thus making full use of the multi-core processing capabilities of modern devices.

tCache takes a creative approach for near lock-free evictions and supports data-aware evictions. Its key features are:

Configuration of features is individual per Cache instance, by using a cache Builder: