Many of our data pipelines interact with external services. The availability of an external service can adversly affect the health our pipelines. This is how we handle it using AWS Step Functions
Posts about Backend
Hello from trivago’s performance & monitoring team. One important part of our job is to ship more than a terabyte of logs and system metrics per day, from various data sources into elasticsearch, several time series databases and other data sinks. We do so by reading most of the data from multiple Kafka clusters and processing them with nearly 100 Logstashes. Our clusters currently consists of ~30 machines running Debian 7 with bare-metal installations of the aforementioned services.
Testing your functionality is important, but what happens if other factors come into play? In this blog post we show how trivago handles non-functional testing for every commit and how we scaled it.
Would you book a hotel without seeing the images first? No, right? Hence, it’s vital to make sure the images are available all the time. In a scenario where a lot of images were deleted, we must have an efficient way of recovering them. This is how we achieved that with Amazon S3 Versioning.
When migrating your data to new technologies, validation of the data becomes challenging as your data structures might change. Rebase tries to make this easier while also giving your more flexibility on your data.
We built a reactive pipeline to move almost a quarter billion database records to AWS and to build a reactive and serverless pipeline. This is the story of the lessons we learned along the way working with Kinesis and Lambdas
One day, Memcached ran out of free memory. The method
get failed and all requests went directly to the database. Of course these calls also failed under the huge load, and eventually it caused downtime for the whole trivago website. Yikes!
We do think that our tech blog is full of interesting things powered by our engineers’ great stories. Let us take you on a journey of how we maintain trivago tech blog from the technical perspective and how we recently automated its deployment process.
We all have been there, done that. You want to build an API that allows you to manipulate your entities so you start checking which specification to use. Maybe REST or JSON API or maybe no specification. Once you have decided which way to go it’s time to create the controllers. Each controller needs at least 5 actions: one to get a single entity, one to get the collection and 3 others to create, update and delete.
Tackling hard problems is like going on an adventure. Solving a technical challenge feels like finding a hidden treasure. Want to go treasure hunting with us?View all current job openings