End-to-end tests retry strategies

Why should you retry all tests on failure? Why not? This article will not go into details, listing pros and cons of each approach. There are already enough resources on the Web about the topic, listing valid points for both opposing views. As trivago Hotel Search frontend QA team over the last years we tried to stay away from a brute-force retry policy for failures and we rather tried to execute test retries only in selected cases. Recently, when we switched to a Continuous Deployment approach for our new frontend Web application (which empowers developers to merge and release some pull requests autonomously), we faced a greater need than before for understandable and stable test results. Due to that, showing as few “red flags” as possible for the automated checks on pull requests became even more important to ensure enough confidence in test results and to avoid slowing down the software development life cycle. The requirements and the balance between deterministic results and success ratio shifted, at least in some cases.

Read Cucable Maven plugin for parallel execution of Cucumber scenarios

Cucable Maven plugin for parallel execution of Cucumber scenarios

At trivago, we are using an in-house developed Selenium framework based on cucumber-jvm to run automated browser tests. As the test suite increased (the time exceeded 45 minutes for a full run), we were looking for ways to move away from sequential towards parallel execution. For Cucumber, there are actually not that many options available: