Case Study: A major logistics and routing company reduces test times by 90%+

Company

In 2020, a major logistics and routing company came to YourBase with some serious continuous integration pains. Their product, built on a Python codebase, packed more than 10,000 tests and took 1-2 hours to build, even for small code changes. Their business requirements were highly sophisticated, including geospatial database interactions that are very expensive and complicated to test.

Product teams and developers were frustrated: They were forced to wait 70-90 minutes on average, due to the nature of their business, they couldn’t afford downtime or major app-breaking bugs.

Over time, the codebase grew along with the costs. More code led to more tests and an increase in compute needs. The team experimented with cost and process improvements, which led to some builds taking up to four hours to complete. They were really in a bind.

When they heard about YourBase’s promise of build time and compute cost reductions, one developer said, “If this works, I will cry.”

Methodology

Before recommending any changes, the YourBase team went with a first-do-no-harm strategy by building a parallel CI environment. The existing CI pipeline remained untouched while developers tested YourBase locally. YourBase CTO and co-founder John Ewart’s primary goals were to create acceleration and provide meaningful feedback without false positives/negatives.

Ewart’s take:

“You can put more cars on the highway by building more lanes, or putting more people on buses. But you save more time and money putting people on buses."

If the existing CI pipeline detected a bug, Ewart made certain YourBase detected it as well. If a test failed in the existing system, Ewart ensured YourBase saw the same failure. Since this implementation focused on intelligent test selection, he made sure failing tests were never skipped.

To prove YourBase’s value, the parallel CI infrastructure ran through the following sequence:

• Catch bugs as they’re introduced
• Continue that sequence for every failure in the existing CI system
• Carry it to two successful subsequent builds
• Accelerate and return to success after the bug is resolved
• Continue on to the next successful build

Broader code changes reflected a wider test selection and smaller changes reflected a narrower selection, just as expected. This work eventually led to an average build time of 20 minutes, with 75% of all builds completing in under nine minutes.

Success

YourBase delivered on its promise. Average build times went down from 90 minutes to 9 minutes, a 90% reduction. No black magic, no special tricks; just intelligent test selection, analysis, and huge cost savings.

You might be wondering whether anyone cried. Tears were indeed shed, but it wasn’t who you’d expect: it was the cloud provider. The cloud provider’s monthly revenue shrank substantially thanks to YourBase. 

Would you also like to reduce your cloud provider bills? Reach out to us at hi@yourbase.io!