YourBase has a number of levels of acceleration, ranging from intelligent caching to process-level optimizations and even all the way down to per-language test optimizations. This post will introduce how the intelligent-caching and process-level optimizations work and a later one will dive into how per-language optimizations work.
At the core of the YourBase build system is a build graph that is responsible for modeling the execution of building and testing software. This build graph tracks what happens during the build, which we will call steps, and what dependencies those steps have such as files, processes or network connections. This is done through a low-overhead system that records the system calls that are being made throughout a given step and then analyzes them to build a graph of processes, file interactions, and other behaviors that it can use later to determine optimal strategies for faster builds.
This build graph allows YourBase to optimize future builds by knowing what inputs have changed over time builds and combining that information with the graph to inform its decision-making process when accelerating current and future builds. As a real-world example, we will take a look at some of the build data generated when we build our own internal Python-based API service using YourBase.
When YourBase performs a build or runs tests, it actively tracks what is happening at the system level. As a result, it is able to discover what is involved in running a step including processes that are created, files that are interacted with, or even network-related operations. That data can be visualized using our interactive build-graph tool; the diagram below represents a portion of our API’s build graph, specifically the process-level information that is stored in the graph.
As you can see, the build graph contains a very detailed view of a build; for scale, the above subset represents only a fraction of all the information stored in our API’s build graph. This rich data set is very powerful in determining how to best optimize future builds; the simplest of which we call intelligent caching.
Legacy CI systems spend a lot of time and resources performing unnecessary operations: repeated downloading of dependencies, installation of packages, compiling libraries that are depended upon, and so on. This results in wasted compute resources as well as taking extra time that prevents developers from shipping features faster as they wait for test results so that they can merge their feature.
To see just how much time this can take, we can examine a graph that shows us how much time each step of our API build takes, without any optimizations.
Our API build, with no optimization, takes just over three minutes to build and run; the majority of which (nearly 75% of the total build) is spent downloading system packages and installing Python dependencies. Being able to avoid those alone would result in a significant drop in build and test time.
Unfortunately, if we just cache the dependencies and never check for new ones we risk stale or missing dependencies in the future. Because YourBase models which files affect each step of the build we don’t need to worry about this; if a change to our source tree is made such that we would need to execute our dependency step then the execution plan will re-run that step automatically for us, ensuring that our dependencies are up to date when needed. Below we can see the time-savings our team benefits from using intelligent caching alone.
The result in our case is a 48 second build, down from just over three minutes, a savings of well over two minutes per build. In the course of a day we may run dozens of builds, so just caching alone results in several hours of wait time being saved; we can spend more time thinking about what we’re working on and less time thinking about when the build will finish so someone can confidently review our pull requests.
Intelligent caching of dependencies is just one way that builds can be accelerated; YourBase leverages the build graph data to know when it can skip a particular process (or sub-process), which means we can do a bit of rearranging of our test suites to improve performance even further. Instead of running our tests in a single process, resulting in an all-or-nothing approach, we are able to break our tests into sub-groups and run them independently. Doing this provides a greater number of opportunities to bypass a set of tests as more focused tests will naturally decrease the number of files that a test suite depends upon.
In many of our internal builds we can’t avoid running every test suite but we are often able to cut down dramatically on the number of test suites that we do run; below is a graph showing a typical build for comparison.
Because we have broken apart the tests in our API service into groups, each process has fewer dependencies than one single monolithic process that ran all of our tests would. Many of our commits result in a change that only affects a handful of our tests, resulting in average build time around 20 seconds, or roughly 10% of our naive build, and a 67% savings over intelligent caching.
To better understand how this is achieved, we can visualize a portion of the build graph that shows how files map to steps. This sub-graph is filtered to show only two test suites and the files that they depend upon. Here, one test suite (our “builds” test suite) depends on thirteen different input files; whereas the other (our “buildlogs” test suite) has a one-to-one dependency.
Having a visual representation of our build graph allows us to see which tests are most likely to run as well as which files are hot-spots and will result in many tests being executed. This helps direct the steps we might take to further optimize our own build process and improve test group isolation.
Because of these visuals, we are able to see that a change to any one of thirteen files in our source tree would require a re-execution of the build API test suite but that the build log API tests depends on only one file.
If we were to make a code change that altered our build log API test suite, the only step that will run is that specific test suite; there is no reason to run the others. Because the build graph understands the relationships between steps and the inputs that affect them YourBase is able to avoid repeating these unnecessary steps. By not executing test suites that are irrelevant to a change we can further reduce the time required to build and test our API.
By only running one test suite (our projects test suite) the total time spent, which once was over three minutes, has now been cut down to around 8 seconds. This fully optimized case is a mere 4% of the original run time, resulting in a substantial savings in time when compared to running all of our test suites and fetching dependencies.
No additional tooling is required to achieve this level of acceleration; because YourBase analyzes your build automatically you do not need to install any new tools, libraries, or other dependencies in order for it to provide immediate value. For many teams building projects using Ruby, Go, Python or NodeJS simply using YourBase will result in significant reductions in build time with no changes to code.
In addition to these mechanisms, YourBase has language-specific harnesses that are able to further expand the granularity of the build graph by analyzing builds at the test and test-group level. By having a deep understanding of the technology and language that is used in your build YourBase is able to learn which specific tests or test-groups are affected by a given change in source code.
By combining process-level and language-specific analysis of builds YourBase is able to pinpoint exactly which tests or steps are required to run and which can be safely bypassed in order to dramatically cut down on build time. With our Ruby-specific acceleration technology we have seen Ruby on Rails projects that have reduced their build times from over one hour and fifteen minutes down to an average of seven minutes (and in some cases as little as three minutes!)
By combining different techniques, YourBase builds and tests code 3 to 10 times faster than legacy CI systems. This means that your team can spend more time focusing on getting work done and less time waiting around for builds, saving as many as three hundred hours a month, or more, in build time!