Building Software at Google Scale Tech Talk

Google software engineers explain the design and operation of Google's internal, language-agnostic, and highly parallelized build system.

Google Tech Talk March 21, 2012 Presented by Michael Barnathan, Software Engineer, Google Greg Estren, Software Engineer, Google Pepper Lebeck-Jobe, Software Engineer, Google ABSTRACT At past Google NYC Tech Talks, we learned about tools that helped Google engineers automate quality testing, so that Google products could be released frequently without extensive manual testing phases or manual invocations of tools like JUnit, WebDriver, or JavaScript Test Driver. This talk covers the Google Build System, which Google engineers use to build software from a unified, language-agnostic, continuously integrated code base, quickly and at scale. When a developer initiates a build, the build system automatically computes the minimal number of artifacts that need to be built and determines the optimal strategy for producing them as fast as possible using the resources of many worker machines. On average, each build request triggers thousands of source file compilations, while still completing within seconds. At Google, all software components are compiled from source, in a highly parallelized fashion, possibly across thousands of machines dedicated to software compilation. Build artifacts that compose software components are also shared across build requests, such that if a developer builds a component and another developer builds a similar component, the artifacts in common between them are not built twice. This talk will discuss in detail how all this "magic" works. More than just a time-saver for our developers, the high performance and unprecedented scale of our build system make possible far-reaching quality goals, such as continuously testing Google's entire codebase at every revision. This talk will be given by three engineers on the build system team.