Case study: Incrementally migrating a Python monorepo from Bazel to Pants

You have a Python monorepo that is buried in a mountain of Bazel metadata, has numerous ad-hoc scripts, is experiencing slowing CI times, and engineers who are quite frustrated. You must transform this monorepo to allow for agility, simplicity, and organizational success.

Background

Watson Orders is an IBM Silicon Valley based technology development group targeting the development of world-class conversational AI. We’re currently focused on revolutionizing the drive-thru experience at McDonalds, fully automating the ordering experience.

Our code is mostly Python, organized in a single monorepo, and leverages a plethora of off-the-shelf packages and tools. Before migration, our build consisted of Bazel for build-and-test and ad-hoc scripts for formatting and static analysis. Our continuous integration (CI) on pull requests (PR) was on the order of 10-12 minutes. Developers who were pulling the latest changes from the default branch would incur several minutes on their next run of tests due to cold caches. Our repo also housed roughly. 19 thousand lines of Bazel BUILD metadata (yes, you read that right, 19 thousand lines), most of which was a (sometimes incorrect) mirror of each Python file’s import statements.

Choosing where to start?

To improve the developer experience in our organization, we evaluated the cost of improving the Bazel+Python ecosystem in addition to migrating to tools like Pants, Please, and Buck. Exploration and testing suggested that we use Pants due to three key features:

• Dependency inference (Pants' ability to statically parse code for dependencies)
• “Changed” files detection (Pants’ ability to only run what is necessary based on the changed files from the default branch)
• Integration with formatters, linters, and other off-the-shelf tools

Our strategy would be to migrate in order of increasing risk: our ad-hoc solution for formatting and linting, followed by tests, followed by our scripts, and lastly our production Docker images. This approach trades risk for increased technical challenges. If possible, I’d recommend migrating wholesale by project instead of by CI step, but we determined that approach was not a good fit for us.

Milestone 1: formatters and linters

An easy first candidate for migration was our ad-hoc scripts for running formatters and linters on dev boxes and CI, even though they were not a part of the migration from Bazel to Pants.

The end-result is a world where we still used Bazel for build and test, and used Pants for format and lint. Even if this was the end of the journey, we’ve swapped an in-house, uncached, and inconsistent interface for an off-the-shelf, speedy, and intuitive one. The first step was teaching Pants about our monorepo.

Trick#1: Straddling Bazel and Pants: seeding Pants from Bazel So that we didn't have to maintain two sets of possibly differing metadata, we decided to “seed’ Pants metadata from Bazel’s. Leveraging Pants macros and Pants’ ability to parse multiple build file names (BUILD + BUILD.pants), we turned each Bazel py_library into a Pants python_source, each py_test into a python_test. , and so on... The result was one consistent set of (still hand-maintained) metadata for both build systems.

After swapping in-house tools with ./pants fmt lint check, we shaved several minutes off our CI execution time on every PR.

Milestone 2: Migrating Tests

From a high-level view, both Bazel and Pants run tests in roughly a similar fashion, so there’s little concern about overall execution. The differences appear when taking that view lower. Bazel has support for special metadata like cpu or manual, for which we had to find equivalent support for in Pants (or just ignore). Otherwise, Pants’ features, such as dependency inference on conftest.py files or “ephemeral tempdir sandboxing” were net positives. The hand-maintained metadata for our test files got wiped clean thanks to Pants’ dependency inference!

Unfortunately, when using a dependency-aware build system, your tests serve a shadow purpose: validating dependencies. Consider the case of missing metadata: your tests fail due to a missing import (or equivalent). As we shed our Bazel metadata and dove into Pants, we left a gap in testing that could be disastrous for deployed code.

Trick #2: Validating Bazel metadata by leveraging Pants In order to mitigate the risk, we whipped up a CI step that compares the hand-crafted Bazel metadata with Pants’ inferred dependencies. If a dependency is missing, we error and fail CI. This uncovered over one thousand missing dependency declarations. In reality, very few of these were active bugs, since you can import a transitive dependency. They were, however, landmines waiting to be stepped on, possibly inflicting numerous hours of painful head scratching.

Now that migration risk is gone, you can eliminate several thousand lines of Bazel metadata. And while we’re at it, you can reclaim several minutes of additional dev box test time when pulling the latest changes from the default branch. Thanks to Pants’ --changed flag, we only test things that are likely affected by changes in the development branch. We can also easily identify flaky tests, by separating the “due-to-changes" test runs from the “everything” test runs (failures in the latter category are suspected to be flaky).

Milestone 3: Migrating pr_binary rules

Leveraging a feature soon-to-be-released in Pants (with support being added as an in-repo plugin until we upgrade), any Python file can be run as a script in Pants. The result was that almost all our 200 py_binary rules had their metadata wiped clean with nothing required for migration. Another couple thousand lines of Bazel metadata gone. This also marked the point at which most developers as well as our PR and CI jobs no longer required Bazel.

Milestone 4: Migrating Docker images

The last and final milestone had the highest risk and highest reward. Migrating our production Docker images bundled with our Python application. The code changes for this task were minimal, with the onus of work being in validating the newly minted images. With each docker image that was migrated we removed large swaths of metadata. As the last image was migrated, this milestone marked the complete removal of Bazel and the remaining 10 thousand lines of Bazel metadata.

Additionally, because we can craft our own Dockerfiles, we shed a shared base image and instead allowed each service to control its instructions, netting us Docker images that are several GB smaller in size.

In conclusion

Overall, moving build systems is like trying to force a fast-moving ship to change course. The faster you spin the wheel the more cargo and crew mates feel the force of the change. Pants provided all the pieces necessary to course-correct, but ultimately the most challenging aspect of migration was ensuring our processes and engineers weren’t thrown about along the way. Frequent communication and support were vital to our migration’s success, both by myself and the wonderful Pants community.

Technically speaking, we went from 20 thousand lines of Bazel metadata to roughly 2.4 thousand lines of Pants metadata (with a goal to reduce it event further). Our CI was reduced from 10-12 minutes to just under 4 minutes. Our developer pre-commit tooling has seen a simplified CLI and faster execution time. The project was all delivered ahead of schedule in a matter of months. And lastly, the author has become a maintainer of Pants himself.