Reproducible and Contained CI Builds on Mac OS without Virtualization

Recently I’ve been researching the state of Continuous Integration for Mac OS and specifically for iOS. I wanted to learn how people are doing CI on Mac OS and what are the common best practices. Unfortunately, there was no pleasant surprise for me, the state of build automation for Mac OS is pretty bad. In this blog post I will do a brain dump of my findings and will describe an experimental solution that I came up with.

What do we need?

First things first, we need to clarify what are we looking for our builds. A good CI setup usually aims to have following guarantees:

1. Flexible and Reproducible Environment. It should be possible to specify an environment where a build is executed. Different types of builds can require different versions of tools and frameworks, and CI should guarantee builds of a certain type will be always executed in the same environment.
2. Isolation of Builds. Sequential or parallel builds can’t interfere execution of each other. Environment should be cleaned from artifacts of the previous build for every new build.
3. Performance. There should be minimum implication to performance of CI builds comparing to builds on local machines of developers.

What do we have?

A common CI system uses a pattern where machines that execute builds A.K.A. workers have a program called CI agent installed that constantly pulls a master for work.

Diagram of CI workers pulling CI master for work

Back 20+ years ago, agents were simply running a shell script and it was a developer’s work to make sure these scripts are well written and provide guarantees described above. But it’s simply not possible to make and most importantly maintain such guarantees with scripts.

Since then, not many things have changed unfortunately. The most notable change happened in this architecture is a shift to use Virtual Machines as CI workers to have that Flexible and Reproducible Environment. Other than that, it just became easier to setup a CI system or even use a fully hosted solution.

Here is an excellent talk from Uber engineers about their journey from one physical Mac Mini to thousands of Virtual Machines (starts at 13:20, Medium doesn’t support time tags). This talk basically shows the whole evolution of things described above.

History of Mobile CI at Uber starts at 13:20

Too long to watch version: Uber engineers put tremendous amount of work to create a CI infrastructure of 1000+ Mac OS Virtual Machines running via vSphere on MacStadium. They’ve also had a lot of problems with scaling vSphere cluster to run 1000s of VMs. Many components like network storage and network itself struggle with the load. Just watch the talk for more details, it’s worth to watch!

Most popular hosted CI solutions like Travis CI and Circle CI are also using a similar to Uber’s setup of a vSphere cluster running on MacStadium.

This approach has also a few downfalls as well:

1. Performance. Running a VM brings a notable performance implication to CI builds. It’s the price of Reproducible Environment and Isolation.
2. Still hard to update an environment. In order to update an application installed in a VM, one need to build a new VM, push it to a Network Storage, test it, reconfigure CI once everything works.

But everything else looks perfect. It’s possible to create any build environment with a VM, no need to think about isolation of VMs, performance degradation is tolerable in most cases. Seems we have all the guarantees we wanted from a CI setup.

Can we do better?

If think about it, there is no actual need for having Virtual Machines. A CI build is just a computation within an environment of pre-installed software, there is no need for full virtualization of an underlying operation system. And I found an evidence that it can work. A recent blog post from Pinterest engineers describes their experience of using Nix package manager to have a reproducible and flexible build environment.

Continuous integration for iOS with Nix and Buildkite

Nix takes a functional approach to package management. Within Nix all packages are treated as immutable values with great isolation to prevent side effects. Please read Pinterest’s blog post and Nix documentation for more details.

Basically, Nix gives us exactly what we want: Flexible and Reproducible Environment. There is no virtualization in this case which gives us Performance guarantee. The only thing left is to guarantee Isolation, how can we securely run scripts?

At this point, I recalled that Bazel build system has a sandboxing mechanism that works on Mac OS. After some digging into Bazel’s sources, I found out it’s using standard Mac OS mechanism conveniently called Sandboxing. Sandboxing allows to create security profiles that can restrict a program to access certain parts of the file system, execution of programs, sending signals to other processes, etc.

Below is a basic security profile for running CI builds in isolation. It has detailed comments for every rule. Basically, it restricts a program from writing to most of the file system, blocks sending signals to processes outside of it’s process group, etc.

At this point I tried to combine reproducible environment from Nix and isolation guarantees of sandboxing. I called it Chamber.

Chamber Diagram

Here is a drop-in chamber.sh script that can be used like ./chamber.sh ./script/ci.sh to run an existing CI script in a chamber.

The script has been tested on several projects but it’s still WIP. I highly encourage to try it out and report any issues. Please read it though, it contains detailed comments on what it does and how to debug/configure things.

Conclusion and Remaining Followups

Running VMs is still hard. If not Apple, then maybe companies like Veertu with their modern Anka virtualization, will bring Docker-like feel to automation for Mac OS. Until then, people will look for workaround and hacks.

Chamber concept described in this post is just an interesting idea that seems to work on a few small and medium size projects. But it’s very experimental. There are still a few remaining questions:

1. Test on variety of projects.
2. Make xcodebuild to work within nix-shell . I wasn’t able to make it work so nix-shell is used to compute PATH for running a script in a sandbox.
3. It’s unclear how to support multiple Xcode versions. Should it be a Nix package? Should it be a separate tool?

I hope it was an interesting read. Please don’t hesitate to ping me on Twitter with any questions. I hope something useful can come out of my research and Chamber experiment.