Optimizing startup time of Cirrus Runners

We recently added OpentTelemetry Traces alongside Metrics in order to gain a deeper understanding of areas for potential optimizations.

Previously with only high level metrics we had a limited visibility into how GitHub Actions jobs are running on Cirrus Runners platform. We only measured several executing related metrics: delivery lag of GitHub webhooks, scheduling time of a job within Cirrus Runners Scheduler and how long it takes to actually start a single use virtual machine to execute a job. Here is a schema that represents our previously limited visibility:

Traces allowed us to zoom into the startup process of virtual machines. Let’s look at an example trace of a GitHub Actions job running in ghcr.io/cirruslabs/macos-sonoma-xcode:latest Tart virtual machine:

jobs:
  macos:
    name: "Run on the fastest infrastructure"
    runs-on: ghcr.io/cirruslabs/macos-sonoma-xcode:latest
    steps:
      - uses: actions/checkout@v4

As we can see it takes 1.35 seconds to start a VM and 9.87 seconds for it to boot into an SSHable state. This all aligned with our high level metrics of 10-15 seconds for the VM to start.

The 10-second GitHub webhook delay results in a frustrating 20-30 second combined wait before Cirrus Runner jobs start. We can’t control the GitHub delivery lag but we can optimize startup time – our servers now have ready-to-go, SSH-able virtual machines.

Introducing “always prepared” images

Check out our new ghcr.io/cirruslabs/macos-runner:sonoma image. There is now a pool of virtual machines based on this Tart image that are always prepared to immediately start executing your jobs. By using this image in your runs-on configuration you can eliminate the startup delay caused by virtual machine boot time.

This image is identical to our -xcode:VERSION images, but has 3 recent Xcode versions preinstalled and the xcodes tool for easy switching:

jobs:
  macos:
    name: "Run on the fastest infrastructure"
    runs-on: ghcr.io/cirruslabs/macos-runner:sonoma
    steps:
      - uses: actions/checkout@v4
      - name: List Installed Xcodes
        run: xcodes installed
      - name: Select Xcode 15.2
        run: sudo xcodes select 15.2

Here is a trace of a job configured above:

As you can see we managed to eliminate the prepare-instance part all together and SSHing into a VM now only takes milliseconds and overall startup time is a little over a second.

What’s Next

We are planning to enable the same optimization for our most popular medium resource class of Linux Arm and Linux x86 runners. If you are already using -md Linux runners then expect faster startup times in the coming weeks. We are also experimenting with filesystem optimizations for Linux runners which deserves a separate blog post in the near future.

In the meantime feel free to try the new ghcr.io/cirruslabs/macos-runner:sonoma image for your macOS jobs. If you have any questions or need assistance with configuration, please don't hesitate to reach out to our support team at support@cirruslabs.org.