Caching Docker builds in GitHub Actions: Which approach is the fastest? 🤔 A research.
Tags: github, actions, ci, research
Published: 2020-04-19 16:24:54

---
title: Caching Docker builds in GitHub Actions: Which approach is the fastest? 🤔 A research.
published: true
description: I tried out 6 different ways to re-use a Docker build cache from previous builds in GitHub Actions to find out which way is the fastest.
tags: github, actions, ci, research
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/q437vfnb05k6ewqsq4q6.png
series: Build performance optimization
---

**Abstract:** In this post, I experimented with 6 different approaches for caching Docker builds in GitHub Actions to speed up the build process and compared the results. After trying out every approach, 10 times each, the results show that using GitHub Packages’ Docker registry as a build cache, as opposed to GitHub Actions’ built-in cache, yields the highest performance gain.

> ⚠️ **Warning:** This article has not been updated since its publication in April 2020. The approaches outlined here are probably out-of-date. Here are more updated takes on this issue:
>
> - **2021-07-29** [Docker's official `build-push-action` now supports GitHub Cache API](https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md#cache-backend-api) where caches are saved to GitHub Actions cache directly, skipping the local filesystem-based cache.
>
> - **2021-03-21** Andy Barnov, Kirill Kuznetsov. [“Build images on GitHub Actions with Docker layer caching”, _Evil Martians_.](https://evilmartians.com/chronicles/build-images-on-github-actions-with-docker-layer-caching)

## Background and motivation 💬

Sick of waiting for builds on GitHub Actions. 😖

Unlike self-hosted runners like Jenkins, most cloud-hosted build runners are _stateless_, providing us with a pristine environment each run. We cannot keep files from the previous runs around; anything that needs to be persisted must be externalized.

[GitHub Actions has a built-in cache](https://help.github.com/en/actions/configuring-and-managing-workflows/caching-dependencies-to-speed-up-workflows) to help do this. But there are many ways of creating that cache (`docker save` and `docker load` first comes to mind). Will the performance gains outweight the overhead caused by saving and loading caches? Are there more approaches other than using GitHub Action’s built-in cache? That’s what this research is about.

## Approaches considered 💡

To conduct this experiment, I first came up with some ideas on how we might cache the built Docker image. I experimented with 8 total approaches.

1. **No cache 🤷‍♂️**

   This is the baseline.

2. **No cache (BuildKit) 🤷‍♀️**

   This is another baseline, but with [BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/).

3. **Use `docker save` and `docker load` with `actions/cache` 📦**

   Once an image is created, we can use [`docker save`](https://docs.docker.com/engine/reference/commandline/save/) to export the image to a tarball and cache it with [`actions/cache`](https://github.com/actions/cache). On subsequent runs, we can use [`docker load`](https://docs.docker.com/engine/reference/commandline/load/) to import the image from the cached tarball. Then we can build the image with the `--cache-from` flag.

   Note that this method is not compatible with BuildKit as [it only supports external caches located on the registry](https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources) (not pulled/imported images).

4. **Cache /var/lib/docker with `actions/cache` 🤯**

   Maybe we don’t need to do all that import-export work and can just cache the whole `/var/lib/docker` directory!

5. **Use a local registry with `actions/cache` 🐳**

   We can [run a filesystem-backed Docker registry](https://docs.docker.com/registry/) on localhost, push the built image to that registry, and cache it with `actions/cache`. On subsequent runs, we can pull the latest image from the localhost registry and use the pulled image with `--cache-from` arg.

6. **Use a local registry with `actions/cache`(BuildKit) 🐋**

   Same as above, but with BuildKit.

   There is a minor difference in the setup: With BuildKit we do not have to run `docker pull`. [BuildKit already expects the cached image to be located on a registry.](https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources) While this may seem like a limitation, but it also comes with a performance gain: if a cache turns out to be not applicable, BuildKit will skip pulling the image.

7. **Use GitHub Packages’ Docker registry as a cache 🐙**

   GitHub provides [a package registry](https://github.com/features/packages) already, so we might not need to run our own filesystem-backed registry at all!

8. **Use GitHub Packages’ Docker registry as a cache (BuildKit)** 🦑

   Same as above, but with BuildKit.

## Failed approaches 💥

- **Approach 🤯, Cache /var/lib/docker with `actions/cache`,** failed because of a lot of permission errors caused when generating the cache. Even with a `sudo chmod` a proper cache could be created.
- **Approach 🦑, Use GitHub Packages as a cache (BuildKit),** failed because as of the time of writing this, GitHub Package Registry does not support requesting a cache manifest. Subsequent builds will cause this message when trying to re-use a cached image: `ERROR: httpReaderSeeker: failed open: could not fetch content descriptor`

So, we are left with 4 approaches (📦, 🐳, 🐋, 🐙), excluding 2 baselines (🤷‍♂️ and 🤷‍♀️).

## Image set up 🏛

When optimizing builds, although we can reduce the time to run `docker build` 🤩, but unfortunately some overhead will be introduced 😞.

Overhead time includes:

- restoring a saved cache (approaches **📦**, **🐳**, **🐋**);
- importing image from the tarball loaded from the cache (approach **📦**);
- running a local Docker registry (approaches **🐳**, **🐋**);
- pulling from a Docker registry (approaches **🐳**, **🐋**, **🐙**);
- exporting the built image to a tarball (approach **📦**);
- pushing an image to a registry (approaches **🐳**, **🐋**, **🐙**); and
- saving a cache (approaches **📦**, **🐳**, **🐋**).

From my experience,

- The **build time** grows mostly with project complexity, i.e. the CPU time to build.
- The **overhead time** grows mostly with the size of things being cached, i.e. the IO time to serialize and transmit stuff.

For this experiment, we want to have an image that’s large enough, such that _both_ the build time and overhead become apparent in our measurements.

This resulted in this Dockerfile which generates a somewhat bloated image 🤪:

    FROM node:12
    RUN yarn create react-app my-react-app
    RUN cd my-react-app && yarn build
    RUN npm install -g @vue/cli && (yes | vue create my-vue-app --default)
    RUN cd my-vue-app && yarn build
    RUN mkdir -p my-tests && cd my-tests && yarn add playwright

## Methodology 🔬

I created a GitHub Actions workflow which runs all the approaches. I ran them 20 times.

- 10 times, each time changing the Dockerfile, invalidating the cache (`actions/cache`). This is the cache MISS scenario.
- 10 times, each time keeping the Dockerfile the same. This is the cache HIT scenario.

After that, the results for each approach and scenario will be averaged together.

There are few exceptions:

- The baselines don't have a cache. So all 20 builds count as cache MISS.
- Approach 🐙 uses an external registry, and hence images are cached and invalidated on a layer-by-layer basis. Due to the way I set up this experiment, all 20 builds ended up being a cache HIT. I am too lazy to perform 10 more builds, so I will assume the cache MISS situation is same as the no cache + the time it takes to push the image the first time.

## Results 📊

After all the builds are run, I used the [GitHub Actions API](https://developer.github.com/v3/actions/) to retrieve the timing for each step. I threw the results into Google Sheets, ran some PivotTable, and summarized the results.

### 🎖 The showdown

![Chart](https://dev-to-uploads.s3.amazonaws.com/i/x8s3k6pur20o6audde1y.png)

The green bar represents the time it takes in cache HIT scenario. The red bar represents the extra time it costs for the cache MISS scenario.

### 🤷‍♂️ The baselines 🤷‍♀️

![Chart](https://dev-to-uploads.s3.amazonaws.com/i/qs1hbhd33a9dkvge06n7.png)

You can see that just using BuildKit without any cache gives some performance gain. One whole minute saved.

### 📦 The tarball

![Chart](https://dev-to-uploads.s3.amazonaws.com/i/k4eo1hsldl70ni031hk9.png)

The cache size is **846 MB**. It turns out that `docker save` is very slow. So, uploading that tarball into the cache also takes a long time.

### 🐳 The local registry

![Chart](https://dev-to-uploads.s3.amazonaws.com/i/w43m94cel0au2ye50j5m.png)

The cache size is **855 MB**. It turns out that doing a `docker push` to a filesystem-backed registry, running in localhost, inside Docker, turned out to be faster than `docker save`. The same is true for `docker pull` vs `docker load`.

### 🐋 The local registry, with BuildKit

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/tsulnctpjq16d9wwmcjb.png)

The cache size is **855 MB**. When you run `docker build`, BuildKit pulls the image from the local registry, so there is no separate `docker pull` step, compared to the approach **🐳**.

### 🐙 The GitHub Package Registry

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/x0zn3rao4rrph20nks0i.png)

Although we cannot take advantage of BuildKit, using an external registry means that builds can be cached on a layer-by-layer basis. Out of all approaches I’ve tried, this approach gives the best result. 🏆

However, you also get a “build-cache” package listed in your repository’s published packages. 😂

## An example closer to real world

To test out the potential gain, I tried running `docker-compose build` on [DEV Community’s repository](https://github.com/thepracticaldev/dev.to). Without any caching, building the web image took [9 minutes and 5 seconds](https://github.com/dtinth/dev.to/runs/599511890). Using GitHub Package Registry as a cache, the time to build the image has reduced to [37 seconds](https://github.com/dtinth/dev.to/runs/599535581).

## Conclusions

You can find the code implementing each of these approaches here:

{% github dtinth/github-actions-docker-layer-caching-poc no-readme %}

When it comes to build performance improvement, there are many articles, each recommending different approaches using different examples. It’s hard to see how each approach fares against one another.

While working on this, I also noticed trying other different approaches together gives way to more novel approaches being discovered. For example, had I not tried out Approach 🐳 (local Docker registry), I wouldn’t have thought “Oh! GitHub Package Registry is a thing! Can I use that and skip Action’s cache entirely? Let’s try!” and came up with Approach 🐙.

So, I’d say this is the kind of article I wish existed when I wanted to set up a CI workflow with Docker. I wish we have more showdowns for more approaches and more tools.

Thanks for reading!