Speeding up Docker Builds x7 (benchmark)
Tags: docker, productivity, benchmark, devops
Published: 2019-12-19 13:19:19

Few things feel as unproductive as waiting for a build. If Docker is part of your development workflow, you know how long `docker build` can take to create an image. Can we speed it up? To find the answer, I benchmarked it in Semaphore and Docker Hub with a real use case scenario.

I found that I can build my image at least 7 times faster on Semaphore. Read on to find out how I did it.

## 10 Minutes or Bust

If your [continuous integration](https://semaphoreci.com/continuous-integration) process takes more than 10 minutes you are not enjoying the full benefits of CI/CD. Around the 10 minutes barrier the feedback loop that breaks down, teams merge fewer times a day, increasing the risk of merge conflicts.

{% link https://dev.to/markoa/what-is-proper-continuous-integration-585c %}

<!-- Teams with slow pipelines merge fewer times a day, increasing the danger of conflicts. Besides, waiting is hard so we are tempted to multitask, which hurts productivity on the final account. -->

<!-- Most teams will do their builds as part of the CI/CD workflow; this is the correct way of doing things. -->

[Docker Hub](https://hub.docker.com) includes the convenient autobuild feature, which takes a Dockerfile from your git repo and builds the image right into the registry. Convenience is nice, but we should also take into account how much of the 10 minutes budget it eats up.

## The Benchmark Setup

The question I want to answer is simple: how long does it take from the moment I do a git push up to the moment the image is ready to use?

Here is my benchmark plan:

![Benchmark Setup](https://thepracticaldev.s3.amazonaws.com/i/ny4oac4y4mfubwi3daew.png)

- I set up a GitHub repository with a Dockerfile. This is the source for both build paths.
- I created two repositories on Docker Hub:
  - **docker-build**: I connected it with GitHub and enabled autobuild.
  - **semaphore-build**: I kept autobuild disabled in this one and used [Semaphore CI/CD](https://semaphoreci.com/cicd) to build and push to it instead.
- Each push triggers both builds at the same time.

I chose to build the official [couchbase](https://www.couchbase.com/) Dockerfile. The resulting image is around 500 MB. The Dockerfile is pretty standard, so the benchmark is likely to be representative of many workloads.

The source repository only needs to have a Dockerfile and the related scripts. This is how I set it up:

```bash
$ wget https://raw.githubusercontent.com/couchbase/docker/201af2d1fd4988d23d980cef5b91763ee5fdc9b7/enterprise/couchbase-server/6.0.3/Dockerfile
$ mkdir scripts
$ cd scripts
$ wget https://raw.githubusercontent.com/couchbase/docker/201af2d1fd4988d23d980cef5b91763ee5fdc9b7/enterprise/couchbase-server/6.0.3/scripts/run
$ wget https://raw.githubusercontent.com/couchbase/docker/201af2d1fd4988d23d980cef5b91763ee5fdc9b7/enterprise/couchbase-server/6.0.3/scripts/entrypoint.sh
$ wget https://raw.githubusercontent.com/couchbase/docker/201af2d1fd4988d23d980cef5b91763ee5fdc9b7/enterprise/couchbase-server/6.0.3/scripts/dummy.sh
```

For the course of this post, I’ll call **build time** to the total time from git push to image ready; this includes push/pull times, queue time, and docker build time.

I resorted to [git reflog](https://git-scm.com/docs/git-reflog) to measure the push time:

```bash
$ git reflog --date=local master
```

To measure the image last modification time, I queried the Docker Hub API with:

```bash
$ curl https://hub.docker.com/v2/repositories/$DOCKER_NAME/$REPO_NAME/ | jq .last_updated
```

## Building With Cache

Docker build can reuse layers from previous images to speed up build time.

In Docker Hub, the cache is enabled by default. In Semaphore, we must use `docker pull` and the `cache-from` options.

I used this [pipeline](https://semaphoreci.com/blog/cicd-pipeline) to build with cache:

```yaml
version: v1.0
name: Docker benchmark
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

blocks:
  - name: "Docker build"
    task:
      secrets:
        - name: dockerhub
      jobs:
      - name: Pull, Build & Push
        commands:
          - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
          - checkout
          - docker pull "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest || true
          - docker build --cache-from "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest -t "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest .
          - docker push "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest
```

I ran both builds simultaneously 11 times. I discarded the first run. All the tests were done during US business hours.

Here are the results:

![Build Times With Cache](https://thepracticaldev.s3.amazonaws.com/i/hujkiy0hg3jqzvn4qnx8.png)

Docker Hub consumes **almost half** of the 10-minute budget. On average, Docker Hub jobs spent one or two minutes queued before starting.

![Median Build Time](https://thepracticaldev.s3.amazonaws.com/i/umi28010ospv3thqgz81.png)

Median build times:

- **Docker Hub**: 265s
- **Semaphore**: 35s (x7.5 faster)

On average, the 1,300 monthly minutes included free plan is enough to build 1,900 couchbase images in Semaphore. For the pro plan, 100 builds should cost less than 50 cents and take about an hour from your day.

For comparison, a 100 builds would take 7:30 hours in Docker Hub—we can gain back a whole workday for the price of lemonade.

## Without Cache

Sometimes Docker’s cache can be too aggressive. Docker will reuse the cached layers unless the Dockerfile has changed, which isn’t always what we want it to do.

I disabled the cache on Docker Hub. I also modified the Semaphore pipeline to force a full build.

```yaml
version: v1.0
name: Docker benchmark
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

blocks:
  - name: "Docker build"
    task:
      secrets:
        - name: dockerhub
      jobs:
      - name: Pull, Build & Push
        commands:
          - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
          - checkout
          - docker build -t "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest .
          - docker push "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest
```

These are the full build times:

![Runs Without Cache](https://thepracticaldev.s3.amazonaws.com/i/a6ggiafjl39q02hm4c4l.png)

- **Docker hub**: 665s per image – 100 builds take 21 hours.
- **Semaphore**: 194s per image – 100 builds take 5:30 hours.

Docker Hub has taken us well over the 10-minute limit. Semaphore keeps on top and is 3.4 times faster.

![Median Build Time Without Cache](https://thepracticaldev.s3.amazonaws.com/i/ytn18tsmz6hz5aytu5c4.png)

The free plan lets us build at least 380 of these images per month. For those on the pro plan, 100 builds will set you back $2.50. By switching from Docker Hub to Semaphore, we gain back 15 hours for the price of a cup of coffee.

<!-- I found that the sweet spot for this image can be found when I switch to the more the middle-tier e1-standard-4 machine. The cost rises to $3.26 per 100 builds, but the total time spent cuts back to 3:30 hours. These results, however, will vary depending on the Dockerfile. -->

## Building Faster With Semaphore

At Semaphore our motto is “build great products at high velocity.” Each second you gain back by optimizing your CI/CD is multiplied hundreds or thousands of times over the project’s lifecycle.

<!--
The Docker Hub [upgraded plan](https://hub.docker.com/pricing) doesn’t offer faster builds; upgrading only allows concurrent builds and additional private repositories.

The Semaphore [pro plan](https://semaphoreci.com/pricing) also enables concurrent builds and offers more powerful machine types. So, if you feel your Semaphore builds are still taking too long, you can try a faster machine.
-->

Work with Docker a lot? Don’t miss the [Docker CI/CD tutorials](https://semaphoreci.com/product/docker).

Did you find this post useful? Hit those ❤️ and 🦄.Questions? Leave a comment below.

Thanks for reading!