Building Multi-Arch Docker Images on Github Actions
Tags: tutorial, docker, githubactions, devops
Published: 2023-10-02 17:13:22

I recently needed to create multi-arch images so that I could spin up containers on my arm-based laptop. Github Actions only runs on x64-86 VMs, which can make building images for other architectures tricky.

### Regular old Docker Build
```yaml
name: Docker
on:
  push:
    branches: ['main']
jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v3
        with:
          push: true
          tags: user/app:latest
```

With the above Github Actions file, we could build and push to Docker Hub easily. Now let's check out how we can use `buildx` to build for multiple architectures.

### QEMU and Buildx

> QEMU is a generic and open source machine emulator and virtualizer. When used as a machine emulator, QEMU can run OSes and programs made for one machine (e.g. an ARM board) on a different machine (e.g. your own PC). By using dynamic translation, it achieves very good performance.

> Docker buildx is a Docker plugin that extends the building ability of images by using the BuildKit builder. It allows us to build images for different platforms and architectures.

With a combination of **QEMU** and **buildx**,  which are conveniently also available as Github Actions, we can easily get our images built on emulated OSes and push onto the docker registry with multi-platform support!

Add the following after the registry checkout on our Github Actions workflow file:
```yaml
- name: Set up QEMU
  uses: docker/setup-qemu-action@v2
  with:
    platforms: 'arm64,arm'
- name: Set up Docker buildx
  uses: docker/setup-buildx-action@v2
```
Note: The QEMU setup step has to be put *before* the buildx step!

By default, `setup-qemu-action` will create many VMs, most of which you might not care about. Make sure to put in the platforms you actually want to build for in the `platforms:` field to limit this to only ones you actually need.

### Arch-specific Dockerfile instructions

The `docker-build-push-action` will provide the `--platform` flag to `docker buildx`. This will set the following [build ARGs](https://docs.docker.com/build/guide/build-args/) automatically:
 
- `TARGETPLATFORM` - platform of the build result. Eg `linux/amd64`, `linux/arm/v7`, `windows/amd64`.
- `TARGETOS` - OS component of `TARGETPLATFORM`
- `TARGETARCH` - architecture component of `TARGETPLATFORM`
- `TARGETVARIANT` - variant component of `TARGETPLATFORM`
- `BUILDPLATFORM` - platform of the node performing the build.
- `BUILDOS` - OS component of `BUILDPLATFORM`
- `BUILDARCH` - architecture component of `BUILDPLATFORM`
- `BUILDVARIANT` - variant component of `BUILDPLATFORM`

To use these ARGs in your Dockerfiles, you need to add them like the following example grabbed from Docker docs:
```dockerfile
FROM alpine
ARG TARGETPLATFORM
RUN echo "I'm building for $TARGETPLATFORM"
```

### End result
As a result, we can build and push multi-arch images by emulating other architectures on Github's x86_64 VMs using `QEMU` and building with `docker buildx`. We can even do arch-specific actions by using the ARGs automatically set by `buildx`.

### Closing notes
Although this is convenient, no land without stones, or meat without bones.

There is one big and one relatively small problem this might cause for you.

#### 1. Super long build times

Your builds might be taking very long after enabling multi-arch builds. This seems to be a known problem with emulating, but you could use some tricks to make less emulated code run while building.

In my case, I was building a Go application inside a build image, then copying that onto a target image in a two-step Dockerfile like so:
```dockerfile
FROM golang:1.20-alpine as builder

WORKDIR /

RUN go env -w GOPRIVATE=github.com/myorganization

ARG GITHUB_TOKEN
RUN echo "machine github.com login github password $GITHUB_TOKEN" >> /root/.netrc

COPY . .
RUN go build "-ldflags=-s -w" -o /my-application cmd/main.go


FROM alpine:3.8

COPY --from=builder /my-application /app/my-app

WORKDIR /app
CMD /app/my-app
```

Instead of running the `go build` inside the Dockerfile, I switched to building directly inside the Github Actions using `GOOS` and `GOARCH` variables to specify building for a specific arch:
```yaml
- name: Set up Go 1.20
  uses: actions/setup-go@v1
  with:
    go-version: 1.20

- name: Setup netrc
  run: |
    echo "machine github.com login github password ${{ secrets.ACCESS_TOKEN }}" > ~/.netrc
- name: Build application
  run: |
    go env -w GOPRIVATE=github.com/myorganization
    GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w" -o build/my-app-amd64 cmd/main.go
    GOOS=linux GOARCH=arm64 go build "-ldflags=-s -w" -o build/my-app-arm64 cmd/main.go
```

And then copying those builds into the container using the buildx ARGs:
```dockerfile
FROM alpine:3.8

ARG TARGETARCH
COPY build/my-app-${TARGETARCH} /app/my-app

WORKDIR /app
CMD /app/my-app
```

In the end, this lowered the whole execution time from ~6m to ~1,5m. I've heard people lower theirs down from ~1h to ~15m as well.

#### Pushing to AWS ECR creates multiple images
If you're pushing to [AWS ECR](https://aws.amazon.com/ecr/), you might realize one push is creating many untagged images of type `Image` and one image of type `Image Index` containing all the tags, which looks like this: 

![image](https://cdn.sanity.io/images/s46c1uqa/blog/fb7074ab18a5b6eff4b4d5a46aa387874bb2e036-1286x1430.png?w=450)

This is actually normal, since pushing a multi-arch image really is just pushing untagged images for each arch and using an Image Index to redirect the user to the correct architecture image when pulling. The arch-specific images are untagged so that they cannot be easily discovered and pulled, and instead the Image Index that handles this is reached.

It looks like AWS might [update the UI](https://github.com/aws/containers-roadmap/issues/1596) to make this look less cluttered in the future.


### References

- https://docs.docker.com/build/building/multi-platform/
- https://blog.thesparktree.com/docker-multi-arch-github-actions
- https://github.com/aws/containers-roadmap/issues/1596