Build DOCKER multi-platform image using buildx REMOTE builder NODE
Tags: docker, cicd, tutorial, devops
Published: 2023-08-02 21:37:43

Running applications on different operating systems and processor architectures is a common scenario, so it is a common practice to build separate distributions for different platforms. This is not easy to achieve when the platform we use to develop the application is different from the target platform for deployment. For example, developing an application on an x86 architecture and deploying it to a machine on an ARM platform usually requires preparing the ARM platform infrastructure for development and compilation.

## Problem

I recently purchased the new Apple M1 Pro MacBook, which is a high-performance laptop. The M1 Pro (aarch64) processor is very fast and very friendly on the battery, but as I mentioned there are also some - not so small - issues associated with working on a new chip architecture.

I’ve tried to create a few multi-platform docker images with buildx and some are just not correctly emulated on the M1.
The parallelism of buildx was also an issue but the emulation gave me the most trouble.


## Solution
Use buildx remote node(s) to build the image on a native architecture.

## What do you need?

- More than one docker-enabled device in your network or on the internet like:
 - Virtual Machine / Server
 - Raspberry Pi
 - Laptop 
 - Desktop
- ssh access to these devices

In my case I have a CX11 VM on [HETZNER](https://hetzner.cloud/?ref=fpAL7qHPphtl) that is `**amd64/x86_64**` based and my own M1 Pro that is `**arm64/aarch64**` based. Since Hetzner now offers arm64 machines, you can also do it the other way around too.


## How I did do this?

---

## Step 1: Installation

Since the newer versions of Docker, Buildx (Buildkit) is a built-in function, to check its availability, use the following command.

```bash
❯ docker buildx version
github.com/docker/buildx v0.11.1 b4df08551fb12eb2ed17d3d9c3977ca0a903415a
```

## Step 2: Setup ssh access for your remote node

The remote host where we also want to run Docker needs to be configured with a password-less SSH connection. This post will not explain how to do that as there are many articles explaining how to set up public/private key access to a device.

If you already have the keys you can copy them to your target device like so:

```bash
ssh-copy-id USERNAME@TARGET_VM_HOST
```

To run a command with the **user environment available** you need to make sure your sshd service allows it.
When running a command through ssh you will have a limited environment and that needs to be adjusted.

To have a correct environment where docker is known you need to set the `#PermitUserEnvironment no` property to `PermitUserEnvironment yes` in the `/etc/ssh/sshd_conf` file on your VM. 
When this is adjusted you need to restart the sshd service. On my CX11 VM with the Ubuntu server, I used this command. It might be different for your device:

```bash
sudo systemctl restart sshd
```

Now you have to set the environment for ssh:
- login to your VM through the terminal (ssh USER_HERE@TARGET_VM_HOST)
- cd ~/.ssh
create a file called **environment** with the following value in it

```bash
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
```
the last `/usr/local/bin` is where the docker command lives. So now it will be available.

Check if you can run docker with the command below from local machine/laptop. It should work:

```bash
docker -H ssh://USERNAME_HERE@TARGET_VM_HOST info
```
Do not forget to change the command's values to appropriate ones for your situation.

## Step 3: Create a buildx local node

Let’s first create the local platform. In my case that is the Apple M1 Pro node. It should build all the arm64/aarch64 targets.

```bash
docker buildx create \
  --name local_remote_builder \
  --node local_remote_builder \
  --platform linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/mips64le,linux/mips64,linux/arm/v8,linux/arm/v7,linux/arm/v6 \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000
```

I’ve played a bit with the settings but this seems to work best for me. I will update as needed or if I find settings that work even better.

Now we have a local config for a builder called local_remote_builder.

You can check if you have it by running `docker buildx ls` command:

```bash
NAME/NODE 	DRIVER/ENDPOINT 	STATUS PLATFORMS
local_remote_builder * docker-container
	local_remote_builder unix:///var/run/docker.sock running linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386

```
## Step 4: Create a target buildx remote node

In my case, this will be the amd64/x86_64 builder node. The platforms corresponding to that architecture should be sent there.
We have already configured the SSH access and the environment and tested that docker can access it.

Now we need to add it to the buildx target

```bash
docker buildx create \
	--name local_remote_builder \
	--append \
	--node intelarch \
	--platform linux/amd64,linux/386 \
	ssh://USERNAME@TARGET_VM_HOST \
	--driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 \
	--driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000
```

> Do not forget to change the command's values to appropriate ones for your situation.

This command will append the remote node to the local_remote_builder and call the node intelarch. You can check it again by running `docker buildx ls` command again:

```bash
NAME/NODE              DRIVER/ENDPOINT        STATUS  BUILDKIT             PLATFORMS
local_remote_builder * docker-container                                    
  local_remote_builder orbstack               running v0.11.6              linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386
  intelarch            ssh://aboozar@167.21.183.24 running v0.11.6              linux/amd64*, linux/386*, linux/amd64/v2, linux/amd64/v3
```
## Step 5: buildx use and bootstrap
In order to start using this builder setup we need to tell buildx to start using it and we need to bootstrap it.

```bash
docker buildx use local_remote_builder
docker buildx inspect --bootstrap
```
output:
```
#1 [intelarch internal] booting buildkit
#1 starting container buildx_buildkit_intelarch
#1 ...

#2 [localremote_builder internal] booting buildkit
#2 pulling image moby/buildkit:buildx-stable-1 1.6s done
#2 creating container buildx_buildkit_local_remote_builder 0.6s done
#2 DONE 2.3s

#1 [intelarch internal] booting buildkit
#1 starting container buildx_buildkit_intelarch 3.6s done
#1 DONE 3.6s
Name:   localremote_builder
Driver: docker-container

Nodes:
Name:      local_remote_builder
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386

Name:      intelarch
Endpoint:  ssh://aboozar@167.21.183.24
Status:    running
Platforms: linux/amd64*, linux/386*, linux/amd64/v2
```

## Step 6: Build a multi-platform image

The command below will build for amd64 and arm64 but will direct the amd64 build to the remote node and the arm64 build will be done on the local machine.

```bash
docker buildx build --platform=linux/amd64,linux/arm64/v8 --push -t name/target:tag .
```
This will result in something like this on the docker hub:

![multiplatform docker image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l8lbg02xps4d7gqz32qe.png)

---

## Conclusion
The most obvious issue in this setup is that I can only build for multiple platforms when within my own network.
That is not necessarily true if you used a remote ssh accessible host or IP. I did not do that and that limits me to these builds when I have my remote node available in the network. For now, that is not a real issue for me.

It solved my concurrency problem. I had one build where I had to start the server, in order to configure it, during the docker build. Buildx does this in parallel and that gave me a port conflict as one was a bit faster but not ready when the other also tried to start. That was a problem when I tried to build all on only my local node. When I added the remote node this problem went away as the port was used on a completely different machine.

Overall, I am currently content with this solution.