Howto: WASM runtimes in Docker / Colima
Tags: webassembly, colima, docker, containerd
Published: 2024-01-12 18:37:15

# TL;DR
I could not find any guide how to add WASM container capability to Docker running on [Colima](https://github.com/abiosoft/colima). This guide provides a few Colima templates for exactly this, which adds [WasmEdge](https://wasmedge.org), [Wasmtime](https://wasmtime.dev) and [Wasmer](https://wasmer.io) runtime types.

# Context
Instead of installing [Docker Desktop](https://www.docker.com/products/docker-desktop/) and [enabling the WASM features](https://docs.docker.com/desktop/wasm/) I decided to see how I could add WASM runtime capabilities to a Colima setup. Colima is a great alternative to Docker Desktop if the licensing terms prevent you from using it.

Forgive me for not explaining everything from scratch, I just finished the last (and best) template and I needed to write it down so I could share it.

# Three alternative Colima templates
I created these three Colima templates:
1. `colima-crun-wasmedge`: Use the 'crun' runtime to launch WASM containers in WasmEdge;
2. `colima-runwasi-git`: Use the [Runwasi](https://github.com/containerd/runwasi) WASM shims in containerd to launch WASM containers in WasmEdge, Wasmtime or Wasmer, where the Runwasi shims are compiled from git master;
3. `colima-runwasi`: Same as `colima-runwasi-git`, but uses the latest releases of the Runwasi shims instead of building them.

# `colima-crun-wasmedge`
This was my first trial of getting WASM containers to work. To enable running WASM loads, [crun](https://github.com/containers/crun) needs to be compiled with WasmEdge (or Wasmtime) support. I could only get WasmEdge working, but the basic idea is the same.

Here is the template (edit the Colima template with `$ colima template`). I added comments to the parts relevant for the WASM config, other comments are removed for brevity:
## Template
```yaml
cpu: 4
disk: 60
memory: 12
arch: host
hostname: colima
autoActivate: true
forwardAgent: false

# I only tested this with 'docker', not 'containerd':
runtime: docker

kubernetes:
  enabled: false
  version: v1.24.3+k3s1
  k3sArgs: []

network:
  address: true
  dns: []
  dnsHosts:
    host.docker.internal: host.lima.internal

# Added:
# - containerd-snapshotter: true (meaning containerd will be used for pulling images)
# - default-runtime / runtimes: crun (instead of the default 'runc')
docker:
  default-runtime: crun
  features:
    buildkit: true
    containerd-snapshotter: true
  runtimes:
    crun:
      path: /usr/local/bin/crun

vmType: vz
rosetta: true
mountType: virtiofs
mountInotify: false
cpuType: host

# This provisioning script installs WasmEdge and builds crun with wasmedge support:
provision:
  - mode: system
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      echo "Install system updates:"
      apt-get update -y
      apt-get upgrade -y
      echo "Install WasmEdge and crun dependencies:"
      # NOTE: packages curl git python3 already installed:
      apt-get install -y make gcc build-essential pkgconf libtool libsystemd-dev libprotobuf-c-dev libcap-dev libseccomp-dev libyajl-dev libgcrypt20-dev go-md2man autoconf automake criu xz-utils
      apt-get clean -y
  - mode: user
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      echo "Installing WasmEdge:"
      curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | sudo bash -s -- -p /usr/local
      echo
      echo "`wasmedge -v` installed!"
      # NOTE: I failed to configure Wasmtime properly - turned off for now:
      #echo "Installing Wasmtime:"
      #curl -sSf https://wasmtime.dev/install.sh | bash
      #sudo cp .wasmtime/bin/* /usr/local/bin/
      #rm -rf .wasmtime
      #echo "`wasmtime -V` installed!"
      echo "Install crun:"
      git clone https://github.com/containers/crun
      cd crun
      ./autogen.sh
      #./configure --with-wasmedge --with-wasmtime
      ./configure --with-wasmedge
      make
      sudo make install
      crun -v
      echo "crun installed! Replacing runc with crun:"
      # NOTE: replacing runc with crun is to simplify containerd config
      TRC=`which runc`
      sudo rm -rf $TRC
      sudo cp `which crun` $TRC
      echo "Configuring containerd:"
      sudo mkdir -p /etc/containerd/
      containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
      echo "Restarting/reloading docker/containerd services:"
      sudo systemctl daemon-reload
      sudo systemctl restart containerd
      # As soon as Colima writes its /etc/docker/daemon.json file (right after this provisioning script),
      # it will also start the Docker daemon. If we stop Docker here, the changes will actually take effect:
      sudo systemctl stop docker

sshConfig: true
mounts: []
env: {}
```
## Caveats
### First colima start time is long
It takes 1.5 minutes on my M1 max mac to start the container.
### Extra docker run parameters
After starting Colima with this template, be aware that you need to supply some specific parameters (`--platform` and `--annotation`) to the `docker run` command, e.g.:
```bash
$ docker run --rm -dp 8080:8080 \
         --platform wasi/wasm32 \
         --annotation "run.oci.handler=wasm" \
         michaelirwin244/wasm-example:latest
```
or:
```bash
$ docker run --rm -dp 8080:8080 \
         --platform wasi/wasm32 \
         --annotation "module.wasm.image/variant=compat-smart" \
         michaelirwin244/wasm-example:latest
```
### Stopping container waits for 10 seconds
For some reason the SIGTERM signal is not honoured in the WASM runtime. Stop the container with a specified timeout:
```bash
$ docker stop -t 0 <container hash/name>
```

# `colima-runwasi-git`
My second attempt was to get the [Runwasi](https://github.com/containerd/runwasi) shims working with Docker, since I did not like the necessary `--annotation` parameter which are not needed for the 'official' Docker Desktop configuration. This template remedies that drawback.

Here is the template (edit the Colima template with `$ colima template`). I added comments to the parts relevant for the WASM config, other comments are removed for brevity:
## Template
```yaml
cpu: 4
disk: 60
memory: 12
arch: host
hostname: colima
autoActivate: true
forwardAgent: false

# I only tested this with 'docker', not 'containerd':
runtime: docker

kubernetes:
  enabled: false
  version: v1.24.3+k3s1
  k3sArgs: []

network:
  address: true
  dns: []
  dnsHosts:
    host.docker.internal: host.lima.internal

# Added:
# - containerd-snapshotter: true (meaning containerd will be used for pulling images)
docker:
  features:
    buildkit: true
    containerd-snapshotter: true

vmType: vz
rosetta: true
mountType: virtiofs
mountInotify: false
cpuType: host

# This provisioning script installs build dependencies, WasmEdge and builds the WASM runtime shims for containerd.
# NOTE: this takes a LOOONG time!
provision:
  - mode: system
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      echo "Installing system updates:"
      apt-get update -y
      apt-get upgrade -y
      echo "Installing WasmEdge and runwasi build dependencies:"
      # NOTE: packages curl, git and python3 already installed:
      apt-get install -y make gcc build-essential pkgconf libtool libsystemd-dev libprotobuf-c-dev libcap-dev libseccomp-dev libyajl-dev libgcrypt20-dev go-md2man autoconf automake criu pkg-config libdbus-glib-1-dev libelf-dev libclang-dev libzstd-dev protobuf-compiler xz-utils
      apt-get clean -y
  - mode: user
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      #
      # Setting vars for this script:
      #
      # Which WASM runtimes to install (wasmedge, wasmtime and wasmer are supported):
      WASM_RUNTIMES="wasmedge wasmtime wasmer"
      #
      # Location of the containerd config file:
      CONTAINERD_CONFIG="/etc/containerd/config.toml"
      #
      # Target location for the WASM runtimes and containerd shims ($TARGET/bin and $TARGET/lib):
      TARGET="/usr/local"
      #
      # Install rustup:
      #
      echo "Installing rustup for building runwasi:"
      curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain none -y
      source "$HOME/.cargo/env"
      #
      # Install selected WASM runtimes and containerd shims:
      #
      [[ -z "${WASM_RUNTIMES// /}" ]] && echo "No WASM runtimes selected - exiting!" && exit 0
      git clone https://github.com/containerd/runwasi
      echo "Installing WASM runtimes and building containerd shims: ${WASM_RUNTIMES}:"
      sudo mkdir -p /etc/containerd/
      containerd config default | sudo tee $CONTAINERD_CONFIG >/dev/null
      for runtimeName in $WASM_RUNTIMES; do
        case $runtimeName in
          wasmedge)
            echo "Installing WasmEdge:"
            curl -sSfL https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | sudo bash -s -- -p $TARGET
            echo
            echo "`wasmedge -v` installed!"
            ;;
          wasmtime)
            echo "Installing wasmtime:"
            curl -sSfL https://wasmtime.dev/install.sh | bash
            sudo cp .wasmtime/bin/* ${TARGET}/bin/
            rm -rf .wasmtime
            echo "`wasmtime -V` installed!"
            ;;
          wasmer)
            echo "Installing wasmer:"
            curl -sSfL https://get.wasmer.io | sh
            sudo cp .wasmer/bin/* ${TARGET}/bin/
            sudo cp .wasmer/lib/* ${TARGET}/lib/
            rm -rf .wasmer
            echo "`wasmer -V` installed!"
            ;;
          *)
            echo "ERROR: WASM runtime $runtimeName is not supported!"
            exit 1
            ;;
        esac
        cd runwasi
        echo "Building containerd-shim-${runtimeName}:"
        cargo build -p containerd-shim-${runtimeName} --release
        echo "Installing containerd-shim-${runtimeName}-v1:"
        sudo install ./target/release/containerd-shim-${runtimeName}-v1 ${TARGET}/bin
        sudo ln -sf ${TARGET}/bin/containerd-shim-${runtimeName}-v1 ${TARGET}/bin/containerd-shim-${runtimeName}d-v1
        sudo ln -sf ${TARGET}/bin/containerd-shim-${runtimeName}-v1 ${TARGET}/bin/containerd-${runtimeName}d
        echo "containerd-shim-${runtimeName} installed."
        cd ..
        echo "[plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.${runtimeName}]" | sudo tee -a $CONTAINERD_CONFIG >/dev/null
        echo "  runtime_type = \"io.containerd.${runtimeName}.v1\"" | sudo tee -a $CONTAINERD_CONFIG >/dev/null
      done
      echo "containerd WASM runtimes and shims installed."
      #
      # Restart the systemctl services to pick up the installed shims.
      # NOTE: We need to 'stop' docker because at this point the actual daemon.json config is not yet provisioned:
      #
      echo "Restarting/reloading docker/containerd services:"
      sudo systemctl daemon-reload
      sudo systemctl restart containerd
      sudo systemctl stop docker

sshConfig: true
mounts: []
env: {}
```
## Caveats
### First Colima start time is super long
It takes a couple of minutes on my M1 max mac to start the container. Have some patience!
### Stopping container waits for 10 seconds
For some reason the SIGTERM signal is not honoured in the WASM runtime. Stop the container with a specified timeout:
```bash
$ docker stop -t 0 <container hash/name>
```

# `colima-runwasi`
This template is the same as the previous template, except it doesn't build the containerd shims, it downloads the latest releases. Starting Colima with this template for the first time takes less than 50 seconds on my M1 max machine.

Here is the template (edit the Colima template with `$ colima template`). I added comments to the parts relevant for the WASM config, other comments are removed for brevity:
## Template
```yaml
cpu: 4
disk: 60
memory: 12
arch: host
hostname: colima
autoActivate: true
forwardAgent: false

# I only tested this with 'docker', not 'containerd':
runtime: docker

kubernetes:
  enabled: false
  version: v1.24.3+k3s1
  k3sArgs: []

network:
  address: true
  dns: []
  dnsHosts:
    host.docker.internal: host.lima.internal

# Added:
# - containerd-snapshotter: true (meaning containerd will be used for pulling images)
docker:
  features:
    buildkit: true
    containerd-snapshotter: true

vmType: vz
rosetta: true
mountType: virtiofs
mountInotify: false
cpuType: host

# Custom provision scripts for the virtual machine.
provision:
  - mode: system
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      echo "Installing system updates:"
      apt-get update -y
      apt-get upgrade -y
      echo "Installing dependency for wasmtime installer:"
      apt-get install -y xz-utils
      apt-get clean -y
  - mode: user
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      #
      # Setting vars for this script:
      #
      # Which WASM runtimes to install (wasmedge, wasmtime and wasmer are supported):
      WASM_RUNTIMES="wasmedge wasmtime wasmer"
      #
      # Location of the containerd config file:
      CONTAINERD_CONFIG="/etc/containerd/config.toml"
      #
      # Target location for the WASM runtimes and containerd shims ($TARGET/bin and $TARGET/lib):
      TARGET="/usr/local"
      #
      # Install selected WASM runtimes and containerd shims:
      #
      [[ -z "${WASM_RUNTIMES// /}" ]] && echo "No WASM runtimes selected - exiting!" && exit 0
      echo "Installing WASM runtimes and containerd shims: ${WASM_RUNTIMES}:"
      sudo mkdir -p /etc/containerd/
      containerd config default | sudo tee $CONTAINERD_CONFIG >/dev/null
      for runtimeName in $WASM_RUNTIMES; do
        case $runtimeName in
          wasmedge)
            echo "Installing WasmEdge:"
            curl -sSfL https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | sudo bash -s -- -p $TARGET
            echo
            echo "`wasmedge -v` installed!"
            ;;
          wasmtime)
            echo "Installing wasmtime:"
            curl -sSfL https://wasmtime.dev/install.sh | bash
            sudo cp .wasmtime/bin/* ${TARGET}/bin/
            rm -rf .wasmtime
            echo "`wasmtime -V` installed!"
            ;;
          wasmer)
            echo "Installing wasmer:"
            curl -sSfL https://get.wasmer.io | sh
            sudo cp .wasmer/bin/* ${TARGET}/bin/
            sudo cp .wasmer/lib/* ${TARGET}/lib/
            rm -rf .wasmer
            echo "`wasmer -V` installed!"
            ;;
          *)
            echo "ERROR: WASM runtime $runtimeName is not supported!"
            exit 1
            ;;
        esac
        shimVersion=$(curl -s https://api.github.com/repos/containerd/runwasi/tags | grep tarball | grep "shim-${runtimeName}" | grep -Eo 'https://[^\"]*' | head -1 | tr "/" "\n" | tail -n 1)
        shimUrl="https://github.com/containerd/runwasi/releases/download/containerd-shim-${runtimeName}/${shimVersion}/containerd-shim-${runtimeName}-`uname -m`.tar.gz"
        echo "Installing runwasi shim version $shimVersion for $runtimeName runtime from ${shimUrl}:"
        curl -sSfL $shimUrl | sudo tar xvz -C ${TARGET}/bin/
        echo "[plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.${runtimeName}]" | sudo tee -a $CONTAINERD_CONFIG >/dev/null
        echo "  runtime_type = \"io.containerd.${runtimeName}.v1\"" | sudo tee -a $CONTAINERD_CONFIG >/dev/null
      done
      echo "containerd WASM runtimes and shims installed."
      #
      # Restart the systemctl services to pick up the installed shims.
      # NOTE: We need to 'stop' docker because at this point the actual daemon.json config is not yet provisioned:
      #
      echo "Restarting/reloading docker/containerd services:"
      sudo systemctl daemon-reload
      sudo systemctl restart containerd
      sudo systemctl stop docker

sshConfig: true
mounts: []
env: {}
```
## Caveats
### Stopping container waits for 10 seconds
For some reason the SIGTERM signal is not honoured in the WASM runtime. Stop the container with a specified timeout:
```bash
$ docker stop -t 0 <container hash/name>
```
# Putting it all together
Let's go from Colima installation to running a WASM program in Docker.

## Prepare Colima
First, if you haven't done this already, install Colima:
```bash
$ brew install colima
```

Next, change the default Colima template:
```bash
$ colima template
```
This command opens (by default) a `vim` editor with the default colima yaml template. In this editor, replace all contents with the template from `colima-runwasi` above.

> In `vim`: Type the following key sequence to empty the template:
> ```
> :%d<Enter>
> ```
> The screen must be empty now.
>
> Next, put `vim` in `-- INSERT --` mode by typing the `i` key.
> Then, paste the contents of the `colima-runwasi` template into the editor.
>
> Finally, save the template by typing the following key sequence:
> ```
> <ESC>:wq!<Enter>
> ```

The Colima template is now configured to install WASM runtimes. Start Colima:
```bash
$ colima start -v
```
Depending on the speed of your machine, this takes approximately between 50 and 90 seconds.

## Install additional Docker plugins
```bash
$ brew install docker-buildx docker-compose
```
Make sure to follow instructions for the installed Docker plugins:
```bash
$ mkdir -p ~/.docker/cli-plugins
$ ln -sfn $(brew --prefix)/opt/docker-buildx/bin/docker-buildx ~/.docker/cli-plugins/docker-buildx
$ ln -sfn $(brew --prefix)/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose
```

## Build a WASM image
Check out a project with rust examples:
```bash
$ git clone https://github.com/second-state/rust-examples.git

Cloning into 'rust-examples'...
remote: Enumerating objects: 338, done.
remote: Counting objects: 100% (73/73), done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 338 (delta 42), reused 40 (delta 20), pack-reused 265
Receiving objects: 100% (338/338), 54.27 KiB | 9.04 MiB/s, done.
Resolving deltas: 100% (214/214), done.
```

Next, enter the `rust-examples/hello` directory and build the wasm image:
```bash
$ cd rust-examples/hello
$ docker buildx build --provenance=false \
         --platform wasi/wasm32 \
         -t secondstate/rust-example-hello .

[+] Building 3.6s (15/15) FINISHED
 => [internal] load .dockerignore
 => => transferring context: 2B
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 563B
 => resolve image config for docker.io/docker/dockerfile:1
 => [auth] docker/dockerfile:pull token for registry-1.docker.io
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021
 => => resolve docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021
 => [internal] load metadata for docker.io/library/rust:1.64
 => [auth] library/rust:pull token for registry-1.docker.io
 => [buildbase 1/3] FROM docker.io/library/rust:1.64@sha256:53ded1c919ea0dc23be959e28037238d8c321cd88fbac04d818f210031fccc3b
 => => resolve docker.io/library/rust:1.64@sha256:53ded1c919ea0dc23be959e28037238d8c321cd88fbac04d818f210031fccc3b
 => [internal] load build context
 => => transferring context: 426B
 => CACHED [buildbase 2/3] WORKDIR /src
 => CACHED [buildbase 3/3] RUN <<EOT bash
 => [build 1/3] COPY Cargo.toml .
 => [build 2/3] COPY src ./src
 => [build 3/3] RUN cargo build --target wasm32-wasi --release
 => exporting to image
 => => exporting layers
 => => exporting manifest sha256:506f9c13793e6fd5ce556e36130a7e6977b68720379a393b1a0d2acc23aef501
 => => exporting config sha256:1274e89bbbc5c385dae15ac1b768cf2acaae290172018a03efd2dae1d2f0e872
 => => naming to docker.io/secondstate/rust-example-hello:latest
 => => unpacking to docker.io/secondstate/rust-example-hello:latest
```
Check the image's existence by running the following command:
```bash
$ docker images

REPOSITORY                       TAG       IMAGE ID       CREATED         SIZE
secondstate/rust-example-hello   latest    506f9c13793e   3 minutes ago   2.53MB
```

Or, check its architecture to verify it is a WASM image:
```bash
$ docker inspect \
         --format='{{.Os}}/{{.Architecture}}' \
         secondstate/rust-example-hello

wasi/wasm32
```

## Run the WASM image
Run the WASM image with the following command:
```bash
$ docker run --rm \
         --platform wasi/wasm32 \
         --runtime io.containerd.wasmedge.v1 \
         secondstate/rust-example-hello:latest

Hello WasmEdge!
```
> NOTE: The specified runtime above is `io.containerd.wasmedge.v1`. You have three options here:
> 1. WasmEdge: `io.containerd.wasmedge.v1`
> 2. Wasmtime: `io.containerd.wasmtime.v1`
> 3. Wasmer: `io.containerd.wasmer.v1`

# Conclusion
If you are serious about using WASM containers in Docker / Colima, I suggest to use the last template `colima-runwasi`. The other templates are merely evolutionary exercises in getting to know the Docker parts better.

Let me know how this works for you!

# Resources
The information in this Howto is assembled from these links:
1. [The differences between Docker, containerd, CRI-O and runc](https://www.tutorialworks.com/difference-docker-containerd-runc-crio-oci/)
2. [Docker - Use containerd shims](https://docs.docker.com/engine/alternative-runtimes/#use-containerd-shims)
3. [WebAssembly and containerd: How it works](https://nigelpoulton.com/webassembly-and-containerd-how-it-works/)

# EDITS
14 FEB 2024 - added `xz-utils` to list of install dependencies for wasmtime