WASM workload on AKS
Tags: azure, webassembly, kubernetes, todayilearned
Published: 2021-10-21 13:36:58

Public preview of AKS supporting WASI (WebAssembly System Interface) workload in Kubernetes is out. And I thought I would try it out.

https://azure.microsoft.com/en-us/updates/public-preview-aks-support-for-webassembly-system-interface-wasi-workloads/

## TOC
  * [Register WasmNodePoolPreview feature](## Register WasmNodePoolPreview feature)
  * [Install aks-preview extension](## Install aks-preview extension)
  * [Add WASM/WASI node pool to AKS cluster](## Add WASM/WASI node pool to AKS cluster)
  * [Run WASM/WASI Workload](## Run WASM/WASI Workload)
  * [Create my own WASM workload with Rust](## Create my own WASM workload with Rust)
  * [Upload the WASM module to Azure Container Registry](## Upload the WASM module to Azure Container Registry)
  * [Deploy my WASM module](## Deploy my WASM module)
  * [Clean up](## Clean up)


## Register WasmNodePoolPreview feature

Register `WasmNodePoolPreview` feature flag to your subscription

```bash
$ az feature register --namespace "Microsoft.ContainerService" --name "WasmNodePoolPreview"
{
  "id": "/subscriptions/<YOUR SUBSCRIPTION ID>/providers/Microsoft.Features/providers/Microsoft.ContainerService/features/WasmNodePoolPreview",
  "name": "Microsoft.ContainerService/WasmNodePoolPreview",
  "properties": {
    "state": "Registering"
  },
  "type": "Microsoft.Features/providers/features"
}

$ az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/WasmNodePoolPreview')].{Name:name,State:properties.state}"
Name                                            State
----------------------------------------------  -----------
Microsoft.ContainerService/WasmNodePoolPreview  Registering

$ az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/WasmNodePoolPreview')].{Name:name,State:properties.state}"
Name                                            State
----------------------------------------------  ----------
Microsoft.ContainerService/WasmNodePoolPreview  Registered
```

## Install aks-preview extension

```bash
$ az extension add --name aks-preview
$ az extension update --name aks-preview
```

## Add WASM/WASI node pool to AKS cluster

```bash
# Create resource group
$ az group create --name wasmRG -l westus
# Create AKS cluster
$ az aks create --resource-group wasmRG --name myAKSCluster
# Add WASM/WASI node pool
$ az aks nodepool add --resource-group wasmRG --cluster-name myAKSCluster --name mywasipool --node-count 1 --workload-runtime wasmwasi
# Check the workloadRuntime
$ az aks nodepool show -g wasmRG --cluster-name myAKSCluster -n mywasipool | jq '.workloadRuntime'
"WasmWasi"
```

```bash
# Get credential for AKS cluster
$ az aks get-credentials -n myakscluster -g wasmRG

$ kubectl get nodes
NAME                                 STATUS   ROLES   AGE   VERSION
aks-mywasipool-11259355-vmss000000   Ready    agent   76m   1.0.0-alpha.1
aks-nodepool1-11259355-vmss000000    Ready    agent   80m   v1.20.9
aks-nodepool1-11259355-vmss000001    Ready    agent   80m   v1.20.9
aks-nodepool1-11259355-vmss000002    Ready    agent   80m   v1.20.9

# Get labels on node
$ kubectl get node aks-mywasipool-11259355-vmss000000 -o jsonpath='{.metadata.labels}' | jq
{
  "agentpool": "mywasipool",
  "beta.kubernetes.io/arch": "wasm32-wagi",
  "beta.kubernetes.io/os": "wasm32-wagi",
  "kubernetes.azure.com/agentpool": "mywasipool",
  "kubernetes.azure.com/cluster": "MC_wasmRG_myAKSCluster_westus",
  "kubernetes.azure.com/mode": "user",
  "kubernetes.azure.com/node-image-version": "AKSUbuntu-1804gen2containerd-2021.10.02",
  "kubernetes.azure.com/os-sku": "Ubuntu",
  "kubernetes.azure.com/role": "agent",
  "kubernetes.azure.com/storageprofile": "managed",
  "kubernetes.azure.com/storagetier": "Premium_LRS",
  "kubernetes.io/arch": "wasm32-wagi",
  "kubernetes.io/hostname": "aks-mywasipool-11259355-vmss000000",
  "kubernetes.io/os": "wasm32-wagi",
  "kubernetes.io/role": "agent",
  "node-role.kubernetes.io/agent": "",
  "storageprofile": "managed",
  "storagetier": "Premium_LRS",
  "type": "krustlet"
}

# Get Taints defined in node
$ kubectl get node aks-mywasipool-11259355-vmss000000 -o jsonpath='{.spec.taints}' | jq
[
  {
    "effect": "NoSchedule",
    "key": "kubernetes.io/arch",
    "value": "wasm32-wagi"
  },
  {
    "effect": "NoExecute",
    "key": "kubernetes.io/arch",
    "value": "wasm32-wagi"
  }
]
```

- label `"type": "krustlet"` shows that the node runs krustlet instead of kubelet
- label `"kubernetes.io/arch": "wasm32-wagi"` shows the node architecture
- taint on the wasi node pool will only schedule/execute pods with toleration of `wasm32-wagi`

## Run WASM/WASI Workload

- Create the following yaml as `wasi-example.yaml`
    - nodeSelector will choose the node with certain label (i.e. node with `kubernetes.io/arch: wasm32-wagi`)
    - toleration will tolerate the taint in the node (i.e.  wasm32-wagi NoSchedule and NoExecute)

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: krustlet-wagi-demo
  labels:
    app: krustlet-wagi-demo
  annotations:
    alpha.wagi.krustlet.dev/default-host: "0.0.0.0:3001"
    alpha.wagi.krustlet.dev/modules: |
      {
        "krustlet-wagi-demo-http-example": {"route": "/http-example", "allowed_hosts": ["https://api.brigade.sh"]},
        "krustlet-wagi-demo-hello": {"route": "/hello/..."},
        "krustlet-wagi-demo-error": {"route": "/error"},
        "krustlet-wagi-demo-log": {"route": "/log"},
        "krustlet-wagi-demo-index": {"route": "/"}
      }
spec:
  hostNetwork: true
  nodeSelector:
    kubernetes.io/arch: wasm32-wagi
  containers:
    - image: webassembly.azurecr.io/krustlet-wagi-demo-http-example:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-http-example
    - image: webassembly.azurecr.io/krustlet-wagi-demo-hello:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-hello
    - image: webassembly.azurecr.io/krustlet-wagi-demo-index:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-index
    - image: webassembly.azurecr.io/krustlet-wagi-demo-error:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-error
    - image: webassembly.azurecr.io/krustlet-wagi-demo-log:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-log
  tolerations:
    - key: "node.kubernetes.io/network-unavailable"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wagi"
      effect: "NoExecute"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wagi"
      effect: "NoSchedule"  
```

```yaml
$ kubectl apply -f wasi-example.yaml

$ kubectl get po -o wide
NAME                 READY   STATUS       RESTARTS   AGE   IP       NODE                                 NOMINATED NODE   READINESS GATES
krustlet-wagi-demo   0/5     Registered   0          78s   <none>   aks-mywasipool-11259355-vmss000000   <none>           <none>
```

- We can see that the pods has been scheduled on wasipool node

```yaml
# Get Internal IP for WASI Node
$ WASINODE_IP=$(kubectl get nodes aks-mywasipool-11259355-vmss000000 -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')

# Create values.yaml for nginx chart
$ cat << EOF > values.yaml
serverBlock: |-
  server {
    listen 0.0.0.0:8080;
    location / {
            proxy_pass http://$WASINODE_IP:3001;
    }
  }
EOF

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo update
$ helm install hello-wasi bitnami/nginx -f values.yaml

$ kubectl get service 
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
hello-wasi-nginx   LoadBalancer   10.0.163.230   40.83.129.0   80:32513/TCP   3m11s
kubernetes         ClusterIP      10.0.0.1       <none>        443/TCP        152m

# Get the external ip for the hello-wasi-nginx service
$ EXTERNAL_IP=$(kubectl get service hello-wasi-nginx -o jsonpath='{.status.loadBalancer.ingress[].ip}')

$ curl $EXTERNAL_IP/hello
hello world
```

## Create my own WASM workload with Rust

- Prerequisite: Rust, [wasmtime]([https://github.com/bytecodealliance/wasmtime](https://github.com/bytecodealliance/wasmtime))
- Set up the Rust environment

```bash
$ mkdir aks-wasm; cd aks-wasm
$ cargo init 
$ rustup target add wasm32-wasi
```

- Create a simple app for wasm (src/main.rs)

```bash
fn main() {
    println!("Content-Type: text/plain\n");
    println!("Hello from wasm!");
}
```

- Build the app and run it locally

```bash
$ cargo build --release --target wasm32-wasi
$ wasmtime target/wasm32-wasi/release/aks-wasm.wasm
Content-Type: text/plain

Hello from wasm!
```

## Upload the WASM module to Azure Container Registry

- Prerequisite: [wasm-to-oci]([https://github.com/engineerd/wasm-to-oci](https://github.com/engineerd/wasm-to-oci))
- Setup Azure ContainerRegistry

```bash
# You may have to choose a different name for the registry 
$ az acr create -n MyWASMRegistry -g wasmRG --sku Standard

# login to Azure Container Registry. Make sure Docker is running
$ az acr login -n MyWASMRegistry

# Enable anonymous pull to allow pulling the WASM module from WASI node
$ az acr update --name myregistry --anonymous-pull-enabled
```

- Push the WASM module to Azure Container Registry

```bash

# Push the WASM module to Azure Container Registry
$ wasm-to-oci push target/wasm32-wasi/release/aks-wasm.wasm mywasmregistry.azurecr.io/mywasm:1.0.0 

$ az acr repository list -n MyWASMRegistry
[
  "mywasm"
]

# Make sure the WASM module has been uploaded
$ az acr repository show -n MyWASMRegistry --image mywasm:1.0.0                                           [18:27:29]
{
  "changeableAttributes": {
    "deleteEnabled": true,
    "listEnabled": true,
    "readEnabled": true,
    "writeEnabled": true
  },
  "createdTime": "2021-10-21T09:21:15.8964565Z",
  "digest": "sha256:3cade5d310ece611b4d87df55c64e4e6d99ae08d052cfaef2e699df56db90fef",
  "lastUpdateTime": "2021-10-21T09:21:15.8964565Z",
  "name": "1.0.0",
  "signed": false
}
```

## Deploy my WASM module

- Update `wasi-example.yaml` by adding the wasm module we created earlier

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: krustlet-wagi-demo
  labels:
    app: krustlet-wagi-demo
  annotations:
    alpha.wagi.krustlet.dev/default-host: "0.0.0.0:3001"
    alpha.wagi.krustlet.dev/modules: |
      {
        "krustlet-wagi-demo-http-example": {"route": "/http-example", "allowed_hosts": ["https://api.brigade.sh"]},
        "krustlet-wagi-demo-hello": {"route": "/hello/..."},
        "krustlet-wagi-demo-error": {"route": "/error"},
        "krustlet-wagi-demo-log": {"route": "/log"},
        "krustlet-wagi-demo-index": {"route": "/"},
        "krustlet-wagi-mywasm": {"route": "/mywasm"}
      }
spec:
  hostNetwork: true
  nodeSelector:
    kubernetes.io/arch: wasm32-wagi
  containers:
    - image: webassembly.azurecr.io/krustlet-wagi-demo-http-example:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-http-example
    - image: webassembly.azurecr.io/krustlet-wagi-demo-hello:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-hello
    - image: webassembly.azurecr.io/krustlet-wagi-demo-index:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-index
    - image: webassembly.azurecr.io/krustlet-wagi-demo-error:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-error
    - image: webassembly.azurecr.io/krustlet-wagi-demo-log:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-log
    - image: mywasmregistry.azurecr.io/mywasm:1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-mywasm
  tolerations:
    - key: "node.kubernetes.io/network-unavailable"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wagi"
      effect: "NoExecute"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wagi"
      effect: "NoSchedule"
```

- Restart the AKS cluster since the WASM module does not get updated for some reason.

```bash
# Delete current WASM workload
$ kubectl delete -f wasi-example.yaml

# Restart AKS cluster
$ az aks stop -n myakscluster -g wasmRG && az aks start -n myakscluster -g wasmRG

# Redeploy WASM workload
$ kubectl applt -f wasi-example.yaml

$ EXTERNAL_IP=$(kubectl get service hello-wasi-nginx -o jsonpath='{.status.loadBalancer.ingress[].ip}')

$ curl $EXTERNAL_IP/mywasm
Hello from wasm!
```

## Clean up

```yaml
$ helm delete hello-wasi
$ kubectl delete -f wasi-example.yaml
$ az aks nodepool delete --name mywasipool -g wasmRG --cluster-name myakscluster
```

## Reference
- https://docs.microsoft.com/en-us/azure/aks/use-wasi-node-pools
- https://blog.baeke.info/2021/10/18/trying-out-webassembly-on-azure-kubernetes-service/