Multi-architecture Kubernetes clusters on Amazon EKS
Tags: kubernetes, aws, docker
Published: 2023-10-30 21:01:12


AWS Graviton processors continue to deliver price performance for cloud workloads. Migrating Kubernetes applications can be tricky, but AWS and Docker provide tools to make the process easier.

You can experiment with multi-architecture clusters to better understand the price performance benefits, and select the best hardware for the task. Multi-architecture clusters also ease migration when some software components are not available for the Arm architecture. 

Refer to the [The Insider’s Guide to Building a Multi-Arch Infrastructure](https://armkeil.blob.core.windows.net/developer/Files/pdf/white-paper/arm-whitepaper-insiders-guide-to-building-a-multi-arch-infrastructure.pdf) for more information on the benefits of multi-architecture Kubernetes clusters.

# Multi-architecture Kubernetes clusters

A multi-architecture Kubernetes cluster runs workloads on multiple hardware architectures, typically arm64 and amd64 (using Docker terminology).

To learn more about multi-architecture Kubernetes you can create a cluster in Amazon EKS and gain some practical experience with arm64 and amd64 nodes. This will also help you understand multi-architecture container images. 

# Before you begin

You will need `docker`, `eksctl` and `kubectl` on your local computer to try out the steps below.

Your local computer can be Windows, Linux, or macOS. The instructions generally assume Linux. 

[Multipass](https://learn.arm.com/install-guides/multipass/) from Canonical is a great tool for Linux virtual machines if you use macOS.

You can refer to the [Kubectl install guide](https://learn.arm.com/install-guides/kubectl/) and the [AWS EKS CLI install guide](https://learn.arm.com/install-guides/eksctl/) for installation details. The guides provide details for Arm Linux machines, but links to installation information about other platforms are also provided. 

There are plenty of places to find Docker installation instructions. One option is the [Docker install guide](https://learn.arm.com/install-guides/docker/).

You will also need an AWS account. 

To create an account, go to https://aws.amazon.com and click on **Create an AWS Account** in the top right corner. Follow the instructions to register. See the [Creating an AWS account documentation](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-creating.html) for full instructions.

Make sure to configure your access key ID and secret access key, which are used to sign programmatic requests that you make to AWS. Refer to [AWS Credentials](https://learn.arm.com/install-guides/aws_access_keys/) for a quick summary of how to run `aws configure`.

# Multi-architecture containers

Before you can try out a multi-architecture Kubernetes cluster, you need a multi-architecture container image. 

Multi-architecture container images are the easiest way to deploy applications, and hide the underlying hardware architecture. Building multi-architecture images is slightly more complex compared to building single-architecture images.

Docker provides two ways to create multi-architecture images:
- `docker buildx` builds both architectures at the same time
- `docker manifest` builds each architecture separately and joins them together into a multi-architecture image

You can read more about how to create container images in the article [Using Docker manifest to create multi-arch images on AWS Graviton processors](https://dev.to/aws-builders/using-docker-manifest-to-create-multi-arch-images-on-aws-graviton-processors-1320)

Below is a simple Go application you can use to learn about multi-architecture Kubernetes clusters.

Use your favorite text editor to create a file named `hello.go` with the contents below:

```go
// Copyright 2022 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
	"runtime"
)

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello from image NODE:%s, POD:%s, CPU PLATFORM:%s/%s",
		os.Getenv("NODE_NAME"), os.Getenv("POD_NAME"), runtime.GOOS, runtime.GOARCH)
}

func main() {
	http.HandleFunc("/", handler)
	log.Fatal(http.ListenAndServe(":8080", nil))
}
```

Next, create a file named `go.mod` with the following two lines:

```go
module example.com/arm
go 1.21
```

Create a third file named `Dockerfile` with the contents:

```dockerfile
ARG T

#
# Build: 1st stage
#
FROM golang:1.21-alpine as builder 
ARG TARCH
WORKDIR /app
COPY go.mod .
COPY hello.go .
RUN GOARCH=${TARCH} go build -o /hello && \
    apk add --update --no-cache file && \
    file /hello   

#
# Release: 2nd stage
#
FROM ${T}alpine
WORKDIR /
COPY --from=builder /hello /hello
RUN apk add --update --no-cache file
CMD [ "/hello" ]
```

## Docker buildx

With the three text files created, you can build the container using `docker buildx`. 

Before you start, log in to Docker Hub:

```bash
docker login
```

Run the commands below to build the image, and make sure to substitute your own Docker ID instead of `jasonrandrews`. 

```bash
docker buildx create --use --name builder
docker buildx build --platform linux/amd64,linux/arm64 -t jasonrandrews/go-arch-x:1.0 --push .
```

When the commands are complete, you will see the new image in your Docker Hub account. Note, this is the `go-arch-x` repository.

## Docker manifest

You can also use `docker manifest` to create a multi-architecture image from two single-architecture images. This is an alternative way to to build the multi-architecture image, but you can also use it to learn how to deploy a container image that only supports one architecture.

Substitute your Docker ID instead of `jasonrandrews` and run these commands on an amd64 machine:

```bash
# Build for amd64
docker build -t jasonrandrews/go-arch:amd64 --build-arg TARCH=amd64 --build-arg T=amd64/ .
docker push jasonrandrews/go-arch:amd64
```

Substitute your Docker ID instead of `jasonrandrews` and run these commands on an arm64 machine:

```bash
# Build for arm64
docker build -t jasonrandrews/go-arch:arm64 --build-arg TARCH=arm64 --build-arg T=arm64v8/ .
docker push jasonrandrews/go-arch:arm64
```

After building individual containers for each architecture, merge them into a single image by running the commands below on either architecture:

```bash
docker manifest create jasonrandrews/go-arch:1.0 \
--amend jasonrandrews/go-arch:arm64 \
--amend jasonrandrews/go-arch:amd64
docker manifest push --purge jasonrandrews/go-arch:1.0
```

You now have two single-architecture images in your Docker Hub account and the combined multi-architecture image in the same repository. Note, this is the `go-arch` repository.

In the sections below, you will learn how to deploy the images in Amazon EKS.

# Create a multi-architecture Kubernetes cluster

You can use `eksctl` to create a cluster with multiple architectures.

First, use a text editor to create a text file `cluster.yaml` with the contents below:

```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster1
  region: us-east-1

nodeGroups:
  - name: ng-1
    instanceType: t2.micro
    desiredCapacity: 2
    volumeSize: 80
  - name: ng-2
    instanceType: t4g.small
    desiredCapacity: 2
    volumeSize: 80
```

The instance types above take advantage of free tier and the T4g free trial. 

Run the `eksctl` command with the yaml file to create an EKS cluster:

```bash
eksctl create cluster -f cluster.yaml
```

The cluster includes two amd64 nodes and two arm64 nodes.

It takes 15-20 minutes to launch the cluster so take a break while it is created.

When the cluster is ready check the nodes by running `kubectl` on your local machine:

```bash
kubectl get nodes
```

The output shows four nodes, but doesn't print the architecture.

```output
NAME                             STATUS   ROLES    AGE    VERSION
ip-192-168-11-18.ec2.internal    Ready    <none>   106s   v1.27.5-eks-43840fb
ip-192-168-27-254.ec2.internal   Ready    <none>   71s    v1.27.5-eks-43840fb
ip-192-168-32-244.ec2.internal   Ready    <none>   106s   v1.27.5-eks-43840fb
ip-192-168-61-31.ec2.internal    Ready    <none>   72s    v1.27.5-eks-43840fb
```

To get the architecture run:

```bash
kubectl get node -o jsonpath='{.items[*].status.nodeInfo.architecture}'
```

The output shows four nodes, two of each architecture. 

```output
arm64 amd64 amd64 arm64
```

The Kubernetes cluster is now ready to use, and you can deploy the Go app. 

# Create a service to deploy

To deploy the application create a text file named `hello-service.yaml` with the contents:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: hello-service
  labels:
    app: hello
    tier: web
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: hello
    tier: web
```

Deploy the service by running:

```bash
kubectl apply -f hello-service.yaml
```

The output is:

```output
service/hello-service created
```

# Deploy the amd64 application

You can deploy the amd64 container first to demonstrate a single-architecture container. 

Create a text file named `amd64-deployment.yaml` with the contents below. Make sure to use your Docker ID instead of `jasonrandrews` in the file. Notice that the container image is `go-arch:amd64`. The amd64 image will only run on amd64 nodes. The nodeSelector is used to make sure the continer is only scheduled on amd64.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: amd-deployment
  labels:
    app: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
      tier: web
  template:
    metadata:
      labels:
        app: hello
        tier: web
    spec:
      containers:
      - name: hello
        image: jasonrandrews/go-arch:amd64
        imagePullPolicy: Always
        ports:
          - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        resources:
          requests:
            cpu: 300m
      nodeSelector:
        kubernetes.io/arch: amd64
```

Deploy the amd64 container by running:

```bash
kubectl apply -f amd64-deployment.yaml
```

You can see the running deployment.

```bash
kubectl get pods
```

The output shows 1 pod running:

```output
NAME                              READY   STATUS    RESTARTS   AGE
amd-deployment-7d4d44889d-vzhpd   1/1     Running   0          9s
```

To get the endpoint of the service run:

```bash
kubectl get services
```

The output prints the IP address of the load balancer. 

```output
NAME            TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)        AGE
hello-service   LoadBalancer   10.100.188.181   a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com   80:30489/TCP   70s
kubernetes      ClusterIP      10.100.0.1       <none>                                                                    443/TCP        14m
```

To confirm the amd64 application is running use `curl` and substitute the endpoint of your load balancer:

```bash
curl -w '\n' http://a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com
```

The output displays the hello message from the application and confirms the architecture is amd64.

```output
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:amd-deployment-7d4d44889d-vzhpd, CPU PLATFORM:linux/amd64
```

Save the address of your load balancer (the http address) and use it for the `curl` commands in the following sections. 

# Deploy the arm64 application

Create a text file named `arm64-deployment.yaml` with the contents below. Again, substitute your Docker ID and notice that the nodeSelector is now arm64.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: arm-deployment
  labels:
    app: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
      tier: web
  template:
    metadata:
      labels:
        app: hello
        tier: web
    spec:
      containers:
      - name: hello
        image: jasonrandrews/go-arch:arm64
        imagePullPolicy: Always
        ports:
          - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        resources:
          requests:
            cpu: 300m
      nodeSelector:
        kubernetes.io/arch: arm64

```

Deploy the arm64 container by running:

```bash
kubectl apply -f arm64-deployment.yaml
```

You can see the running deployment.

```bash
kubectl get pods
```

Now, the output shows both deployments, amd64 and arm64.

```output
NAME                              READY   STATUS    RESTARTS   AGE
amd-deployment-7d4d44889d-vzhpd   1/1     Running   0          3m1s
arm-deployment-5996f6b85d-wp8cv   1/1     Running   0          10s
```

Run the `curl` command a few times and you will see a mix of amd64 and arm64 messages printed, depending on which node services the request.

```bash
curl -w '\n' http://a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com
```

# Deploy the multi-architecture application

You can also deploy the multi-architecture image.

Create a text file named `multi-arch-deployment.yaml` with the contents below. Again, substitute your Docker ID. The image is the multi-architecture image created with `docker buildx` and 6 replicas are specified.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-arch-deployment
  labels:
    app: hello
spec:
  replicas: 6
  selector:
    matchLabels:
      app: hello
      tier: web
  template:
    metadata:
      labels:
        app: hello
        tier: web
    spec:
      containers:
      - name: hello
        image: jasonrandrews/go-arch-x:1.0
        imagePullPolicy: Always
        ports:
          - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        resources:
          requests:
            cpu: 300m
```

Deploy the multi-architecture container by running:

```bash
kubectl apply -f multi-arch-deployment.yaml
```

You can see the running deployment.

```bash
kubectl get pods
```

Now, the output shows all three deployments:

```output
NAME                                     READY   STATUS    RESTARTS   AGE
amd-deployment-7d4d44889d-vzhpd          1/1     Running   0          4m24s
arm-deployment-5996f6b85d-wp8cv          1/1     Running   0          93s
multi-arch-deployment-547684cd44-7w6vf   1/1     Running   0          7s
multi-arch-deployment-547684cd44-chcc8   1/1     Running   0          7s
multi-arch-deployment-547684cd44-kw966   1/1     Running   0          7s
multi-arch-deployment-547684cd44-r4789   1/1     Running   0          7s
multi-arch-deployment-547684cd44-vqwc7   1/1     Running   0          7s
multi-arch-deployment-547684cd44-w4x8d   1/1     Running   0          7s
```

To test the appliation, run a loop of `curl` commands and see the output messages from the application.

```bash
for i in $(seq 1 10); do curl -w '\n' http://a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com; done
```

The output will show a variety of amd64 and arm64 messages:

```output
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-r4789, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-r4789, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-27-254.ec2.internal, POD:multi-arch-deployment-547684cd44-kw966, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-chcc8, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-27-254.ec2.internal, POD:arm-deployment-5996f6b85d-wp8cv, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-r4789, CPU PLATFORM:linux/arm64
```

You have now deployed single-architecture images for amd64 and arm64 as well as six copies of a multi-architecture image. 

You can use the same techniques to incrementally migrate applications from amd64 to arm64 and invesigate the price performance provided by AWS Graviton processors. 

# Delete the clusters

When you are done, make sure to delete all AWS resources. 

You can clean up with a single `eckctl` command. 

```bash
eksctl delete cluster -n cluster1
```