Kubernetes with Amazon Elastic Kubernetes Service
Tags: kubernetes, aws
Published: 2023-01-16 08:13:33

There are several options to roll your own Kubernetes cluster. In addition to provisioning on managed infrastructure, you can also create a managed cluster that is integrated with a particular cloud environment, enabling you to use other services and concepts from the cloud environment.

This article is a hands-on tutorial to setup the Kubernetes platform Elastic Kubernetes Service (EKS) on AWS. Step-by-step, you will learn how to download the required binaries, get the AWS authorization, draft a EKS configuration, and the cluster installation procedure. You will also see how to access the cluster with both `kubectl` and SSH and learn how to perform a cluster update.

_This article originally appeared at my blog [admantium.com](https://admantium.com/blog/kube15_kubernetes_on_aws)_.

## Part 1: Prerequisites

To follow this tutorial, you need to have a fully registered and activated AWS account. This account also needs to have complete access rights for managing theses required resources: AWS instances, VPC, IAM user.

You also need a dedicated computer to install the required binaries and perform the installation steps, and you will also connect from this machine to the cluster, either by using `kubectl` or SSH. This machine is called the EKS controller in the remainder of this article.

Finally, resource creation and utilization generate costs - be sure you are willing to pay for this experience! In a three-day period, I paid $30. You can keep a good overview about all costs with the [cost dashboard](https://us-east-1.console.aws.amazon.com/cost-management/home?region=us-east-1&skipRegion=true#/dashboard) - but only after they incurred.

## Part 2: Tool Installation

The first task is to install all the necessary binary tools on the EKS controller.

### kubectl

The binary for interacting with a Kubernetes cluster. It can be installed from your package manager, or you can just grab the binary suitable for your architecture.

To install version 1.24.0 On a Linux host, execute these commands:

```bash
cd ~
curl -LO https://dl.k8s.io/release/v1.24.0/bin/linux/amd64/kubectl
chmod +x kubectl
mkdir -p ~/.local/bin
mv ./kubectl ~/.local/bin/kubectl
```

### eksctl

This CLI tool creates and manages the Kubernetes cluster. It’s provided by Amazon as a binary file. Go to its [Github releases page](https://github.com/weaveworks/eksctl/releases) and grab the binary for your OS.

For a Linux host, execute these commands:

```bash
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
chmod +x /tmp/eksctl
mkdir -p ~/.local/bin
mv /tmp/eksctl ~/.local/bin/eksctl
```

## Part 3: Cluster Configuration

The cluster creation is a one-time command. The cluster itself is either preconfigured, or can be customized by flags or a config file. Configuration options are extensive - see the full [configuration file documentation](https://eksctl.io/usage/schema/) for all options. In this tutorial, I'm following the preconfigured path, using mostly default settings.

To get started, we need to define the following things:

- Name: The name of the cluster
- Region: The AWS availability region in which you want to deploy the cluster
- Kubernetes version: Check the [supported EKS versions](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html) and ensure that you have a compatible `kubectl` version installed
- Node number: How many worker nodes the cluster should have
- AWS instance (optionally): The instance type of the nodes, see the [AWS documentation](https://aws.amazon.com/ec2/instance-types/). If you do not provide a setting, the default instance type `m5.large?` will be used.
- Node type: In EKS, K8S Workloads can run either as container on the ECS instances, which means container in VMs, or as containers on bare-metal server with Amazon Fargate.

In addition to these, you need to provide your AWS credentials, for example as the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.

### Part 4: Cluster Installation

#### First Try: Console

My first try to install a cluster was to pass all options as flags. This was the command:

```bash
eksctl create cluster \
  --region=eu-central-1 \
  --version=1.23 \
  --nodes=2 \
  --fargate
```

Log output:

```bash
2022-11-05 12:37:51 [ℹ]  eksctl version 0.117.0
2022-11-05 12:37:51 [ℹ]  using region eu-central-1
2022-11-05 12:37:51 [ℹ]  setting availability zones to [eu-central-1a eu-central-1b eu-central-1c]
2022-11-05 12:37:51 [ℹ]  subnets for eu-central-1a - public:192.168.0.0/19 private:192.168.96.0/19
2022-11-05 12:37:51 [ℹ]  subnets for eu-central-1b - public:192.168.32.0/19 private:192.168.128.0/19
2022-11-05 12:37:51 [ℹ]  subnets for eu-central-1c - public:192.168.64.0/19 private:192.168.160.0/19
2022-11-05 12:37:51 [ℹ]  nodegroup "ng-83b22df4" will use "" [AmazonLinux2/1.23]
2022-11-05 12:37:51 [ℹ]  using Kubernetes version 1.23
2022-11-05 12:37:51 [ℹ]  creating EKS cluster "scrumptious-hideout-1667648262" in "eu-central-1" region with managed nodes
2022-11-05 12:37:51 [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial managed nodegroup
2022-11-05 12:37:51 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=eu-central-1 --cluster=scrumptious-hideout-1667648262'
2022-11-05 12:37:51 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "scrumptious-hideout-1667648262" in "eu-central-1"
2022-11-05 12:37:51 [ℹ]  CloudWatch logging will not be enabled for cluster "scrumptious-hideout-1667648262" in "eu-central-1"
2022-11-05 12:37:51 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=eu-central-1 --cluster=scrumptious-hideout-1667648262'
2022-11-05 12:37:51 [ℹ]
2 sequential tasks: { create cluster control plane "scrumptious-hideout-1667648262",
    2 sequential sub-tasks: {
        wait for control plane to become ready,
        create managed nodegroup "ng-83b22df4",
    }
}
2022-11-05 12:37:51 [ℹ]  building cluster stack "eksctl-scrumptious-hideout-1667648262-cluster"
2022-11-05 12:37:52 [ℹ]  deploying stack "eksctl-scrumptious-hideout-1667648262-cluster"
```

My first attempt was not successful. The eksctl command was stuck with the `deploying stack` message as shown above. So, I ran the command again, but gave a dedicated name to the cluster:

```bash
eksctl create cluster \
  --name=staging \
  --region=eu-central-1 \
  --version=1.23 \
  --nodes=2 \
  --fargate
```

This resulted in the same message. And running it again showed an error that the cluster is available, but I could not access it. What’s happing here? Logging into the [AWS management console](https://eu-central-1.console.aws.amazon.com/), I saw that several security groups, VPS, network interfaces and elastic IPs of the cluster name were created, but no instances. Hmm.

The deletion of all these resources was a cumbersome, manual process, which can only be achieved in this order: First, detach all network interfaces, then delete the elastic IPS, then delete the network interfaces, and finally delete the VPCs.

### Second Try: Config File

The second try was to use a config file. You can either use a bare-bones configuration file, e.g. from the [eksctl.io documentation](https://eksctl.io/), or create one with the command `eksctl --dry-run`.  I used the latter and then customized the resulting file.

But take care! Running just `eksctl create cluster --dry-run` creates a very different file then a more customized `eksctl create cluster --name=staging --region=eu-central-1  --dry-run`. Use the first, bare command. For completions sake, here is the full config file with which the installation was successful.

```yaml
# eksctl create cluster --name=staging --dry-run

apiVersion: eksctl.io/v1alpha5
availabilityZones:
- eu-central-1c
- eu-central-1b
- eu-central-1a
cloudWatch:
  clusterLogging: {}
iam:
  vpcResourceControllerPolicy: true
  withOIDC: false
kind: ClusterConfig
kubernetesNetworkConfig:
  ipFamily: IPv4
managedNodeGroups:
- amiFamily: AmazonLinux2
  desiredCapacity: 2
  disableIMDSv1: false
  disablePodIMDS: false
  iam:
    withAddonPolicies:
      albIngress: false
      appMesh: false
      appMeshPreview: false
      autoScaler: false
      awsLoadBalancerController: false
      certManager: false
      cloudWatch: false
      ebs: false
      efs: false
      externalDNS: false
      fsx: false
      imageBuilder: false
      xRay: false
  instanceSelector: {}
  labels:
    alpha.eksctl.io/cluster-name: staging
    alpha.eksctl.io/nodegroup-name: ng-c3cdc337
  maxSize: 2
  minSize: 2
  name: ng-c3cdc337
  privateNetworking: false
  releaseVersion: ""
  securityGroups:
    withLocal: null
    withShared: null
  ssh:
    allow: false
    publicKeyPath: ""
  tags:
    alpha.eksctl.io/nodegroup-name: ng-c3cdc337
    alpha.eksctl.io/nodegroup-type: managed
  volumeIOPS: 3000
  volumeSize: 80
  volumeThroughput: 125
  volumeType: gp3
metadata:
  name: staging
  region: eu-central-1
  version: "1.23"
privateCluster:
  enabled: false
  skipEndpointCreation: false
vpc:
  autoAllocateIPv6: false
  cidr: 192.168.0.0/16
  clusterEndpoints:
    privateAccess: false
    publicAccess: true
  manageSharedNodeSecurityGroupRules: true
  nat:
    gateway: Single
```

In this file, I merely changed the cluster name to `eksctl-superb-hedgehog` and modified `managedNodeGroups.ssh` to obtain SSH access to the nodes:

```yaml
managedNodeGroups:
  - amiFamily: AmazonLinux2
    #...
    ssh:
      allow: true
      publicKeyPath: "~/.ssh/eksctl_rsa.key"
```

The cluster creation took a long time - searching the internet about eksctl installations hanging, I found this [stackoverflow thread](https://stackoverflow.com/questions/65756827/eksctl-create-cluster-stuck-waiting-for-cloudformation-stack): cluster creation can take up to 20 minutes!

And indeed, it took about this time:

```bash
2022-11-07 20:08:36 [ℹ]  eksctl version 0.117.0
2022-11-07 20:08:36 [ℹ]  using region eu-central-1
2022-11-07 20:08:36 [ℹ]  subnets for eu-central-1b - public:192.168.0.0/19 private:192.168.96.0/19
2022-11-07 20:08:36 [ℹ]  subnets for eu-central-1a - public:192.168.32.0/19 private:192.168.128.0/19
2022-11-07 20:08:36 [ℹ]  subnets for eu-central-1c - public:192.168.64.0/19 private:192.168.160.0/19
2022-11-07 20:08:36 [ℹ]  nodegroup "ng-9bf41c12" will use "" [AmazonLinux2/1.23]
2022-11-07 20:08:36 [ℹ]  using Kubernetes version 1.23
2022-11-07 20:08:36 [ℹ]  creating EKS cluster "superb-hedgehog-staging" in "eu-central-1" region with managed nodes
2022-11-07 20:08:36 [ℹ]  1 nodegroup (ng-9bf41c12) was included (based on the include/exclude rules)
2022-11-07 20:08:36 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
2022-11-07 20:08:36 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
# ....
2022-11-07 20:08:37 [ℹ]  deploying stack "eksctl-superb-hedgehog-staging-cluster"
2022-11-07 20:09:07 [ℹ]  waiting for CloudFormation stack "eksctl-superb-hedgehog-staging-cluster"
# ...
2022-11-07 20:26:49 [ℹ]  waiting for CloudFormation stack "eksctl-superb-hedgehog-staging-nodegroup-ng-9bf41c12"
2022-11-07 20:26:49 [ℹ]  waiting for the control plane to become ready
2022-11-07 20:26:51 [✔]  saved kubeconfig as "~.kube/config"
# ...
2022-11-07 20:26:53 [✔]  EKS cluster "superb-hedgehog-staging" in "eu-central-1" region is ready
```

Looking into the AWS management console, I could see the created instances and volumes:

![](https://admantium.com/images/blog/kube15_eksctl_instances.png)

![](https://admantium.com/images/blog/kube15_eksctl_volumes.png)

## Part 5: Cluster Access and Maintenance

With the installation completed, a custom kubeconfig file is automatically placed at `~/.kube/config`. Use it to gain cluster access:

```bash
export KUBECONFIG=~/.kube/config

kubectl  get nodes

NAME                                              STATUS   ROLES    AGE   VERSION
ip-192-168-34-211.eu-central-1.compute.internal   Ready    <none>   12m   v1.23.9-eks-ba74326
ip-192-168-73-218.eu-central-1.compute.internal   Ready    <none>   11m   v1.23.9-eks-ba7432
```

For the SSH access, you need to get the nodes' public DNS name from the AWS management console, and then connect as the `ec2-user`.

```bash
ssh ec2-user@ec2-3-70-174-13.eu-central-1.compute.amazonaws.com
Last login: Sat Nov  5 02:57:35 2022 from 205.251.233.105

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
12 package(s) needed for security, out of 18 available
Run "sudo yum update" to apply all updates.

[ec2-user@ip-192-168-69-138 ~]$ kubelet --version
Kubernetes v1.23.9-eks-ba74326
```

## Part 6: Cluster Updates

For checking how cluster updates work, I completely reinstalled a new cluster and used Kubernetes version v1.22.

As before, the cluster creation took some time to complete.

```bash
2022-11-09 20:11:11 [ℹ]  eksctl version 0.117.0
2022-11-09 20:11:11 [ℹ]  using region eu-central-1
2022-11-09 20:11:11 [ℹ]  subnets for eu-central-1b - public:192.168.0.0/19 private:192.168.96.0/19
#...
22-11-09 20:28:59 [ℹ]  node "ip-192-168-69-138.eu-central-1.compute.internal" is ready
2022-11-09 20:29:01 [ℹ]  kubectl command should work with "/Users/guenthers/.kube/config", try 'kubectl get nodes'
2022-11-09 20:29:01 [✔]  EKS cluster "staging" in "eu-central-1" region is ready
```

The upgrade happens gradually. You need to choose the new Kubernetes version, and then pass it to upgrade commands. The upgrade commands targets are: control plane node, kube-proxy, aws-nodes, and finally coredns.

Here are the concrete update commands and their results:

```bash
eksctl upgrade cluster --name=staging --version=1.23
2022-11-09 20:47:44 [ℹ]  re-building cluster stack "eksctl-staging-cluster"
2022-11-09 20:47:44 [✔]  all resources in cluster stack "eksctl-staging-cluster" are up-to-date
2022-11-09 20:47:44 [ℹ]  checking security group configuration for all nodegroups
2022-11-09 20:47:44 [ℹ]  all nodegroups have up-to-date cloudformation templates

eksctl utils update-kube-proxy --cluster=staging --approve
2022-11-09 20:48:53 [ℹ]  "kube-proxy" is now up-to-date

eksctl utils update-aws-node --cluster=staging --approve
2022-11-09 20:49:21 [ℹ]  skipped existing "kube-system:ServiceAccount/aws-node"
2022-11-09 20:49:21 [ℹ]  replaced "CustomResourceDefinition.apiextensions.k8s.io/eniconfigs.crd.k8s.amazonaws.com"
2022-11-09 20:49:21 [ℹ]  replaced "ClusterRole.rbac.authorization.k8s.io/aws-node"
2022-11-09 20:49:22 [ℹ]  replaced "ClusterRoleBinding.rbac.authorization.k8s.io/aws-node"
2022-11-09 20:49:22 [ℹ]  replaced "kube-system:DaemonSet.apps/aws-node"
2022-11-09 20:49:22 [ℹ]  "aws-node" is now up-to-date

eksctl utils update-coredns --cluster=staging --approve
2022-11-09 20:49:47 [ℹ]  replaced "kube-system:Service/kube-dns"
2022-11-09 20:49:47 [ℹ]  replaced "kube-system:ServiceAccount/coredns"
2022-11-09 20:49:47 [ℹ]  replaced "kube-system:ConfigMap/coredns"
2022-11-09 20:49:47 [ℹ]  replaced "kube-system:Deployment.apps/coredns"
2022-11-09 20:49:48 [ℹ]  replaced "ClusterRole.rbac.authorization.k8s.io/system:coredns"
2022-11-09 20:49:48 [ℹ]  replaced "ClusterRoleBinding.rbac.authorization.k8s.io/system:coredns"
2022-11-09 20:49:48 [ℹ]  "coredns" is now up-to-date
```

I encountered no error, all deployments/daemonsets are running. To check the results, I connected via SSH to the nodes took the `kubectl` version. Also, you can see that image tags of certain deployments or daemonsets reflect the new Kubernetes version:

```bash
kubectl describe daemonset kube-proxy -n kube-system
Name:           kube-proxy
#...
  Containers:
   kube-proxy:
    Image:      602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/kube-proxy:v1.23.13-eksbuild.1
```

## Part 7: EKS Cluster Internals

I was curious to find out which Kubernetes components the cluster uses. Running `kubectl get all -A` revealed these facts:

- Linux distribution: amazon-eks-ami
- Container runtime: containerd
- Network communication: coredns, kube-proxy, amazon-vpc-cni
- Ingress: aws-load-balancer-controller

## Conclusion

Amazon EKS is a managed Kubernetes distribution. In this article, you learned all the essentials to get you started: a) Ensure to have a fully registered AWS account, b) install the required binaries `eksctl` and `kubectl`, c) create the cluster by issuing a dry-run installation, then customize the resulting configuration file. Cluster installation can take up to 20 minutes - don’t stop this in the middle, or you will need to manually cleanup several AWS resources. Once completed, you can connect via `kubectl` and SSH to the nodes. Finally, you also learned the required steps for upgrading the Kubernetes version of the cluster, and saw some cluster internals.