9 Ways to Spin Up an EKS Cluster - Way 3 - eksctl
Tags: kubernetes, eks, iac
Published: 2024-06-27 11:00:29

---
title: 9 Ways to Spin Up an EKS Cluster - Way 3 - eksctl
published: true
description: 
tags: kubernetes, eks, IaC
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vvbl36a0fvbfhnaf1ekf.png
# Use a ratio of 100:42 for best results.
# published_at: 2024-03-03 18:26 +0000
---

In my [previous post](https://dev.to/aws-builders/9-ways-to-an-eks-cluster-way-2-aws-cli-3g94) I showed how to spin up an EKS cluster with pure shell and AWS CLI. (All the links to other posts in this series will be [here](https://dev.to/aws-builders/8-ways-to-spin-up-an-eks-cluster-210b))

This used to be the easiest way of getting to a cluster without leaving your terminal. But pretty early in EKS history (2017) some smart folks from a company named Weaveworks(RIP) realized it was too cumbersome to do this using the `aws cli` subcommand and that EKS is complex enough to deserve a command-line client of its own.  That's how `eksctl` was born. 

A few months ago Weaveworks (who brought us a plethora of great OSS tools like Flux, Flagger and Weave) was shut down. But AWS announced full support for eksctl in 2019 - so `eksctl` is now the de-facto standard EKS CLI tool.

The great thing about `eksctl` is that it allows one to create and manage clusters not only using one-off commands with arguments but also with YAML configuration files - in a true and familiar IaC way.

We'll check out both options but first let's install eksctl and generate an SSH key so we can connect to the nodes in the clusters we create if needed. Please note - I'm not endorsing SSH connections to your EKS nodes. Do avoid this if possible - so as not to cause inadvertent configuration drift. But sometimes we still need this for troubleshooting, especially in training environments. So let's have the SSH key handy.

## Install eksctl
If you're on Linux - here are the official instructions:
```bash
# for ARM systems, set ARCH to: `arm64`, `armv6` or `armv7`
ARCH=amd64
PLATFORM=$(uname -s)_$ARCH

curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz"

# (Optional) Verify checksum
curl -sL "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_checksums.txt" | grep $PLATFORM | sha256sum --check

tar -xzf eksctl_$PLATFORM.tar.gz -C /tmp && rm eksctl_$PLATFORM.tar.gz

sudo mv /tmp/eksctl /usr/local/bin
```
Please note this doesn't install such eksctl prerequisites as `kubectl` and `aws-iam-authenticator`.

And if, like me -  you're on a Mac - definitely use `brew` as it takes care of all dependencies. (even though the official `eksctl` docs don't recommend it)
```
brew tap weaveworks/tap
brew install weaveworks/tap/eksctl
```
And now - let's generate that ssh key:
```
ssh-keygen  -f ./id_rsa -N ''
```
This will create an `id_rsa` and `id_rsa.pub` in your current directory. Make sure to run the following `eksctl` commands from the same directory and it will pick up this key by default.

### Sidenote - the VPC
If you've read the previous post in this series (where we created an EKS cluster using the AWS CLI), you'd notice that creating the VPC was a separate step. The added value of `eksctl` is it takes care of most dependencies and add-ons for us without the need of running additional commands. The same is true for VPC creation. A new VPC with default subnet configuration is created for us each time we spin up a new cluster, unless we specifically define we want to re-use an existing VPC.

### 1. Create an EKS cluster - eksctl with arguments

The most straightforward way of creating an EKS cluster with `eksctl` is providing all the arguments on the command-line and letting the tool take care of the defaults. This approach, while limited and not repeatable enough can definitely give us a cluster. 

The command I provide here defines quite a number of settings I personally find important even for small toy clusters I spin up for fun and games. But `eksctl` can do its job even with less stuff defined. Look in the official "Getting Started" docs if you want just the bare bones.

So here's what I decided to use:

```bash
# First - define the environment. 
export CLUSTER_NAME=way3
export AWS_REGION=eu-central-1
export K8S_VERSION=1.30
export NODE_TYPE=t2.medium
export MIN_NODES=1
export MAX_NODES=3
```

I'm starting out with small nodes and already preparing the cluster for auto-scaling with min and max nodes definitions. It's important to note that `eksctl` allows us to enable the IAM policy for ASG acces and define the auto-scaling range. But it doesn't take care of installing `cluster-autoscaler`. We'd need to do that separately. If we wanted to... On the other hand - these days it makes total sense to start out with Karpenter. For which `eksctl` does provide support, but not on the command line. whcih means we'll see how to configure Karpenter in the next section.

And now - time to spin up the cluster:
```bash
eksctl create cluster --name $CLUSTER_NAME \
                      --region $AWS_REGION \
                      --with-oidc --version $K8S_VERSION \
                      --nodegroup-name ng-$CLUSTER_NAME-1 \
                      --node-type t2.medium \
                      --nodes 1 --nodes-min 1 --nodes-max 3 \ 
                      --spot \
                      --ssh-access \
                      --asg-access \
                      --external-dns-access \
                      --full-ecr-access \
                      --alb-ingress-access
```

This command gives us a full-featured cluster with IAM policies for ECR access (`--full-ecr-access`), external dns controller (`--external-dns-access`) , ALB ingress controller (`--alb-ingress-access`), OIDC support and more. It also runs its nodes on spot instances for cost optimization. Which is totally fine for a toy cluster but may be not appropriate if the application you're planning to deploy isn't disruption-tolerant.

From the command output we learn that in the background our command is converted into a couple of CloudFormation stacks:
```

2024-06-27 12:51:47 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
2024-06-27 12:51:47 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
```

After about 15 minutes (depending on the weather and the region you've decided to use) CloudFormation returns and we can access our cluster:
```bash
kubectl get node
NAME                                             STATUS   ROLES    AGE   VERSION
ip-192-168-56-76.eu-central-1.compute.internal   Ready    <none>   35m   v1.29.3-eks-ae9a62a
```

Note that the new cluster context is added to your `kubeconfig` automatically. 
If you want to update the `kubeconfig` at a later time you can use:
```bash
eksctl utils write-kubeconfig -c $CLUSTER_NAME -r $AWS_REGION
```

But, as we already said - the CLI approach is limited. To do real IaC  we want to put the cluster definitions in a YAML config file. This gives us a lot more capabilities, and allows to commit the config file to source control for further collaboration, change tracking and  automation.

But first - let's remove the cluster we just created:
```
eksctl delete cluster --region=$AWS_REGION --name=$CLUSTER_NAME
```

### 2. Create an EKS cluster - eksctl with a config file.

The config file I provide here gives us everything we defined at the command line and more. As mentioned - it also allows us to install Karpenter in the same `eksctl` execution - thus giving us an industry-standard auto-scaling EKS cluster with just-in-time node provisioning. You can grab this file in [Github](https://github.com/antweiss/9-ways-2-EKS/tree/main/way-3-eksctl) too.

```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: way3
  region: eu-central-1
  version: "1.30"
  tags:
    karpenter.sh/discovery: way3
iam:
  withOIDC: true

managedNodeGroups:
  - name: ng-way3-1
    labels: { role: worker }
    instanceType: t2.medium
    desiredCapacity: 2
    minSize: 1
    maxSize: 3
    tags:
      nodegrouprole: way3
    volumeSize: 20
    iam:
      withAddonPolicies:
        externalDNS: true
        certManager: true
        awsLoadBalancerController: true
        albIngress: true
        ebs: true
        efs: true
        imageBuilder: true
        cloudWatch: true
    ssh:
      allow: true # will use ~/.ssh/id_rsa.pub as the default ssh key

karpenter:
  version: '0.37.0'
  createServiceAccount: true
  withSpotInterruptionQueue: true
```

An attentive eye will also notice I've also defined some additional stuff such as [CloudWatch logging of the control plane](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html),  EBS and EFS access. Consider removing these lines if you don't need them. 
Also you'll notice that not only it installs Karpenter, it also takes care of setting up the SpotInterruptionQueue, which allows Karpenter to replace spot instances before they die.
And there are many additional options available.
So yes - this is a very scalable approach, which takes care of more or less everything one might need in an EKS cluster.

Execute this plan with:
```
eksctl create cluster -f cluster.yaml
```

This again creates a CloudFormation execution that, granted we have all the necessary permissions, should complete successfully.

Let's check that Karpenter got installed:
```bash
kubectl get pod -A
NAMESPACE     NAME                         READY   STATUS    RESTARTS   AGE
karpenter     karpenter-79db484bbf-flzzq   1/1     Running   0          32s
karpenter     karpenter-79db484bbf-nfhsp   1/1     Running   0          32s
kube-system   aws-node-8h4ln               2/2     Running   0          17m
kube-system   aws-node-vq8wj               2/2     Running   0          18m
kube-system   coredns-6f6d89bcc9-qx497     1/1     Running   0          24m
kube-system   coredns-6f6d89bcc9-wwjtp     1/1     Running   0          24m
kube-system   kube-proxy-8mnd2             1/1     Running   0          18m
kube-system   kube-proxy-c5zkp             1/1     Running   0          17m
```

Yup, here it is! 

The upside of using the config file is of course the ability to manage stuff in a somewhat idempotent way. So for example if we want to change our node group config - we can update the following lines:

```yaml
- name: ng-1
    labels: { role: worker }
    instanceType: t2.medium
    desiredCapacity: 1
    minSize: 1
    maxSize: 5
```

and then run `eksctl update nodegroup -f cluster.yaml` - this will update our NodeGroup autoscaling range.

And of course eksctl provides us with a plethora of addtional commands that come very handy for ongoing management of EKS clusters:

```bash
eksctl -h
The official CLI for Amazon EKS

Usage: eksctl [command] [flags]

Commands:
  eksctl anywhere                        EKS anywhere
  eksctl associate                       Associate resources with a cluster
  eksctl completion                      Generates shell completion scripts for bash, zsh or fish
  eksctl create                          Create resource(s)
  eksctl delete                          Delete resource(s)
  eksctl deregister                      Deregister a non-EKS cluster
  eksctl disassociate                    Disassociate resources from a cluster
  eksctl drain                           Drain resource(s)
  eksctl enable                          Enable features in a cluster
  eksctl get                             Get resource(s)
  eksctl help                            Help about any command
  eksctl info                            Output the version of eksctl, kubectl and OS info
  eksctl register                        Register a non-EKS cluster
  eksctl scale                           Scale resources(s)
  eksctl set                             Set values
  eksctl unset                           Unset values
  eksctl update                          Update resource(s)
  eksctl upgrade                         Upgrade resource(s)
  eksctl utils                           Various utils
  eksctl version                         Output the version of eksctl
```

All in all - eksctl is the go to tool for EKS management if you haven't already standardized your cloud platform on another IaC solution such as Terraform, Pulumi, CDK or others which we'll look into in the folowing posts.

Thanks for reading and may your clusters be lean!


P.S. now you got a cluster - why not start managing its cost and performance for free with [PerfectScale](https://perfectscale.io) - the leading Kubernetes cost optimization solution? 

Join now to build clusters you can be proud of: https://perfectscale.io.