Ultimate EKS Baseline Cluster: Part 2 - Storage
Tags: awscloud, kubernetes, devops, tutorial
Published: 2023-07-24 00:56:03

This may come as a surprise to some, but the AWS managed Kubernetes service, [EKS](https://aws.amazon.com/eks/) ([Elastic Kubernetes Service](https://aws.amazon.com/eks/)), no longer comes with support for storage through Kubernetes [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/), which is required for many applications, such as databases.

You will need to add a ([CSI](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/)) driver to enable [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) support.

This guide, Part 2 of *Ultimate EKS Baseline Cluster* series,  covers doing just that by installing [CSI](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/)) called [AWS EBS CSI Driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver).

## 0. Prerequisites

These are are some prerequisites and initial steps needed to get started before provisioning a Kubernetes cluster and installing add-ons.

### 0.1 Knowledge: Systems

![image of system concept icons](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aqzshuffp4kni7i9t487.png)

Basic concepts of systems, such as Linux and the shell (redirection, pipes, process substitution, command substitution, environment variables), as well as virtualization and containers are useful. The concept of a service (daemon) is important.

### 0.2 Knowledge: Kubernetes

![image of Kubernetes concept icons](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o5i7xyp8oty9jp44r0dv.png)

In Kubernetes, familiarity with service types: ClusterIP, NodePort, LoadBalancer, ExternalName and the ingress resource are important.

Exposure to other types of Kubernetes resource objects used in this guide are helpful: `persistentvolumeclaims` (`pvc`), `storageclass` (`sc`), `pods`, `deployments`, `statefulsets` (`sts`), `configmaps`, `serviceaccount` (`sa`) and `networkpolicies`.

### 0.3 Tools

![image of tool icons](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7u87et62nyhbwrksb29d.png)

These are the tools used in this article series:

* [AWS CLI](https://aws.amazon.com/cli/) [`aws`] is a tool that interacts with AWS.
* [kubectl client](https://kubernetes.io/docs/reference/kubectl/) [`kubectl`] a the tool that can interact with the Kubernetes cluster. This can be installed using adsf tool.
* [eksctl](https://eksctl.io/) [eksctl] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.
* [POSIX Shell](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html) [`sh`] such as [bash](https://www.gnu.org/software/bash/)[`bash`] or [zsh](https://www.zsh.org/) [`zsh`] are used to run the commands.

These tools are highly recommended:

* [adsf](https://asdf-vm.com/) [`adsf`] is a tool that installs versions of popular tools like kubectl.
* [jq](https://jqlang.github.io/jq/) [`jq`] is a tool to query and print JSON data
* [GNU Grep](https://www.gnu.org/software/grep/) [`grep`] supports extracting string patterns using extended [Regex](https://en.wikipedia.org/wiki/Regular_expression) and [PCRE](https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions).

### 0.3 Setup Environment Variables

These environment variables will be used throughout this guide. If opening up a new browser tab, make sure to set the environment variables accordingly.

```bash
# variables used to create EKS
export AWS_PROFILE="my-aws-profile" # CHANGEME
export EKS_CLUSTER_NAME="my-unique-cluster-name" # CHANGEME
export EKS_REGION="us-west-2"
export EKS_VERSION="1.26"

# KUBECONFIG variable
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# account id
export ACCOUNT_ID=$(aws sts get-caller-identity \
  --query "Account" \
  --output text
)

# ebs-csi-driver
export ROLE_NAME_ECSI="${EKS_CLUSTER_NAME}_EBS_CSI_DriverRole"
export ACCOUNT_ROLE_ARN_ECSI="arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME_ECSI"
POLICY_NAME_ECSI="AmazonEBSCSIDriverPolicy" # preinstalled by AWS
export POLICY_ARN_ECSI="arn:aws:iam::aws:policy/service-role/$POLICY_NAME_ECSI"

```

### 0.4 AWS Setup

There's an assumption that AWS CLI have been setup and configured with a profile. This is required before creating or interacting with an EKS cluster.

You can test access to a configured profile with the following command:

```bash
export AWS_PROFILE="<your-profile-goes-here>"
aws sts get-caller-identity
```

This should show something like the following with values appropriate to your environment, e.g. example output to IAM user named `kwisatzhaderach`:

```json
{
    "UserId": "AIDAXXXXXXXXXXXXXXXXX",
    "Account": "XXXXXXXXXXXX",
    "Arn": "arn:aws:iam::XXXXXXXXXXXX:user/kwisatzhaderach"
}
```

### 0.5 EKS Cluster

This article requires that an EKS cluster has been previously provisioned using `eksctl` tool.

I wrote a previous article that covered this how to provision EKS with two commands:

* [Ultimate EKS Baseline Cluster: Part 1 - Provision EKS ](https://dev.to/joachim8675309/ultimate-eks-baseline-cluster-part-1-provision-eks-17f)

#### 0.5.1 Existing EKS Cluster

If you have an existing EKS cluster, but need to configure `KUBECONFIG` for access, you can run this:

```bash
mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# update config pointed to by $KUBECONFIG 
aws eks update-kubeconfig \
  --name $CLUSTER \
  --region $REGION \
  --profile $PROFILE
```

#### 0.5.2 Create a new EKS Cluster

If you have not setup an EKS cluster, you can set it up with the following commands (~20 minutes process):

```bash
mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# provision EKS + add config to $KUBECONFIG 
eksctl create cluster \
  --version $EKS_VERSION \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME \
  --nodes 3

# setup OIDC provider for least privilege
eksctl utils associate-iam-oidc-provider \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --approve
```

Details of this were covered in the [previous article](https://dev.to/joachim8675309/ultimate-eks-baseline-cluster-part-1-provision-eks-17f)

### 0.6 Kubernetes Client Setup

If you use `asdf` to install `kubectl`, you can get the latest version with the following:

```bash
# install kubectl plugin for asdf
asdf plugin-add kubectl \
  https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl latest

# fetch latest kubectl 
asdf install kubectl latest
asdf global kubectl latest

# test results of latest kubectl 
kubectl version --short --client 2> /dev/null
```

This should show something like:

```yaml
Client Version: v1.27.4
Kustomize Version: v5.0.1
```

## 1. AWS EBS CSI driver

![Title bar with icons for EBS and CSI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bug7a5d2vpxzirfr6hs7.png)

The current versions of [EKS](https://aws.amazon.com/eks/) starting with 1.23 no longer come with a persistent volume support, so you have to install it on your own. The best method or at least the easiest way to install this, is using [EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) facility. This will install the [EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html).

Installation of this component will require the following steps:

1. Create [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) (e.g. `EBS_CSI_DriverRole`) and associate it to [Kubernetes service account](https://kubernetes.io/docs/concepts/security/service-accounts/) (i.e. `ebs-csi-controller-sa`).
2. Deploy AWS EBS CSI driver using EKS add-ons facility, which also sets up the [Kubernetes service account](https://kubernetes.io/docs/concepts/security/service-accounts/) (i.e. `ebs-csi-controller-sa`) with an association back to the above [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) (e.g. `EBS_CSI_DriverRole`).
3. Create storage class that uses new the [EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html)

### 1.1 Setup IAM Role and K8S SA association

The following process will create an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) with permissions to access AWS EBS API. The service account `ebs-csi-controller-sa` will be created later when installing the driver.


```bash
# AWS IAM role bound to a Kubernetes service account
eksctl create iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --attach-policy-arn $POLICY_ARN_ECSI \
  --role-only \
  --role-name $ROLE_NAME_ECSI \
  --approve
```
This will create a [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html), which you can verify with:

```bash
aws iam get-role --role-name $ROLE_NAME_ECSI
```

This should show something like this:

```json
{
    "Role": {
        "Path": "/",
        "RoleName": "mycluster_EBS_CSI_DriverRole",
        "RoleId": "AROAZYKZFDW7YZGW3Q5S7",
        "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/mycluster_EBS_CSI_DriverRole",
        "CreateDate": "2023-07-07T21:19:32+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:aud": "sts.amazonaws.com",
                            "oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
                        }
                    }
                }
            ]
        },
        "Description": "",
        "MaxSessionDuration": 3600,
        "Tags": [
            {
                "Key": "alpha.eksctl.io/cluster-name",
                "Value": "mycluster"
            },
            {
                "Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
                "Value": "mycluster"
            },
            {
                "Key": "alpha.eksctl.io/iamserviceaccount-name",
                "Value": "kube-system/ebs-csi-controller-sa"
            },
            {
                "Key": "alpha.eksctl.io/eksctl-version",
                "Value": "0.141.0-dev+5c8318ed5.2023-05-12T11:33:48Z"
            }
        ],
        "RoleLastUsed": {}
    }
}
```



### 1.2 Install AWS EBS CSI Drvier

This installation uses the [EKS Addons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) feature to install the component. 

```bash
# Install Addon
eksctl create addon \
  --name "aws-ebs-csi-driver" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --service-account-role-arn $ACCOUNT_ROLE_ARN_ECSI \
  --force

# Pause here until driver is active
ACTIVE=""; while [[ -z "$ACTIVE" ]]; do
  if eksctl get addon \
       --name "aws-ebs-csi-driver" \
       --region $EKS_REGION \
       --cluster $EKS_CLUSTER_NAME \
    | tail -1 \
    | awk '{print $3}' \
    | grep -q "ACTIVE"
  then
    ACTIVE="1"
  fi
done
```

It is important to wait until status changes to `ACTIVE` before proceeding.

You can inspect the pods created by running the following command:

```bash
kubectl get pods \
  --namespace "kube-system" \
  --selector "app.kubernetes.io/name=aws-ebs-csi-driver"
```

This should show something like:

```fwf
NAME                                  READY   STATUS    RESTARTS   AGE
ebs-csi-controller-6d5b7bfd56-bwr5x   6/6     Running   0          2m1s
ebs-csi-controller-6d5b7bfd56-wtxf6   6/6     Running   0          2m2s
ebs-csi-node-hjmf5                    3/3     Running   0          2m2s
ebs-csi-node-tzpgs                    3/3     Running   0          2m2s
```

You can verify the service account annotations references the IAM Role for the EBS CSI driver.

```bash
kubectl get serviceaccount "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --output yaml
```

This should show something like the following

```yaml
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/my-unique-cluster-name_EBS_CSI_DriverRole
  creationTimestamp: "2023-07-24T21:12:02Z"
  labels:
    app.kubernetes.io/component: csi-driver
    app.kubernetes.io/managed-by: EKS
    app.kubernetes.io/name: aws-ebs-csi-driver
    app.kubernetes.io/version: 1.21.0
  name: ebs-csi-controller-sa
  namespace: kube-system
  resourceVersion: "1922"
  uid: 1b8ecfdb-ce64-491c-8ced-e08b5519755c
```


#### 1.2.1 Sidebar: eks-addons vs helm chart?

The CSI driver can be installed using either [EKS Addons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) facility from the previous step, or using the [aws-ebs-csi-driver](https://artifacthub.io/packages/helm/aws-ebs-csi-driver/aws-ebs-csi-driver) Helm chart. I prefer the EKS addons because of simplicity, but also because it installs an extra snapshot container that doesn't come by default with the helm chart. 

I documented the full process using the helm chart in this article:

* [EKS + EBS Storage with eksctl](https://medium.com/@joachim8675309/eks-ebs-storage-with-eksctl-3e526f534215)

### 1.3 Create storage class that uses the EBS CSI driver

In order to use the driver, we will need to create a storage class. You can do so by running the following command:

```bash
# create ebs-sc storage class
cat <<EOF | kubectl apply --filename -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF
```

When completed you can verify the storage class was created with:

```bash
kubectl get storageclass
```

This should show something like this:

```fwf
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-sc          ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   17s
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  30m
```

### 1.4 Set new storage class to the default (optional)

This is an optional step. As there’s no functional default storage class, we can set the newly created storage class to be the default with the following commands:

```bash
kubectl patch storageclass gp2 --patch \
 '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass ebs-sc --patch \
 '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```

After this, you can verify the change with:

```bash
kubectl get storageclass
```

This should show something like:

```fwf
NAME               PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-sc (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   31s
gp2                kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  30m
```

### 1.5 Testing Persistent Volume

![Image for title bar with icon for pvc](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k7uochrbvzgnd47q5zts.png)

In this small test, we deploy a pod that continually writes to the external volume, and a volume claim to allocate storage using the storage class we created earlier.

If this works, the storage will be provisioned in the cloud to create the volume, and then it will be attached to the node and mounted in the pod. If this fails, you will see that the pod will be stuck in `pending` mode.

```bash
# create pod with persistent volume
kubectl create namespace "ebs-test"

# deploy application with mounted volume
cat <<-'EOF' | kubectl apply --namespace "ebs-test" --filename -
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: ubuntu
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
      volumeMounts:
      - name: persistent-storage
        mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: ebs-claim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 4Gi
EOF
```

You can test the results of the volume creation with the following command:

```bash
kubectl get all,pvc --namespace "ebs-test"
```

We can also look at the events that took place in this namespace with:

```bash
kubectl events --namespace "ebs-test"
```

This will show something like this: 

```fwf
LAST SEEN           TYPE      REASON                   OBJECT                            MESSAGE
56s                 Warning   FailedScheduling         Pod/app                           0/3 nodes are available: persistentvolumeclaim "ebs-claim" not found. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..
55s                 Normal    WaitForPodScheduled      PersistentVolumeClaim/ebs-claim   waiting for pod app to be scheduled
54s                 Normal    Provisioning             PersistentVolumeClaim/ebs-claim   External provisioner is provisioning volume for claim "ebs-test/ebs-claim"
54s (x2 over 54s)   Normal    ExternalProvisioning     PersistentVolumeClaim/ebs-claim   waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
50s                 Normal    Scheduled                Pod/app                           Successfully assigned ebs-test/app to ip-192-168-22-207.us-west-2.compute.internal
50s                 Normal    ProvisioningSucceeded    PersistentVolumeClaim/ebs-claim   Successfully provisioned volume pvc-39b9cf94-8b35-436c-b56d-e2fa587245ee
48s                 Normal    SuccessfulAttachVolume   Pod/app                           AttachVolume.Attach succeeded for volume "pvc-39b9cf94-8b35-436c-b56d-e2fa587245ee"
44s                 Normal    Pulling                  Pod/app                           Pulling image "ubuntu"
41s                 Normal    Pulled                   Pod/app                           Successfully pulled image "ubuntu" in 2.952872938s (2.952907339s including waiting)
41s                 Normal    Created                  Pod/app                           Created container app
41s                 Normal    Started                  Pod/app                           Started container app
```


### 1.6 Delete test application

```bash
kubectl delete pod app --namespace "ebs-test"
kubectl delete pvc ebs-claim --namespace "ebs-test"
kubectl delete ns "ebs-test"
```

## 2.0 Deleting EKS

When deleting up the EKS cluster, you may want to run through these steps.

### 2.1 Deleting persistent volume claims

When deleting EKS cluster, if you did not delete persistent volumes, you will have left over unused EBS volumes eating costs.

You should delete all the persistent volume claims, which will delete associate persistent volumes.  You can list all of the `persistentvolumeclaim` resources with the following command:

```bash
kubectl get persistentvolumeclaim \
  --all-namespaces | grep -v none
```

Not that some of these will not be deleted if there is an application running that has a lock to the storage.  So you will need to delete the associated application as well.

### 2.2 Reset Default to original Storage Class

As a precaution, we don’t want to have any resources locked that may prevent deletion of the Kubernetes cluster. Run this command if we changed the defaults earlier.

```bash
kubectl patch storageclass ebs-sc --patch \
  '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass gp2 --patch \
  '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```

### 2.3 IAM Roles

These should be removed when removing Kubernetes with `eksctl`, but it is a good practice to remove them just in case.

```bash
eksctl delete iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION
```

### 2.4 Kubernetes cluster

Finally, the Kubernetes cluster itself can be deleted.

```bash
eksctl delete cluster \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME
```

## Conclusion

This second article shows add support for storage, called persistent volumes on Kubernetes.  

The automation using the `eksctl` will do the following additional things besides provisioning EKS:

* setup restricted access to AWS cloud resources using [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html), which associates [KSA](https://kubernetes.io/docs/concepts/security/service-accounts/) with [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) using an [OIDC](https://openid.net/developers/how-connect-works/) provider.
* install applications using [EKS addons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) facility, specifically the EBS CSI driver. 

In future articles, I will cover how to add support for load balancers and network policies, as well as a demo application that demonstrates all of these features.