Dynamic Volume Provisioning in Kubernetes with AWS and Terraform
Tags: devops, kubernetes, terraform, aws
Published: 2023-04-29 16:54:08

## Introduction
In this article, we will be dealing with understanding Persistent Volumes in Kubernetes, how they are provisioned, how they are managed, and how they are configured in AWS. We will do a comparative analysis between Static Volume provisioning and Dynamic volume provisioning, how they overlap, and which provisioning mechanism you should employ. Finally, we will provision a Kubernetes cluster in AWS using Terraform and then setup dynamic provisioning with the cluster, we will test the validity of our setup by attempting to install a helm chart that will require a reasonable amount of storage, at first we will fail due to a lack of sufficient storage, and then after sufficient setup, we will succeed.  This article will require a basic understanding of Kubernetes, helm, and AWS. You can find all code examples [here](https://github.com/Monarene/dynamic-volume-k8s).


## Dynamic vs Static provisioning
Understanding when to use dynamic or static volume  provisioning is very vital depending on the type of application you are trying to build.  If you want to pre-populate data in a volume, then you choose static provisioning however, if you want to create volumes on demand and then delete or retain after a successful run, then you should make use of Dynamic provisioning. Another difference is that in the Static provisioning workflow, you have to explicitly define and provision a PersistentVolume while in Dynamic  provisioning, Kubernetes handles the process of creating a PersistentVolume and binding that Persistent volume to the necessary components. 

Again, The major difference is having to explicitly create a PersistentVolume, as against allowing Kubernetes to create a PersistentVolume whenever it needs it. Additionally, dynamic provisioning abstracts the implementation details from your workloads, this means you can switch storage providers depending on the workload type; whether you are running on-premise, on the public cloud, or hybrid. Another important detail to note in Dynamic provisioning is that when the PersistentVolumeClaim is deleted, the volume is also deleted. However, this setting can be adjusted even after the Persistent Volume is created.

## Dynamic Provisioning in AWS 
In this article, I will assume you have a properly setup  Kubernetes cluster however, if you do not have that running I have provided a Terraform project to help you quickly set up a Kubernetes cluster in AWS [here](https://github.com/Monarene/dynamic-volume-k8s/tree/master/eks), all you have to do is set up the right variables and you are good to go. Finally, ensure that your IAM user has the right permissions to set up a Kubernetes cluster as that is very key. 

In order to demonstrate volume provisioning, we will attempt to install the Drupal chart on the bitnami helm repo. I am also assuming that you have helm installed, you don't have it installed, [Install Helm](https://helm.sh/docs/intro/install/) is a good place to start. 

At first, we are going to attempt to install the Drupal chart by running `helm install mysite bitnami/drupal` , this command should run successfully. However, when we start debugging the pods we will discover something interesting. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9xj5u4s0feunjl89fijj.png)

The helm installation is successful as helm only verifies that the yaml files provided are accepted by Kubernetes, to check the status of the deployments, we run `kubectl get pods`


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wczlw1rzszjh4gk5xyv5.png)

We realize that our pods are stuck in a Pending state, so we take the next step in debugging, which is to run `kubectl describe` for each of the created pods 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gqp434b8n2buc5i4u3v8.png)

The actual reason that  our pods are not coming up is found when we review the helm installation that we are trying to run. If you check the dependencies in the GitHub repository (https://github.com/bitnami/charts/blob/main/bitnami/drupal/values.yaml) you find out that persistent storage is enabled by default and set to 8Gi. Also, the helm package uses MariaDB and the database size is specified to a default of 8Gi, thus setting the minimum storage for this installation to be 16Gi. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ovk5j18v015tasg7ft60.png)

The obvious reason for the failure of Kubernetes is that our pods do not have the exclusive rights to provision storage on demand, which is Dynamic provisioning, Let's look at a couple of steps that will give our pods this ability and will enable the deployment to be successful. 


The first step here is to create an IAM OpenID connect issuer (OIDC) provider and attach it to your cluster, you can find further documentation on OIDC issuers [here](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html). To do this, we run the following command,

```
eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve
```

Here `my-cluster` is your cluster name, My cluster name is `article-helm` 😂, so don't freak out if you see it in the commands. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3jbm0gva8n8xuuo6ny6q.png)

The next step is to create a service account and attach the EBSDriverPolicy to it. We use the eksutil  command to create the IAM role, and attach the IAM policy to it. We will do this with the following command. 

```
eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole
```

This should create a cloud formation stack to create the service account and attach the necessary policies. You can find more documentation on this process [here](https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html). 

Finally, we will add the Amazon EBS CSI driver as an add-on to our cluster, we can do this by running the following command, 

```
eksctl create addon --name aws-ebs-csi-driver --cluster my-cluster --service-account-role-arn arn:aws:iam::111122223333:role/AmazonEKS_EBS_CSI_DriverRole --force
```

Replace `my-cluster` with the name of your cluster, `111122223333` with your account ID, and `AmazonEKS_EBS_CSI_DriverRole` with the name of the IAM role created earlier. Find additional documentation on this [here](https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0em4m1xln4hhi4dwoeq9.png)

Our cluster is fully set up with the right policies to request additional storage from AWS, whenever Kubernetes needs it. 

The next step is to create a storage class and then ask Helm to refer to our storage class when it is asking for permission to set up additional storage for Kubernetes. I have included the details for the storage class in the official article repo, you can find it in `yaml/storage-class.yaml`.

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: helm-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Delete
allowVolumeExpansion: true 
```
We set up this storage class declaratively by running `kubectl apply -f storage-class.yaml`. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6me7qn3ydp1bp2mg7l9f.png)

When we run `kubectl get sc` to retrieve all the storage classes, we realise that we have two default storage class. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qspfdkj6bkfpojih0eep.png)

To rectify this, we patch the default storage class and remove the default flag on it, we do this by running the following command

```
kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
```

Now, we have just one default storage class, and we are ready to go!

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aghe5tny5k9nrixhpyiv.png)

If you have followed  the instructions to the teeth, your storage class should be all set, and dynamic provisioning should be enabled on your cluster. 

## Running our helm installation successfully
Now that we are done with the necessary setup we can go on to install our helm chart successfully. I have provisioned another `values.yaml` file for our installation, this is to overrun some of the defaults that the bitnami/drupal installation covers. You can check the values.yaml file at `/yaml/values.yaml` in the article repository.   I set all the storage class names to point to `helm-storage` and then I reduced the total volume required to 5Gi, so as not to blow up cost. 

We set up our new installation by running the following command, 

```
helm install mysite bitnami/drupal --values values.yaml 

```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/585dkr37jtrr0gb61igi.png)

Now, we check that our pods are running, 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l9faeflthly0ozofiocs.png)

We also verify that Kubernetes has used our cluster to provision the necessary PersistentVolumes without us explicitly creating the volumes.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0pftxsvct46atzp1shtg.png)

Finally, to get our load balancer service link and test the connection in a web browser, we run the following command

```
kubectl get svc --namespace default -w mysite-drupal

```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8spbdvhe7ioz9u4t8ohe.png)

Then we visit the load balancer link in the browser to see that our service is set up and running efficiently. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rsco6rhm7qhiek2db32t.png)

Voila! We have now enabled Dynamic provisioning for our Kubernetes cluster. 

## Conclusion
In this article, we worked on the concepts of Dynamic and Static volume provisioning in the Kubernetes cluster, we also looked at scenarios where you should use dynamic volume provisioning or static volume provisioning. Finally we set up a cluster with dynamic provisioning and then we deployed a helm chart with high storage requirements. Thank you for reading this article, Please leave a comment and star the repository [here](https://github.com/Monarene/dynamic-volume-k8s). Best regards.