Resizing persistent volumes in EKS/AKS/GKE
Tags: kubernetes, aws, eks, gke
Published: 2019-08-25 16:36:00

Persistent volumes (PVs) in Kubernetes is the way to abstract out block volumes used by pods in the cluster. The way pods request PVs is by creating Persistent Volume Claims (PVCs), which work like resource demand documents. Storage Class is responsible for actual implementation of how the PVs are actually handled, it can be provided by ceph volumes, or in the case of cloud providers, by EBS volumes in AWS, Managed Disks in Azure and Disks in GCP. 


This post is a note made to myself, when I was checking how does the PV resize works in managed kuberentes services. Feel free to replicate the commands and see what happens by yourself.


## Amazon Elastic Kubernetes Service

Let's first create the cluster fast with `eksctl`. Mind it takes a significant time. 


`eksctl create cluster -n darek -r eu-central-1`

After a while you should see some nodes running. 

```
➜  ~ kubectl get nodes 
NAME                                              STATUS   ROLES    AGE   VERSION
ip-192-168-26-190.eu-central-1.compute.internal   Ready    <none>   67s   v1.13.7-eks-c57ff8
ip-192-168-55-249.eu-central-1.compute.internal   Ready    <none>   67s   v1.13.7-eks-c57ff8

```

Let's see the storage class. 

```yaml
➜  ~ kubectl get sc gp2 -o yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2019-08-24T09:23:35Z"
  name: gp2
  resourceVersion: "281"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: d903b89e-c650-11e9-968c-0273bdd9611a
parameters:
  fsType: ext4
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate
```

To resize volumes, storage class needs to have the `allowVolumeExpansion` set to true. Fortunatelly, it can be patched. 

`k patch sc gp2 -p '{"allowVolumeExpansion": true}'`

Now, create a basic PVC of size 10Gi and storageclass set to `gp2`. We will us the same yaml for AKS and GKE later too, save it as `pvc.yml`. 

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-volume-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp2
```

Create the PVC. 

`kubectl apply -f pvc.yml`

The PVC should eventually create a PV of size 10, which is reflected in a real EBS volume created in the AWS. The name of the EBS is encoded in `Source/VolumeID`. You can get it by doing describe operation. 

```
➜  kubectl describe pv/pvc-0262b546-c65a-11e9-8f74-06e2726fa54c
Name:              pvc-0262b546-c65a-11e9-8f74-06e2726fa54c
Labels:            failure-domain.beta.kubernetes.io/region=eu-central-1
                   failure-domain.beta.kubernetes.io/zone=eu-central-1b
Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                   pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      gp2
Status:            Bound
Claim:             default/test-volume-pvc
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          10Gi
Node Affinity:
  Required Terms:
    Term 0:        failure-domain.beta.kubernetes.io/zone in [eu-central-1b]
                   failure-domain.beta.kubernetes.io/region in [eu-central-1]
Message:
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://eu-central-1b/vol-056a0bc0115292add
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>
```

The above command shows the EBS volume name to be `vol-056a0bc0115292add`. 
Now let's resize the PVC. Change the storage size in `pvc.yml` to 12Gi and do kubectl apply again. 

```
➜  kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-0262b546-c65a-11e9-8f74-06e2726fa54c   12Gi       RWO            Delete           Bound    default/test-volume-pvc   gp2                     19h
```

Ok so the PV shows 12Gi. Let's check the size of the actual EBS volume with aws cli. 

`aws ec2 describe-volumes --volume-ids vol-056a0bc0115292add --query 'Volumes[0].Size'`

It should now show 12.

> Have in mind that in Amazon you can do resize operation on EBS only once every 6h. So next resize won't work, unless you wait for it. 

You can delete the clutser by doing `eksctl delete cluster -n darek`. 


## Azure AKS

Let's now check AKS. Create a cluster in AKS. The below snippet creates a resource group darek and a cluster called `darekEKS`. 

```
az group create --name darek --location westeurope
az aks create --resource-group darek --name darekEKS --node-count 2 --enable-addons monitoring --generate-ssh-keys
az aks get-credentials --resource-group darek --name darekEKS
```

Storage class in AKS is called default, it will provision azure managed disk. 

```
➜  kubectl get sc
NAME                PROVISIONER                AGE
default (default)   kubernetes.io/azure-disk   13m
managed-premium     kubernetes.io/azure-disk   13m

```

Let's see the whole class, similarly as in EKS the resize is not enbled. 

```
➜  kubectl get storageclasses.storage.k8s.io default -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"},"labels":{"kubernetes.io/cluster-service":"true"},"name":"default","namespace":""},"parameters":{"cachingmode":"ReadOnly","kind":"Managed","storageaccounttype":"Standard_LRS"},"provisioner":"kubernetes.io/azure-disk"}
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2019-08-24T09:31:20Z"
  labels:
    kubernetes.io/cluster-service: "true"
  name: default
  resourceVersion: "459"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/default
  uid: ee21d5ae-c651-11e9-9f0b-aac03b918570
parameters:
  cachingmode: ReadOnly
  kind: Managed
  storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: Immediate
```

Let's patch it: `kubectl patch sc default -p '{"allowVolumeExpansion": true}' ` 
And create the PVC, remember to change the `storageClassName` to default. 

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-volume-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: default
```

Is PVC and PV created?


```
➜  kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-volume-pvc   Bound    pvc-eb332309-c654-11e9-a0fd-36761452f1dd   10Gi       RWO            default        26s

➜  kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-eb332309-c654-11e9-a0fd-36761452f1dd   10Gi       RWO            Delete           Bound    default/test-volume-pvc   default                 41s
```

The name of the Azure disk will be seen in `Source/DiskURI`. 

```
➜ ~ kubectl describe pv
Name:            pvc-eb332309-c654-11e9-a0fd-36761452f1dd
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller: yes
                 pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk
                 volumehelper.VolumeDynamicallyCreatedByKey: azure-disk-dynamic-provisioner
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    default
Status:          Bound
Claim:           default/test-volume-pvc
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        12Gi
Node Affinity:   <none>
Message:
Source:
    Type:         AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
    DiskName:     kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd
    DiskURI:      /subscriptions/6690b014-bdbd-4496-98ee-f2f255699f70/resourceGroups/MC_darek_darekEKS_westeurope/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd
    Kind:         Managed
    FSType:
    CachingMode:  ReadOnly
    ReadOnly:     false
Events:           <none>

```

Now change the disk size to 12 and see. We can check the disk size with the following command. 

```
> az disk  show --ids /subscriptions/6690b014-bdbd-4496-98ee-f2f255699f70/resourceGroups/MC_darek_darekEKS_westeurope/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd --query 'diskSizeGb'
12
```

Ok works in AKS too. What is quite weird, when you do `kubectl get pv` it shows the new size but the PVC still shows 10Gi.

Clean resources you have created: `az group delete -n darek -y` 

## Google Kubernetes Engine

I created a [zonal cluster](https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters) using the CLI in my project called `turnkey-cooler-31343`. 

```
gcloud config set project turnkey-cooler-31343
gcloud container clusters create darek --zone europe-north1-a --cluster-version "1.13.7-gke.19" --machine-type n1-standard-1 --num-nodes 2
```

And generate a kubeconfig:

```
gcloud container clusters get-credentials darek --zone europe-north1-a --project turnkey-cooler-31343
```

In GKE the default storageclass is called `standard`.

```
NAME                 PROVISIONER            AGE
standard (default)   kubernetes.io/gce-pd   3m44s
```

Just as in AKS and EKS the auto expand was not enabled, let's patch the storage class.

`kubectl patch sc standard -p '{"allowVolumeExpansion": true}'`

And create a pvc using our PVC yaml. Remember to change `storageClassName` to `standard`.

`kubectl apply -f pvc.yml`

And check the disk:

```
kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-f79ca9c5-c750-11e9-911b-42010aa601a8   10Gi       RWO            Delete           Bound    default/test-volume-pvc   standard                8m30s
```

Now check the name of the disk in GCP:

```
kubectl get pv/pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 -o jsonpath='{.spec.gcePersistentDisk.pdName}'
gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8
```

And the size of the disk. 

```
➜  gcloud compute disks describe gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 --zone europe-north1-a | grep sizeGb
sizeGb: '10'
```

Let's now resize the disk, change the pvc.yml or patch the PVC. 

```
➜  gcloud compute disks describe gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 --zone europe-north1-a | grep sizeGb
sizeGb: '12'
```

Ok all fine but as in AKS and EKS the PVC still shows 10Gi. 

Clean the infra: `gcloud container clusters delete darek --zone europe-north1-a `.