Azure Kubernetes Service — Next level persistent storage with Azure Disk CSI driver
Tags: azure, kubernetes, cloudnative, devops
Published: 2021-10-13 17:01:06

When talking about persistent storage with Kubernetes Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) are our tools of choice. We can use them with Azure Disks and Azure Files for a long time now. But now it is time to bring them to the next level!

This post mainly focuses on Azure Disks (don’t miss to review the links below to get more details on Azure Files).

## What’s new?

With Azure Disk Container Storage Interface (CSI) driver for Azure Disk, we are now getting some great features that help us to run our stateful services much smoother on Azure Kubernetes Service:

* Zone Redundant Storage (ZRS) Azure Disks
* ReadWriteMany with Azure Disks
* Kubernetes-native volume snapshots
* Volume resizing
* Volume cloning

## Azure Disk Container Storage Interface driver

Since Kubernetes 1.21 (on Azure Kubernetes Service) we can use the new Container Storage Interface implementation which is available for Azure Disk and Azure Files.

### Wait, what is Container Storage Interface?

The Container Storage Interface (CSI) is an abstraction layer that allows third-party storage providers to write plugins exposing new storage systems in Kubernetes. CSI is generally available for some time now and replaces the previous volume plugin system. With CSI third-party providers can develop and maintain their plugins outside of the Kubernetes project lifecycle which brings them more flexibility. CSI is the future for any storage integration and is already used in many scenarios.

### How to use the Azure Disk CSI driver with Azure Kubernetes Service?

You will need at least Kubernetes 1.21 or newer to be able to use the Azure Disk Container Storage Interface driver. Besides this you will also need to enable some Azure features gates to be able to use all of the above-mentioned features:

```bash
az feature register --name SsdZrsManagedDisks --namespace Microsoft.Compute
az feature register --name EnableAzureDiskFileCSIDriver --namespace Microsoft.ContainerService
az feature list -o table --query "[?contains(name, 'Microsoft.Compute/SsdZrsManagedDisks')].{Name:name,State:properties.state}"
az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/EnableAzureDiskFileCSIDriver')].{Name:name,State:properties.state}"
az provider register --namespace Microsoft.Compute
az provider register -n Microsoft.ContainerService
```

You will then need to spin up an Azure Kubernetes Cluster with the CSI driver feature enabled:

```bash
az group create -g aks-1-22 -l westeurope
az aks create -n aks-1-22 \
  -g aks-1-22 \
  -l westeurope \
  -c 3 \
  -s Standard_B2ms \
  --node-zones 1 2 3 \
  --aks-custom-headers EnableAzureDiskFileCSIDriver=true
az aks get-credentials -n aks-1-22 -g aks-1-22
```

Furthermore, you will need to create a `StorageClass` and `VolumeSnapshotClass` to use all mentioned features:


```bash
cat <<EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azuredisk-csi-premium-zrs
provisioner: disk.csi.azure.com
parameters:
  skuname: Premium_ZRS
  maxShares: "3"
  cachingMode: None
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
  name: azuredisk-csi-vsc
driver: disk.csi.azure.com
deletionPolicy: Delete
parameters:
  incremental: "true"
EOF
```

The `StorageClass` is configured to use the new Premium_ZRS Azure Disks that allow to mount the Disk to multiple Pods across multiple Availability Zones.

The `VolumeSnapshotClass` allows us to use the Kubernetes-native volume snapshot feature.

We are now ready to create a `PersistentVolumeClaim` based on the above `StorageClass`:

```bash
cat <<EOF | kubectl apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-azuredisk-csi-premium-zrs
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 256Gi
  volumeMode: Block
  storageClassName: azuredisk-csi-premium-zrs
EOF
```

## Kubernetes-native volume snapshots

With the Azure Disk CSI driver, we are able to create volume snapshots. You will find more details on the volume snapshot feature itself [here](https://kubernetes.io/docs/concepts/storage/volume-snapshots). Use the following code snippet to create a snapshot of the above create PVC:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: azuredisk-volume-snapshot
spec:
  volumeSnapshotClassName: azuredisk-csi-vsc
  source:
    persistentVolumeClaimName: pvc-azuredisk-csi-premium-zrs
EOF
```

You can now verify the snapshot with the following `kubectl` command:

```bash
kubectl describe volumesnapshot azuredisk-volume-snapshot
```

## Volume cloning

Volume snapshots themselves are great but we can also use them to clone an existing PVC. For example to attach it to another workload. This can be done with the following manifest:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-azuredisk-snapshot-restored
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: azuredisk-csi-premium-zrs
  resources:
    requests:
      storage: 265Gi
  dataSource:
    name: azuredisk-volume-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
EOF
```

Once again you can review it with `kubectl`:

```bash
kubectl get pvc,pv
```

## Volume resizing

The CSI driver also supports resizing an existing PVC. Some important notes on this one:

* The PVC needs to be unmounted for resizing
* The PV will be resized directly, the PVC after mounting it

You can resize the above-cloned PVC via kubectl by executing the following command:

```bash
kubectl patch pvc pvc-azuredisk-snapshot-restored --type merge --patch '{"spec": {"resources": {"requests": {"storage": "512Gi"}}}}'
```

## ReadWriteMany across Availability Zones

As already mentioned above we are now also able to mount our PVC to multiple workloads via ReadWriteMany. With the new ZRS Azure Disks, we are even further able to use them across Availability Zones.

Let’s test these new features with the below `Deployment` manifest:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          volumeDevices:
            - name: azuredisk
              devicePath: /dev/sdx
      volumes:
        - name: azuredisk
          persistentVolumeClaim:
            claimName: pvc-azuredisk-csi-premium-zrs
EOF
```

As you see above are spinning up a deployment with three replicas across our spread Kubernetes nodes mounting the same Azure Disk with read-write access. You can review it by executing the following `kubectl` command:

```bash
kubectl get pods,volumeattachments -owide
```

That said, (as you may have already seen) ZRS only supports Block devices so far.

## More details

Don’t miss to review the following links to get some more insights and further documentation:

* https://kubernetes-csi.github.io/docs
* https://github.com/kubernetes-sigs/azuredisk-csi-driver
* https://github.com/kubernetes-sigs/azurefile-csi-driver
* https://docs.microsoft.com/azure/aks/csi-storage-drivers