Azure Container Storage: A New Dawn in Kubernetes Storage Solutions
Tags: kubernetes, azure, aks, containers
Published: 2024-04-04 01:46:49

In this article, I’ll explore the Azure Container Storage service, explaining its relationships with the other Kubernetes storage functionalities. However, before talking about it, let’s first review the Kubernetes storage services.

Containers, as we know, changed the shape of software development. However, their ephemeral nature means any data saved during a container’s lifecycle is lost upon its termination. To ensure data persistence and sharing across containers, Kubernetes introduced the concept of volumes.

# What are Volumes?

A volume in Kubernetes is a directory that can be accessed by containers within a pod to store data. The location and underlying storage of this directory depend on the volume type. For example, the emptyDir volume type is a temporary directory on the host node. This directory is accessible to all containers within the pod that have configured the volume. However, it is not accessible by containers running on other nodes’ pods.

In the example provided, I’ve deployed the following resources in my AKS cluster:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: weather-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: weather-app
  template:
    metadata:
      labels:
        app: weather-app
    spec:
      volumes:
        - name: weather-app
          emptyDir: {}
      containers:
        - name: temperature-api
          image: registry.k8s.io/ubuntu-slim:0.1
          volumeMounts:
            - name: weather-app
              mountPath: /temperature        
          args:
            - "-c"
            - "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"
          
        - name: humidity-api
          image: registry.k8s.io/ubuntu-slim:0.1
          volumeMounts:
            - name: weather-app
              mountPath: /humidity    
          command: ["/bin/sh"]
          args:
            - "-c"
            - "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"

        - name: web-application
          image: registry.k8s.io/ubuntu-slim:0.1
          volumeMounts:
            - name: weather-app
              mountPath: /temperature  
            - name: weather-app
              mountPath: /humidity  
          command: ["/bin/sh"]
          args:
            - "-c"
            - "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"
```
If I access the temperature-api container in any of the pods via the shell and list its contents, I can the temperature directory. This directory matches the mount path specified for the volume’s container.
```
kubectl get pod    
NAME                           READY   STATUS    RESTARTS   AGE
weather-app-78855b8f7d-bcbql   3/3     Running   0          19h
weather-app-78855b8f7d-nc645   3/3     Running   0          19h
weather-app-78855b8f7d-xmfsm   3/3     Running   0          19h

kubectl get pods weather-app-78855b8f7d-nc645 -o=jsonpath='{.metadata.uid}'
fe46222e-3cfa-4e41-9a07-9ff940435548

kubectl exec -it weather-app-78855b8f7d-nc645 -c temperature-api -- sh
# ls
bin   dev  home  lib64  mnt  proc  run   srv  temperature  usr
boot  etc  lib   media  opt  root  sbin  sys  tmp          var
# touch temperature/sample.json
# ls
sample.json
```
If I log into the node and explore the content within the `/var/lib/kubelet/pods/<POD-ID>` directory I will be able to view not only the weather-app volume but also its content. In this case, that includes the sample.json file I’ve previously created.
```
$ ls /var/lib/kubelet/pods
0ac8291c-24cd-4b4c-9de5-4c6f37c188c8  792c4a3a-6238-4d56-a477-515f026f1338
0c748dad-91b9-4dfb-90f4-8b3c4847b6f7  8f0cecd1-ffd0-4556-a902-d25cab7e3fd5
1111b71b-0051-4c28-a15c-92580b8b7b72  90414e12-8073-4df9-b5cd-4c723d40a496
2384bdf2-65db-4c15-97b3-640276ea8bc3  a4d74959-0021-436e-b7c6-34f8d249a9ca
2acba766-6202-4288-9588-acb972ca77cb  c70bb0b8-06fc-4148-9ea8-e9ff370b85b8
43195978-5512-4c8a-86cd-bb72c8e5d162  ceb29f4c-a18a-45ce-92ed-0ccde4d06586
43b23be2-5fb1-4dcd-9e75-160151b65300  e4bb764c-9d92-4306-9006-66a924197147
4ca23b04-5220-4832-be02-aa77e34fbd28  ea6fb720-b4e7-42f3-ba56-e687992d2c38
4e7cb21b-c0a1-48d1-a03b-ff4d4a71b181  fe46222e-3cfa-4e41-9a07-9ff940435548

$ tree /var/lib/kubelet/pods/fe46222e-3cfa-4e41-9a07-9ff940435548
/var/lib/kubelet/pods/fe46222e-3cfa-4e41-9a07-9ff940435548
├── containers
│   ├── humidity-api
│   │   └── 6e6db8d8
│   ├── temperature-api
│   │   └── 994c912c
│   └── web-application
│       └── c9d6fc18
├── etc-hosts
├── plugins
│   └── kubernetes.io~empty-dir
│       ├── weather-app
│       │   └── ready
│       └── wrapped_kube-api-access-hpbxl
│           └── ready
└── volumes
    ├── kubernetes.io~empty-dir
    │   └── weather-app
    │       └── sample.json
    └── kubernetes.io~projected
        └── kube-api-access-hpbxl
            ├── ca.crt -> ..data/ca.crt
            ├── namespace -> ..data/namespace
            └── token -> ..data/token
```
As you can see in this example, the usage of volumes makes, the data sharing between pods much easier. However, it’s crucial to understand that each replica of the weather-app pod has its own unique `emptyDir` volume. With three replicas of the weather-app pod deployed, there are three separate `emptyDir` volumes.

## Volume categories
Volumes fall under three primary categories:

- **Ephemeral Volumes**: They are tied to a pod’s lifecycle, and all data are lost once the pod terminates. Examples include `emptyDir`, `downwardAPI`, `configMap`, `secret`, and `ephemeral`.
- **Persistent Volumes (PVs)**: They persist beyond a pod’s lifecycle, and the data is maintained even if the pod restarts. Examples span cloud-based storages like `gcePersistentDisk`, `awsElasticBlockStore`, `azureDisk`, network storage systems such as nfs, cephfs, and software-defined storage solutions.
- **Projected Volumes**: Map multiple existing volumes to the same directory. Examples include `downwardAPI`, `configMap`, `secret`, and `serviceAccountToken`. We’ve already seen an example in the previous code snippet.
```
$ tree /var/lib/kubelet/pods/fe46222e-3cfa-4e41-9a07-9ff940435548
/var/lib/kubelet/pods/fe46222e-3cfa-4e41-9a07-9ff940435548
├── containers
├── etc-hosts
├── plugins
└── volumes
    └── kubernetes.io~projected
        └── kube-api-access-hpbxl
            ├── ca.crt -> ..data/ca.crt
            ├── namespace -> ..data/namespace
            └── token -> ..data/token
```
## Persistent Volumes, Persistent Volume Claims and Storage Classes
Persistent Volumes has different performance characteristics and properties which are influenced by the underlying service. For example, while Azure Files offers built-in scalability and Azure AD integration, NFS provide cross-platform versatility without specifying drivers and support the RWX (Read-Write-Many) access mode, permitting multiple nodes to mount the volume as read-write.

An import consideration about PV (Persistent Volumes) is that they signal Kubernetes about a storage ready for use, by defining the underlying storage service and its configuration, but it won’t allocate it for any workload. In the following example, I’ve defined the configuration needed to create a Persistent Volume that points to an Azure File service.
```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: azurefile-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  azureFile:
    secretName: azure-storage-account-gtrekter-secret
    shareName: fstraininguks01
    readOnly: true
```
Once Kubernetes knows about the existence of your Persistent volume, you can then start assigning it to your pods by using Persistent Volume Claim (PVC). This claim abstracts the storage from the underlying infrastructure, by preventing you from specifying secrets, and configurations in the deployment manifest; you will simply specify your requirements. For example, This PVC (Persistent Volume Claim) request 5Gi of storage of our previously created PV with ReadWriteOnce access mode.
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: azurefile-pvc
spec:
  resources:
    requests:
      storage: 5Gi
  accessModes:
    - ReadWriteOnce
```
PV and PVC are based on the precondition that the Kubernetes administrator has manually provisioned the storage defined in the PV and that the resource is reachable. With the introduction of Storage Classes, we can make the entire provisioning dynamic, and in line with the PVC requirements. In this example, whenever a `PersistentVolumeClaim` that requests this `StorageClass` is created, Kubernetes will use the parameters specified in the `StorageClass` to dynamically provision a new Azure Files share.
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: azurefile-pvc
spec:
  storageClassName: azurefile
  resources:
    requests:
      storage: 5Gi
  accessModes:
    - ReadWriteOnce
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile
provisioner: kubernetes.io/azure-file
parameters:
  storageAccount: myazurestorageaccountname
  location: eastus
  skuName: Standard_LRS  
  secretNamespace: default  
```
# Container Storage Interface (CSI)
Prior to version 1.13, Kubernetes used “in-tree” plugins to interface with storage services like Azure Files and AWS Elastic Block Store. These plugins were embedded directly within the Kubernetes binaries. This approach posed several challenges. For example, reliability and security concerns arose due to the inclusion of third-party storage code in the core Kubernetes binaries. It also added to the maintenance burden of Kubernetes maintainers who had to test and oversee the code for various storage services. Additionally, vendors seeking to introduce support for their storage systems in Kubernetes, or fix a bug in an existing volume plugin, were compelled to conform to the Kubernetes release process, resulting in potentially delay of releases of bug fixes or updates.

To address these challenges, Kubernetes introduced the Container Storage Interface (CSI). This standard for exposing arbitrary block and file storage systems allowed vendors to develop, deploy, and update plugins for integrating new or enhancing existing storage systems in Kubernetes without altering the core Kubernetes codebase.

The CSI requires the Kubernetes administrator to install the drivers in the cluster, and many vendors already provided the procedure to do so, for example, to enable CSI storage drivers on an AKS cluster, you can use the following code:
```
az aks update -n myAKSCluster -g myResourceGroup --enable-disk-driver --enable-file-driver --enable-blob-driver --enable-snapshot-controller
```
This command activates the Azure Disks CSI driver (`--enable-disk-driver`), Azure Files CSI driver (`--enable-file-driver`), Azure Blob Storage CSI driver (`--enable-blob-driver`), and the Snapshot controller (`--enable-snapshot-controller`).

# Azure Container Storage
Now that we have a foundational understanding of Kubernetes storage mechanisms, let’s move to the recently introduced Azure Container Storage.

Using specific CSI drivers to facilitate cloud storage for containers meant adjusting storage services, initially designed for IaaS workloads, for containerized environments. This adaptation often resulted in operational challenges, increasing risks concerning application availability due to potential bugs or incompatibilities in CSI drivers. Additionally, it could affect scalability and introduce performance delays.

Inspired by OpenEBS, an open-source Kubernetes storage solution, Azure Container Storage introduces a true container-native storage. By providing a managed volume orchestration system using microservice-based storage controllers within a Kubernetes framework, Azure Container Storage separates the storage management layer from both pods and the underlying storage, removing the need for many CSI drivers. In fact, while, by default, when we provision a new Azure Kubernetes cluster we will have the following CSI drivers enabled:
```
$ kubectl get csidrivers.storage.k8s.io
NAME                 ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES                  AGE
disk.csi.azure.com   true             false            false             <unset>         false               Persistent             11m
file.csi.azure.com   false            true             false             <unset>         false               Persistent,Ephemeral   11m
```
we will just need one CSI diver for all storage types.
```
$ kubectl get csidrivers.storage.k8s.io                                                    13:41:52
NAME                             ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES                  AGE
containerstorage.csi.azure.com   true             false            false             <unset>         false               Persistent             44h
...
```
## Components
Azure Container Instances components include:

- **A Storage Pool**: This is an aggregation of storage resources presented as a storage entity for an AKS cluster.
- **A Data Services Layer**: This layer undertakes responsibilities like replication, encryption, and other supplementary functions that the foundational storage provider might not offer.
- **A Protocol Layer**: This layer exposes the provisioned volumes to application pods using the NVMe-oF protocol. NVMe over Fabrics (NVMe-oF) is a protocol that extends the NVMe storage interface over network fabrics, which offers a faster attach/detach of PVs.

## Installation
The installation of Azure Container Sotorage on a AKS cluster is pretty straightforward. First we need to add or upgrade to the latest version of both the `aks-preview` cli extension and `k8s-extension`.
```
$ az extension add --upgrade --name aks-preview
$ az extension add --upgrade --name k8s-extension
``
Register the `Microsoft.containerService`, and `Microsoft.KubernetesConfiguration` resource providers.
```
$ az provider register --namespace Microsoft.ContainerService --wait 
$ az provider register --namespace Microsoft.KubernetesConfiguration --wait 
```
Previously we had to grant permissions to the Azure Kubernetes Service so that the service was able to provision storage for the cluster, update the existing node pool label to associate it with the correct IO engine for Azure Container Storage, and create a new extension of type `microsoft.azurecontainerstorage` to the Kubernetes cluster.
```
$ az aks nodepool update --resource-group rg-training-dev-krc --cluster-name aks-training-dev-krc --name agentpool --labels acstor.azure.com/io-engine=acstor

$ export AKS_MI_OBJECT_ID=$(az aks show --name aks-training-dev-krc --resource-group rg-training-dev-krc --query "identityProfile.kubeletidentity.objectId" -o tsv)
$ export AKS_NODE_RG=$(az aks show --name aks-training-dev-krc --resource-group rg-training-dev-krc --query "nodeResourceGroup" -o tsv)
$ az role assignment create --assignee $AKS_MI_OBJECT_ID --role "Contributor" --resource-group "$AKS_NODE_RG"

$ az k8s-extension create --cluster-type managedClusters --cluster-name aks-training-dev-krc --resource-group rg-training-dev-krc --name acstor --extension-type microsoft.azurecontainerstorage --scope cluster --release-train stable --release-namespace acstor
```
However, with the latest version of the `aks-preview` and `k8s-extension` extensions, we just need to execute the following command. It will take care of the AAD role propagation and everything else.
```
az aks update --name aks-training-dev-krc --resource-group training-dev-krs --enable-azure-container-storage azureDisk
```
In this scenario, I’ve activated AzureDisk for container storage. However, there are other options available:

- **Azure Elastic SAN (elasticSan)**: This option allows for storage to be allocated as needed for each volume or snapshot created. It supports simultaneous access from multiple clusters to a single SAN, but each persistent volume can only be connected to by one consumer at any given time.
- **Azure Disks (azureDisk)**: Here, storage allocation is based on the specified size of the container storage pool and the maximum size of each volume.
- **Ephemeral Disk (ephemeralDisk)**: With this choice, AKS identifies the ephemeral storage available on its nodes and utilizes these drives for deploying volumes.

Before getting into an example of Azure Container Storage usage, let’s take a moment to double-check what the extension added to our cluster, starting from the available resource types.
```
$ kubectl api-resources 
NAME                              SHORTNAMES          APIVERSION                             NAMESPACED   KIND
...
challenges                                            acme.cert-manager.io/v1                true         Challenge
orders                                                acme.cert-manager.io/v1                true         Order
...
certificaterequests               cr,crs              cert-manager.io/v1                     true         CertificateRequest
certificates                      cert,certs          cert-manager.io/v1                     true         Certificate
clusterissuers                                        cert-manager.io/v1                     false        ClusterIssuer
issuers                                               cert-manager.io/v1                     true         Issuer
...
configsyncstatuses                                    clusterconfig.azure.com/v1beta1        true         ConfigSyncStatus
extensionconfigs                  ec                  clusterconfig.azure.com/v1beta1        true         ExtensionConfig
capacityprovisionerconfigs                            containerstorage.azure.com/v1alpha1    true         CapacityProvisionerConfig
diskpools                         dp                  containerstorage.azure.com/v1alpha1    true         DiskPool
storagepools                      sp                  containerstorage.azure.com/v1alpha1    true         StoragePool
...
etcdclusters                      etcd                etcd.database.coreos.com/v1beta2       true         EtcdCluster
...
jaegers                                               jaegertracing.io/v1                    true         Jaeger
...
alertmanagerconfigs               amcfg               monitoring.coreos.com/v1alpha1         true         AlertmanagerConfig
alertmanagers                     am                  monitoring.coreos.com/v1               true         Alertmanager
podmonitors                       pmon                monitoring.coreos.com/v1               true         PodMonitor
probes                            prb                 monitoring.coreos.com/v1               true         Probe
prometheuses                      prom                monitoring.coreos.com/v1               true         Prometheus
prometheusrules                   promrule            monitoring.coreos.com/v1               true         PrometheusRule
servicemonitors                   smon                monitoring.coreos.com/v1               true         ServiceMonitor
thanosrulers                      ruler               monitoring.coreos.com/v1               true         ThanosRuler
...
blockdeviceclaims                 bdc                 openebs.io/v1alpha1                    true         BlockDeviceClaim
blockdevices                      bd                  openebs.io/v1alpha1                    true         BlockDevice
diskpools                         dsp                 openebs.io/v1alpha1                    true         DiskPool
...
```
As you can see, we have a bunch of new operators. Within them you can see both resources specific to the Azure container storage and OpenEBS. Next, let’s inspect the Storage Classes.
```
$ kubectl get storageclass
NAME                                  PROVISIONER                      RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
acstor-azuredisk                      containerstorage.csi.azure.com   Delete          WaitForFirstConsumer   true                   28m
acstor-azuredisk-internal             disk.csi.azure.com               Retain          WaitForFirstConsumer   true                   32m
acstor-azuredisk-internal-azuredisk   disk.csi.azure.com               Retain          WaitForFirstConsumer   true                   28m
...

$ kubectl describe sc acstor-azuredisk
Name:                  acstor-azuredisk
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           containerstorage.csi.azure.com
Parameters:            acstor.azure.com/storagepool=azuredisk,ioTimeout=60,proto=nvmf,repl=1
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>
```
The newly introduced `acstor-azuredisk` Storage Class is specifically designed for the Azure Container Storage operations.

# Example
In this example, we’ll explore how Azure Disks are integrated into a Kubernetes cluster through Azure Container Storage, with a focus on the relationships among the various resources involved. The first step is to create a new Storage Pool.
```
apiVersion: containerstorage.azure.com/v1alpha1
kind: StoragePool
metadata:
  name: training-azuredisk
  namespace: acstor
spec:
  poolType:
    azureDisk: {}
  resources:
    requests: {"storage": 100Gi}
```
Executing this configuration results in the creation of a new storage pool resource:
```
$ kubectl get sp -A
NAMESPACE   NAME                 CAPACITY       AVAILABLE      USED         RESERVED     READY   AGE
...
acstor      training-azuredisk   10737418240    10562019328    175398912    108290048    True    38s

$ kubectl describe sp -n acstor training-azuredisk
Name:         training-azuredisk
API Version:  containerstorage.azure.com/v1beta1
Kind:         StoragePool
...
Spec:
  Pool Type:
    Azure Disk:
      Sku Name:  Premium_LRS
  Resources:
    Requests:
      Storage:  10Gi
  Zones:

Status:
  Available:           10562015232
  Available Replicas:  0
  Capacity:            10737418240
  Conditions:
    Last Transition Time:  2024-03-05T03:04:14Z
    Message:               Storage pool is ready.
    Reason:                Ready
    Status:                True
    Type:                  Ready
    Last Transition Time:  2024-03-05T03:03:57Z
    Message:               Storage pool is ready.
    Reason:                Ready
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2024-03-05T03:03:57Z
    Message:               StoragePool is scheduled.
    Reason:                Scheduled
    Status:                True
    Type:                  PoolScheduled
  Max Allocation:          10562015232
  Reserved:                108294144
  Used:                    175403008
Events:                    <none>
```
Behind the scenes, Azure initializes a new Storage Class named `actor-azuredisk`
```
$ kubectl get sc
NAME                                           PROVISIONER                      RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
...
acstor-azuredisk-internal-training-azuredisk   disk.csi.azure.com               Retain          WaitForFirstConsumer   true                   112s
acstor-training-azuredisk                      containerstorage.csi.azure.com   Delete          WaitForFirstConsumer   true                   112s
...

$ kubectl describe sc acstor-azuredisk-internal-training-azuredisk
Name:                  acstor-azuredisk-internal-training-azuredisk
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           disk.csi.azure.com
Parameters:            skuName=Premium_LRS
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Retain
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

$ kubectl describe sc acstor-training-azuredisk
Name:                  acstor-training-azuredisk
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           containerstorage.csi.azure.com
Parameters:            acstor.azure.com/storagepool=training-azuredisk,ioTimeout=60,proto=nvmf,repl=1
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>
```
A Disk Pool serves as a reservoir of Azure Disks, meeting the storage needs within the cluster, managed by the `capacity-provisioner` responsible for provisioning and reclaiming storage.
```
$ kubectl get dp -A
NAMESPACE   NAME                                CAPACITY       AVAILABLE      USED         RESERVED     READY   AGE
...
acstor      training-azuredisk-diskpool-cbarq   10737418240    10562015232    175403008    108294144    True    4m5s

$ kubectl describe dp training-azuredisk-diskpool-cbarq -n acstor
Name:         training-azuredisk-diskpool-cbarq
Namespace:    acstor
...
API Version:  containerstorage.azure.com/v1beta1
Kind:         DiskPool
Spec:
  Disk Pool Source:
    Azure Disk:
      Storage Class Name:  acstor-azuredisk-internal-training-azuredisk
  Resources:
    Requests:
      Storage:  10Gi
Status:
  Available:           10562015232
  Available Replicas:  0
  Capacity:            10737418240
  Conditions:
    Last Transition Time:  2024-03-05T03:04:14Z
    Message:               StoragePool is ready.
    Reason:                PoolReady
    Status:                True
    Type:                  Ready
    Last Transition Time:  2024-03-05T03:04:14Z
    Message:               All resources are Online.
    Reason:                PoolReady
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2024-03-05T03:03:57Z
    Message:               StoragePool is scheduled.
    Reason:                PoolScheduled
    Status:                True
    Type:                  PoolScheduled
  Max Allocation:          10562015232
  Reserved:                108294144
  Used:                    175403008
Events:                    <none>
```
As mentioned earlier, Azure Container Storage leverages OpenEBS. Kubernetes, in turn, generates an OpenEBS DiskPool resource, which oversees the creation and deletion of volumes on the underlying storage infrastructure.
```
$ kubectl get dsp -A
NAMESPACE   NAME        NODE                                STATUS   CAPACITY       USED         AVAILABLE      RESERVED
acstor      csi-2ht4k   aks-agentpool-10525397-vmss000002   Online   10737418240    175403008    10562015232    108294144
...

$ kubectl describe dsp -n acstor csi-2ht4k
Name:         csi-2ht4k
Namespace:    acstor
...
Annotations:  acstor.azure.com/expansion-requested: 10737418240
API Version:  openebs.io/v1alpha1
Kind:         DiskPool
...
Spec:
  Disks:
    /dev/sdc
  Node:  aks-agentpool-10525397-vmss000002
Status:
  Available:  10562015232
  Capacity:   10737418240
  Node:       aks-agentpool-10525397-vmss000002
  Reserved:   108294144
  State:      Online
  Used:       175403008
Events:
  Type    Reason   Age    From          Message
  ----    ------   ----   ----          -------
  Normal  Created  6m49s  dsp-operator  Created or imported pool
  Normal  Online   6m49s  dsp-operator  Pool online and ready to roll!
```
Ultimately, Azure configures the Persistent Volumes (PVs) for us, based on Azure Disks. These disks communicate with Azure through the CSI `driverdisk.csi.azure.com`.
```
$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                            STORAGECLASS                                   REASON   AGE
pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7   10Gi       RWO            Retain           Bound    acstor/training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf   acstor-azuredisk-internal-training-azuredisk            7m48s
...

$ kubectl describe pv pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7
Name:              pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: disk.csi.azure.com
                   volume.kubernetes.io/provisioner-deletion-secret-name:
                   volume.kubernetes.io/provisioner-deletion-secret-namespace:
Finalizers:        [kubernetes.io/pv-protection external-attacher/disk-csi-azure-com]
StorageClass:      acstor-azuredisk-internal-training-azuredisk
Status:            Bound
Claim:             acstor/training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Block
Capacity:          10Gi
Node Affinity:
  Required Terms:
    Term 0:        topology.disk.csi.azure.com/zone in []
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            disk.csi.azure.com
    FSType:
    VolumeHandle:      /subscriptions/49bd11f3-09ce-4ecf-9f96-e026a292f054/resourceGroups/aks-training-dev-krc-infra/providers/Microsoft.Compute/disks/pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7
    ReadOnly:          false
    VolumeAttributes:      csi.storage.k8s.io/pv/name=pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7
                           csi.storage.k8s.io/pvc/name=training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf
                           csi.storage.k8s.io/pvc/namespace=acstor
                           requestedsizegib=10
                           skuName=Premium_LRS
                           storage.kubernetes.io/csiProvisionerIdentity=1709604784805-9137-disk.csi.azure.com
Events:                <none>
```
To utilize the storage provided by PVs, a Persistent Volume Claim (PVC) is essential. This PVC, associated with the acstor-azuredisk-internal storage class, links to the PV (indicated by the Bound status).
```
$ kubectl get pvc -A
NAMESPACE   NAME                                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                                   AGE
acstor      training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf   Bound    pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7   10Gi       RWO            acstor-azuredisk-internal-training-azuredisk   9m4s

$ kubectl describe pvc training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf -n acstor
Name:          training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf
Namespace:     acstor
StorageClass:  acstor-azuredisk-internal-training-azuredisk
Status:        Bound
Volume:        pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7
Labels:        acstor.azure.com/diskpool=training-azuredisk-diskpool-cbarq
               acstor.azure.com/managedby=capacity-provisioner
               acstor.azure.com/storagepool=training-azuredisk
               app.kubernetes.io/component=storage-pool
               app.kubernetes.io/managed-by=capacity-provisioner
               app.kubernetes.io/part-of=acstor
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: disk.csi.azure.com
               volume.kubernetes.io/selected-node: aks-agentpool-10525397-vmss000002
               volume.kubernetes.io/storage-provisioner: disk.csi.azure.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Block
Used By:       diskpool-worker-lbxwt
Events:
  Type    Reason                 Age    From                                                                                               Message
  ----    ------                 ----   ----                                                                                               -------
  Normal  WaitForFirstConsumer   9m26s  persistentvolume-controller                                                                        waiting for first consumer to be created before binding
  Normal  ExternalProvisioning   9m26s  persistentvolume-controller                                                                        waiting for a volume to be created, either by external provisioner "disk.csi.azure.com" or manually created by system administrator
  Normal  Provisioning           9m26s  disk.csi.azure.com_csi-azuredisk-controller-69b978cbc6-wvmft_bdc14308-5f21-4fe5-933b-058b2633d611  External provisioner is provisioning volume for claim "acstor/training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf"
  Normal  ProvisioningSucceeded  9m24s  disk.csi.azure.com_csi-azuredisk-controller-69b978cbc6-wvmft_bdc14308-5f21-4fe5-933b-058b2633d611  Successfully provisioned volume pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7 Additionally, an Azure Resource of the type specified is created
```
This also confirms the name of the Azure resource being provisioned.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7m08f2dt93gqkpcvbnd0.png)

In the details, we observe that the disk PVC is utilized by the pod `diskpool-worker-lbxwt`. This pod accesses the Azure Disk via the PVC, directing all IO operations to the disk. Given its `system-node-critical` priority class, this pod is crucial.
```
$  kubectl get pods -n acstor
NAME                                                             READY   STATUS    RESTARTS      AGE
...
diskpool-worker-lbxwt                                            1/1     Running   0             14m

$ kubectl describe pod diskpool-worker-lbxwt -n acstor
Name:                 diskpool-worker-lbxwt
Namespace:            acstor
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      capacity-provisioner-pod-sa
Node:                 aks-agentpool-10525397-vmss000002/10.224.0.6
Start Time:           Tue, 05 Mar 2024 12:04:00 +0900
Labels:               acstor.azure.com/diskpool=training-azuredisk-diskpool-cbarq
                      acstor.azure.com/managedby=capacity-provisioner
                      acstor.azure.com/storagepool=training-azuredisk
                      app.kubernetes.io/component=storage-pool
                      app.kubernetes.io/instance=azurecontainerstorage-diskpool-worker
                      app.kubernetes.io/managed-by=capacity-provisioner
                      app.kubernetes.io/name=diskpool-worker
                      app.kubernetes.io/part-of=acstor
Annotations:          cni.projectcalico.org/containerID: a46436e3b06a0aaaac5a3c2a3f9f4a350d9562de1d874b99a083c32f3d26e536
                      cni.projectcalico.org/podIP: 10.244.2.8/32
                      cni.projectcalico.org/podIPs: 10.244.2.8/32
Status:               Running
IP:                   10.244.2.8
IPs:
  IP:           10.244.2.8
Controlled By:  DiskPool/training-azuredisk-diskpool-cbarq
Containers:
  capacity-provisioner-pod:
    Container ID:  containerd://f6f570b9dcccec1d4058fedfa75d0341ee24226bf028152cc05f118d731427a4
    Image:         mcr.microsoft.com/acstor/capacity-provisioner:v1.0.3-preview
    Image ID:      mcr.microsoft.com/acstor/capacity-provisioner@sha256:07c8585b8ecc37a27ad89d6efa8d400d109d823228b3090ca671a8d588942b5e
    Port:          <none>
    Host Port:     <none>
    Args:
      --mode=worker
      -zap-log-level=debug
    State:          Running
      Started:      Tue, 05 Mar 2024 12:04:14 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      MY_NODE_NAME:        (v1:spec.nodeName)
      MY_POD_NAME:        diskpool-worker-lbxwt (v1:metadata.name)
      RELEASE_NAMESPACE:  acstor (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from serviceaccount (ro)
    Devices:
      /dev/azure-cp from training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  training-azuredisk-diskpool-cbarq-diskpool-pvc-utmernhf
    ReadOnly:   false
  serviceaccount:
    Type:                     Secret (a volume populated by a Secret)
    SecretName:               capacity-provisioner-pod-sa
    Optional:                 false
QoS Class:                    BestEffort
Node-Selectors:               acstor.azure.com/io-engine=acstor
                              kubernetes.io/arch=amd64
Tolerations:                  node.kubernetes.io/not-ready:NoExecute op=Exists for 10s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 10s
Topology Spread Constraints:  topology.kubernetes.io/zone:DoNotSchedule when max skew 1 is exceeded for selector acstor.azure.com/storagepool=training-azuredisk
Events:
  Type    Reason                  Age   From                     Message
  ----    ------                  ----  ----                     -------
  Normal  Scheduled               15m   default-scheduler        Successfully assigned acstor/diskpool-worker-lbxwt to aks-agentpool-10525397-vmss000002
  Normal  SuccessfulAttachVolume  15m   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7"
  Normal  SuccessfulMountVolume   15m   kubelet                  MapVolume.MapPodDevice succeeded for volume "pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7/dev"
  Normal  SuccessfulMountVolume   15m   kubelet                  MapVolume.MapPodDevice succeeded for volume "pvc-0f4663a2-8ee1-48db-b0be-49cb0a1cf2f7" volumeMapPath "/var/lib/kubelet/pods/9063e076-1594-4bc9-ba80-238b576cdc8f/volumeDevices/kubernetes.io~csi"
  Normal  Pulling                 15m   kubelet                  Pulling image "mcr.microsoft.com/acstor/capacity-provisioner:v1.0.3-preview"
  Normal  Pulled                  15m   kubelet                  Successfully pulled image "mcr.microsoft.com/acstor/capacity-provisioner:v1.0.3-preview" in 1.863449186s (1.863461493s including waiting)
  Normal  Created                 15m   kubelet                  Created container capacity-provisioner-pod
  Normal  Started                 15m   kubelet                  Started container capacity-provisioner-pod
```
With our pool prepared, we can proceed to allocate storage using a Persistent Volume Claim (PVC) derived from the newly established storage class.
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-azuredisk-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: training-azuredisk
  resources:
    requests:
      storage: 1Gi
```
This action initiates a new PVC, which remains in a pending state until it is bound to a new consumer.
```
$ kubectl get pvc
NAME                     STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS         AGE
training-azuredisk-pvc   Pending                                      training-azuredisk   19s

$ kubectl describe pvc training-azuredisk-pvc
Name:          training-azuredisk-pvc
Namespace:     default
StorageClass:  acstor-training-azuredisk
Status:        Pending
Volume:
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age   From                         Message
  ----    ------                ----  ----                         -------
  Normal  WaitForFirstConsumer  0s    persistentvolume-controller  waiting for first consumer to be created before binding
```
We can then deploy our pod, which will request storage supported by the azurediskpvc PVC and supplied by the bound PV. Notably, the storage class name includes the namespace prefix. Consequently, our pod can store data on the provisioned Azure Storage resource.
```
apiVersion: v1
kind: Pod
metadata:
  name: training-pod
spec:
  nodeSelector:
    acstor.azure.com/io-engine: acstor
  volumes:
    - name: managedpv
      persistentVolumeClaim:
        claimName: training-azuredisk-pvc
  containers:
    - name: ubuntu-container
      image: ubuntu:latest
      command: ["/bin/bash", "-c", "--"]
      args: ["tail -f /dev/null"]
      volumeMounts:
        - mountPath: "/volume"
          name: managedpv
```
Finally, we can access the pod’s shell and create a new file, which will be stored on the corresponding Azure Disk resource.
```
$ kubectl exec -it training-pod -- sh
# ls
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  volume
# touch volume/sample.json
# ls volume
lost+found  sample.json
```
# Resources
- **Official Documentation:** [https://learn.microsoft.com/en-us/azure/storage/container-storage/container-storage-introduction](https://learn.microsoft.com/en-us/azure/storage/container-storage/container-storage-introduction)