How To Use Pre-existing Disk as a K8s Persistent Volume for a GKE Cluster
Tags: devops, kubernetes, gke
Published: 2021-12-12 22:09:28

_Updated on December 17th 2021._

Managing storage for implementing stateful set solutions is a distinct problem from managing compute instances. 

##🎿 0a. What's under the hood##

There's no emoji for "only _one_ lost ski", so the author will use a common icon for this _theoretical_ part of the article. ☃️

In Kubernetes terms, the `PersistentVolume` subsystem [provides an API](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this, the K8s team introduced two API resources: `PersistentVolume` and `PersistentVolumeClaim`.

A `PersistentVolume` (PV) is **a piece of storage in the cluster** that has been provisioned by an administrator or dynamically provisioned using [Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/). It is a resource in the cluster, just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system. From K8s version 1.11b `PersistentVolume`s can be even configured [to be expandable](https://github.com/kubernetes/enhancements/issues/284).

![PVs' basics](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fu8i88mxc22sy6kbikhn.png)

A `PersistentVolumeClaim` (PVC) is **a request for storage by a user**. It is similar to a pod. Pods consume node resources and PVCs consume PV resources.

Let's look at 3 (three) methods to get PV + PVC pairs tied to each other robustly.

##🎿 I. Volume Name: yes, we need a hack for GKE, again##

And the easiest one is about `volumeName`!

There are situations _when in your GKE environment you already have [a persistent disk](https://cloud.google.com/compute/docs/disks/add-persistent-disk)_ (say named `pd-name`), and, let's imagine obviously you want to use strictly this disk and do not want to create a new one. 

The manifest file below describes a corresponding `PersistentVolume` and `PersistentVolumeClaim` that your cluster can use. 
```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-demo
spec:
  storageClassName: ""
  capacity:
    storage: 50G
  accessModes:
    - ReadWriteOnce
  gcePersistentDisk:
    pdName: pd-name
    fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pv-claim-demo
spec:
  storageClassName: ""
  volumeName: pv-demo
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50G
```
⚠️ Note: it is necessary to specify `""` as the `storageClassName` so that the default storage class [won't be used](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1).

Now you may use `kubectl apply -f existing-pd.yaml` command as described above to create the `PersistentVolume` and `PersistentVolumeClaim`.

Pods access storage by using the claim as a volume. _Claims must exist in the same namespace as the pod using the claim._ The cluster finds the claim in the pod's namespace and uses it to get the `PersistentVolume` backing the claim. The volume is then mounted to the host and into the Pod.

After the `PersistentVolume` and `PersistentVolumeClaim` exist in the cluster, you can give a Pod's containers access to the volume by specifying values for the container's `volumeMounts.mountPath` and `volumeMounts.name`, as shown in the following example:

```yaml
kind: Pod
apiVersion: v1
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
       claimName: pv-claim-demo
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: task-pv-storage
```
This way, when you apply this manifest to a cluster, the Pod is created, and the `task-pv-container` container has access to the volume in its `/usr/share/nginx/html/` directory.

##🎿 0b. Formalities matter! We have to know what we want to do##

In GKE, for a container to access your pre-existing persistent disk, you'll need [to do the following](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/preexisting-pd):

* Provision the existing persistent disk as a `PersistentVolume`.
* Bind the `PersistentVolume` to a `PersistentVolumeClaim`.
* Give the containers in the Pod access to the `PersistentVolume`.

As you may have guessed already, there are several ways to bind a `PersistentVolumeClaim` to a specific `PersistentVolume`. For example, the following YAML manifest creates a new `PersistentVolume` and `PersistentVolumeClaim`, and then binds the claim to the volume using a `claimRef` (see below), which ensures that the `PersistentVolume` can only be bound to that `PersistentVolumeClaim`.

☑️ Claims will remain unbound indefinitely _if a matching volume does not exist_. Claims will be bound as matching volumes become available. For example, a cluster provisioned with many 50Gi PVs would not match a PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added to the cluster.

☑️ If you know exactly what `PersistentVolume` you want your `PersistentVolumeClaim` to bind to, you can specify the PV in your PVC using the `volumeName` field. This method _skips_ the normal matching and binding process. _The PVC will only be able to bind to a PV that has the same name specified in_ `volumeName`. If such a PV with that `volumeName` exists and is `Available`, the PV and PVC will be bound _regardless of whether_ the PV satisfies the PVC’s label selector, access modes, and resource requests.

To bind a `PersistentVolume` to a `PersistentVolumeClaim`, the `storageClassName` of the two resources must match, as well as capacity, `accessModes`, and `volumeMode`. 

⚠️ Please note again, you can omit the `storageClassName`, but you must specify `""` to prevent Kubernetes from using the default `StorageClass`.

The `storageClassName` does not need to refer to an existing StorageClass object. If all you need is to bind the claim to a volume, you can use any name you want. However, if you require extra functionality configured by a StorageClass, like volume resizing, then `storageClassName` **must refer to an existing** StorageClass **object**.

## 🎿 II. Roulette or poker? A magic claim reference ##

It’s important to understand that the application developer can create the manifests for the Pod and the `PersistentVolumeClaim` objects without knowing anything about the infrastructure on which the application will run. Similarly, the cluster administrator can provision a set of storage volumes of varying sizes in advance without knowing much about the applications that will use them. You may see some schematics about this process' internals below. 

Kubernets [can operate with](https://kubernetes.io/docs/concepts/storage/volumes/) different network file systems, not only with NFS, of course.

![PV + PVC + cluster admin](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z1oc30m5dgye12o1e4ix.png)

You may also want your cluster administrator to _"reserve"_ the volume for _only your claim_ so that nobody else’s claim can bind to it before yours does. In this case, the administrator can specify the PVC in the PV using the `claimRef` field. 

```yaml
apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
  name: "claim1"
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: "1Gi"
  volumeName: "pv0001"
```

The PV will only be able to bind to a PVC that has the same name and namespace specified in `claimRef`. The PVC’s access modes and resource requests must still be satisfied in order for the PV and PVC to be bound, though the label selector is ignored. Persistent Volume Object Definition with `claimRef` example is shown below:

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  nfs:
    path: /tmp
    server: 172.17.0.2
  persistentVolumeReclaimPolicy: Recycle
  claimRef:
    name: claim1
    namespace: default
```
Specifying a `volumeName` in your PVC does not prevent a different PVC from binding to the specified PV before yours does. Your claim will remain `Pending` until the PV is `Available`.

☑️ It should be kept in mind that specifying a `claimRef` in a PV _does not prevent_ the _specified PVC_ from _being bound to a different PV_! The PVC is free to choose another PV to bind to according to the normal binding process. Therefore, _to avoid_ these scenarios and _ensure_ your claim gets bound _to the volume you want_, you must _ensure_ that _both_ `volumeName` and `claimRef` are specified.

You can tell that your setting of `volumeName` and/or `claimRef` influenced the matching and binding process by inspecting a `Bound` PV and PVC pair for the `pv.kubernetes.io/bound-by-controller:` annotation. The PVs and PVCs where you set the `volumeName` and/or `claimRef` yourself will have no such annotation, but ordinary PVs and PVCs will have it set to "`yes`". See GitLab's [article](https://docs.gitlab.com/charts/advanced/persistent-volumes/) related to such situation.

When a PV has its `claimRef` set to some PVC name and namespace, and is reclaimed according to a Retain or Recycle reclaim policy, its `claimRef` will remain set to the same PVC name and namespace even if the PVC or the whole namespace no longer exists.

⚠️ The ability to set `claimRef`s is a temporary workaround for the described use cases. A long-term solution for limiting who can claim a volume is in development. The cluster administrator should first consider configuring selector-label volume binding before resorting to setting `claimRef`s on behalf of users.

##🎿 III. Selectors & lables, or Wait, I have everything written in advance!##

So, there is a more elegant approach to enable binding of persistent volume claims (PVCs) to persistent volumes (PVs) via `selector` and `label` attributes. By implementing selectors and labels, regular users are able to target provisioned storage by identifiers defined by a cluster administrator.

Why it can be needed? In cases of statically provisioned storage, developers seeking persistent storage are required to know a handful identifying attributes of a PV in order to deploy and bind a PVC. This creates several problematic situations. Regular users might have to contact a cluster administrator to either deploy the PVC or provide the PV values. PV attributes alone _do not convey the intended use of the storage volumes, nor do they provide methods by which volumes can be grouped_.

Selector and label attributes can be used _to abstract away PV details from the user_ while providing cluster administrators a way of identifying volumes by a descriptive and customizable tag. Through the selector-label method of binding, users are _only required to know which labels are defined_ by the administrator. Let's look at Persistent Volume with Labels example:

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gluster-volume
  labels: 
    storage-tier: gold
    aws-availability-zone: us-east-1
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster 
    path: myVol1
    readOnly: false
  persistentVolumeReclaimPolicy: Retain
```

Use _labels_ to identify common attributes or characteristics shared among volumes. In this case, we defined the [Gluster](https://docs.gluster.org/en/latest/Quick-Start-Guide/Quickstart/) volume to have a custom attribute (key) named `storage-tier` with a value of `gold` assigned. A claim will be able to select a PV with `storage-tier=gold` to match this PV.

And as a next step we need to compose a claim, a Persistent Volume Claim with Selectors. A claim with a _selector_ stanza (see example below) _attempts to match existing, unclaimed, and non-prebound_ PVs. The existence of a PVC selector _ignores_ a PV’s _capacity_. However, `accessModes` are _still_ considered in the _matching_ criteria.

⚠️ It is important to note that a claim must match all of the key-value pairs included in its selector stanza. If no PV matches the claim, then the PVC will remain unbound (`Pending`). A PV can subsequently be created and the claim will automatically _check for a label match_.
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gluster-claim
spec:
  accessModes:
  - ReadWriteMany
  resources:
     requests:
       storage: 1Gi
  selector: 
    matchLabels:
      storage-tier: gold
      aws-availability-zone: us-east-1
```
##🎿 IV. What if we don't want to produce useless entities?##

Real life often comes with surprises. What if you can’t recover a data from a Pod that was using a PersistentVolume or you're looking for ways to safely delete pods without affecting storage stored in a PV?

![GCP GKE PV Edit button](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/42ihjq5oe6y19k0qkkxg.png)

⚠️ Make sure the PersistentVolume `ReclaimPolicy` [is set](https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/) to `Retain`. For dynamically provisioned PersistentVolumes, the default reclaim policy is `Delete`. This means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim. This automatic behavior _might be inappropriate if the volume contains precious data_.  

If `ReclaimPolicy` is currently set to `Delete`, **which is a common logic in GKE to be applied for pods** (look at `spec.persistentVolumeReclaimPolicy: Delete` in YAML) you can  patch the PV by issuing a command:
```
kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
```
Next, proceed to delete the resources that was using that PV. The `STATUS` of the PV will then become `Released` after the PVC/Pod hold have been deleted.

In a Released state, edit the PV and remove `spec.claimRef` block:
```
apiVersion: v1
kind: PersistentVolume
...
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  claimRef:
...
```
After removing the `spec.claimRef` block, the PersistentVolume will be available. You'll be capable to proceeed re-adding your application involving the PV we discussed above.

##How to check what we've done at all##

The `kubectl get pv` and `kubectl get pvc` commands can be used to see what `PersistentVolume` and `PersistentVolumeClaim` have been defined for the application. If the `STATUS` is `Bound`, the application _do have_ access to the necessary storage. The `kubectl describe pv` command may be used to see detailed information about the Persistent Volume used by the application.

The `kubectl describe pv` command is used to see detailed information about the Persistent Volume used by the application. An example (thanks [IBM](https://www.ibm.com/docs/hr/mvi/1.1.1?topic=environment-checking-kubernetes-storage-status)) is shown below:
```
# kubectl describe pv
NAME                  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                             STORAGECLASS   REASON    AGE
powerai-vision-data   40Gi       RWX            Retain           Bound     default/powerai-vision-data-pvc                            48d
[root@dlf01 ~]# /opt/powerai-vision/bin/kubectl.sh describe pv
Name:            powerai-vision-data
Labels:          assign-to=powerai-vision-data
                 type=local
Annotations:     pv.kubernetes.io/bound-by-controller=yes
StorageClass:    
Status:          Bound
Claim:           default/powerai-vision-data-pvc
Reclaim Policy:  Retain
Access Modes:    RWX
Capacity:        40Gi
Message:         
Source:
    Type:  HostPath (bare host directory volume)
    Path:  /opt/powerai-vision/volume/
Events:    <none>
```
The `kubectl describe pvc` command is used to see detailed information about the Persistent Volume Claim for the application.
```
# kubectl describe pvc

Name:          powerai-vision-data-pvc
Namespace:     default
StorageClass:  
Status:        Bound
Volume:        powerai-vision-data
Labels:        app=powerai-vision
               chart=ibm-powerai-vision-prod-1.1.0
               heritage=Tiller
               release=vision
Annotations:   pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
Capacity:      40Gi
Access Modes:  RWX
Events:        <none>
```
Respectively and again, the above output shows more details about the Persistent Volume Claim used by the application. The `Volume` section references the underlying Persistent Volume, and the `STATUS` should be `Bound` if it has been successfully allocated to the application. The Events section will show if there were issues with the Persistent Volume Claim.

Also, the author is strongly excited with Utku Özdemir's `pv-migrate`, a CLI tool/`kubectl` [plugin to easily migrate](https://github.com/utkuozdemir/pv-migrate) the contents of one Kubernetes `PersistentVolume[Claim]` to another.

YAML examples of GKE Pod with Persistent Volume (+ WebUI screenshots) may be found [here](https://devopscube.com/persistent-volume-google-kubernetes-engine/) (thanks to DevOpsCube team).

![Pingu like to ski!](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z9enban60vm0rz0b3gao.jpeg)

> Sources: RedHat, loft.sh, K8s docs, GitLab, GitHub, letsdoclod.com by Erick Gubi
📖 "Kubernetes In Action" by Marko Lukša, ISBN 9781617293726