Kubernetes Storage Using Ceph
Tags: kubernetes, ceph, devops, opensource
Published: 2020-03-18 09:34:00

###__What is Kubernetes?__###
Kubernetes, the awesome container orchestration tool is changing the way applications are being developed and deployed. You can specify the required resources you want and have it available without worrying about the underlying infrastructure. Kubernetes is way ahead in terms of high availability, scaling, managing your application, but storage section in the k8s is still evolving. Many storage supports are getting added and are production-ready.

People are preferring clustered applications to store the data. But, what about the non-clustered applications? Where do these applications store data to make it highly available? Considering these questions, let’s go through the Ceph storage and its integration with Kubernetes.

###__What is Ceph Storage?__###
Ceph is open source, software-defined storage maintained by RedHat. It’s capable of block, object, and file storage. The clusters of Ceph are designed in order to run on any hardware with the help of an algorithm called CRUSH (Controlled Replication Under Scalable Hashing). This algorithm ensures that all the data is properly distributed across the cluster and data quickly without any constraints. Replication, Thin provisioning, Snapshots are the key features of the Ceph storage.

There are good storage solutions like Gluster, Swift but we are going with Ceph for following reasons:
               _1. File, Block, and Object storage in the same wrapper._
               _2. Better transfer speed and lower latency._
               _3. Easily accessible storage that can quickly scale up or down._

__We are going to use Ceph-RBD storage in this blog to integrate with Kubernetes.__

####__Ceph Deployment__####
Deploying highly available Ceph cluster is pretty straightforward and easy. I am assuming that you are familiar with setting up the Ceph cluster.

If you check the status, you should see something like:
![alt text](https://github.com/ajinkyaingole30/ceph-manual/blob/master/Screenshot%20(133).png?raw=true)

Here notice that my Ceph monitors hostnames are ip-10-0-0-58, ip-10-0-0-78 and ip-10-0-0-237

__There are few things you need to do once your ceph cluster is up and running.__

_create a pool_
```javascript
ceph osd pool create kubernetes 1024
```
_List pools_

```javascript
ceph osd lspools
```
![alt text](https://github.com/ajinkyaingole30/ceph-manual/blob/master/pool.png?raw=true)
_Create an RBD_image inside kubernetes pool_  
```javascript
rbd create kube --size 64 -p kubernetes
```
_View created RBD_image_
```javascript
rbd info kube -p kubernetes
```
_Remove image permissions_
```javascript
rbd feature disable kube -p kubernetes object-map fast-diff deep-flatten
```
_View created RBD_image and chech if permissions are removed or not_
```javascript
rbd info kube -p kubernetes
```

####__K8s Integration__####
After setting up the Ceph cluster, we would consume it with Kubernetes. I am assuming that your Kubernetes cluster is up and running. We will be using Ceph-RBD as a storage in Kubernetes.
####__Ceph-RBD and Kubernetes__####
We need a Ceph RBD client to achieve interaction between Kubernetes cluster and CephFS. 
_Check the repo https://github.com/ajinkyaingole30/K8s-Ceph.git_

__Setting up kubernetes master as ceph client so that we can use rbd_image as storage in Kubernetes__

_Copy ceph.repo in /etc/yum.repos.d/ and download ceph-common_ 
```javascript
cp ceph.repo /etc/yum.repos.d/ceph.repo
yum install ceph-common
```
_start and enable rbdmap service_
```javascript
systemctl start rbdmap
systemctl enable rbdmap
```
_Copy ceph.client,admin.keyring & also ceph.conf from ceph node and paste it in /etc/ceph of kubernetes-master (ceph client)._
_you can direct scp it from ceph node to kubernetes master_
```javascript
scp /etc/ceph/ceph.client.admin.keyring root@master-node:/etc/ceph/
scp /etc/ceph/ceph.conf root@master-node:/etc/ceph/
```
_Check if you can see the pools from kubernetes master (ceph client)_
```javascript
rados lspools
```
_Now map the rbd_image to your ceph client_
```javascript
rbd map kube -p kubernetes
```
_check mapped rbd_image_ 
```javascript
rbd showmapped

lsblk
```
__This client is not in the official kube-controller-manager container so let’s try to create the external storage plugin for Ceph.__

_download https://github.com/ajinkyaingole30/K8s-Ceph.git_

```javascript
git clone https://github.com/ajinkyaingole30/K8s-Ceph.git
```
_Generate a csi-config-map.yaml file similar to the example below, substituting the fsid for “clusterID”, and the monitor addresses for “monitors”:_
```javascript
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "671be3f7-44b4-a79d-07cf4ae58ee1",
        "monitors": [
          "10.0.0.58:6789",
          "10.0.0.78:6789",
          "10.0.0.237:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config
```
_Once generated, store the new ConfigMap object in Kubernetes:_
```javascript
kubectl apply -f csi-config-map.yaml
```
_Also generate a ksm-config.yaml_ 
```javascript
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    {
      "vault-test": {
        "encryptionKMSType": "vault",
        "vaultAddress": "http://vault.default.svc.cluster.local:8200",
        "vaultAuthPath": "/v1/auth/kubernetes/login",
        "vaultRole": "csi-kubernetes",
        "vaultPassphraseRoot": "/v1/secret",
        "vaultPassphrasePath": "ceph-csi/",
        "vaultCAVerify": "false"
      }
    }
metadata:
  name: ceph-csi-encryption-kms-config


kubectl apply -f kms-config.yaml
```

_ceph-csi requires the cephx credentials for communicating with the Ceph cluster. Generate a csi-rbd-secret.yaml file similar to the example below, using the newly created Kubernetes user id and cephx key:_
```javascript
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: admin
  userKey: AQD9o0Fd6hQRChAAt7fMaSZXduT3NWEqylNpmg==
```
_Once generated, store the new Secret object in Kubernetes:_
```javascript
kubectl apply -f csi-rbd-secret.yaml
```
__Configure ceph-csi plugins__

_Create the required ServiceAccount and RBAC ClusterRole/ClusterRoleBinding Kubernetes objects. These objects do not necessarily need to be customized for your Kubernetes environment and therefore can be used as-is from the ceph-csi deployment YAMLs:_
```javascript
kubectl apply -f csi-provisioner-rbac.yaml
kubectl apply -f csi-nodeplugin-rbac.yaml
```
_Finally, create the ceph-csi provisioner and node plugins. With the possible exception of the ceph-csi container release version, these objects do not necessarily need to be customized for your Kubernetes environment and therefore can be used as-is from the ceph-csi deployment YAMLs:_
```javascript
kubectl apply -f csi-rbdplugin-provisioner.yaml
kubectl apply -f csi-rbdplugin.yaml
```
__Using Ceph Block Devices create a storageclass__

_The Kubernetes StorageClass defines a class of storage. Multiple StorageClass objects can be created to map to different quality-of-service levels (i.e. NVMe vs HDD-based pools) and features._

_For example, to create a ceph-csi StorageClass that maps to the kubernetes pool created above, the following YAML file can be used after ensuring that the “clusterID” property matches your Ceph cluster’s fsid:_

```javascript
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 671be3f7-44b4-a79d-07cf4ae58ee1
   pool: kubernetes
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
mountOptions:
   - discard

kubectl apply -f csi-rbd-sc.yaml
```

__Create a Persistent Volume Claim__ 

_A PersistentVolumeClaim is a request for abstract storage resources by a user. The PersistentVolumeClaim would then be associated to a Pod resource to provision a PersistentVolume, which would be backed by a Ceph block image. An optional volumeMode can be included to select between a mounted file system (default) or raw block device-based volume._

_To create a block-based PersistentVolumeClaim that utilizes the ceph-csi-based StorageClass created above, the following YAML can be used to request raw block storage from the csi-rbd-sc StorageClass:_
```javascript
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 50Mi
  storageClassName: csi-rbd-sc

kubectl apply -f raw-block-pvc.yaml
```
_Check the pod if it is running or not_
```javascript
kubectl get pods
```
![alt text](https://github.com/ajinkyaingole30/ceph-manual/blob/master/1.pod.png?raw=true)
_Check if the pvc is bound_
```javascript
kubectl get pvc
```
_Check if the pv is bound_
```javascript
kubectl get pv
```
![alt text](https://github.com/ajinkyaingole30/ceph-manual/blob/master/1.pv.png?raw=true)

_To create a file-system-based PersistentVolumeClaim that utilizes the ceph-csi-based StorageClass created above, the following YAML can be used to request a mounted file system (backed by an RBD image) from the csi-rbd-sc StorageClass:_
```javascript
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 50Mi
  storageClassName: csi-rbd-sc


kubectl apply -f fs_pvc.yaml
```
_Check if the pvc rbd-pvc is bound or not_
```javascript
kubectl get pvc
```
_List Rbd images in kubernetes pools and again remove the permission of the new created rbd_
```javascript
# rbd ls -p kubernetes
# rbd feature disable csi-vol-db6a4c87-c4c8-11ea-87a0-721582595d51 -p kubernetes object-map fast-diff deep-flatten
```
_here "csi-vol-db6a4c87-c4c8-11ea-87a0-721582595d51" is the name of my new rdb image created after creating pvc._  
![alt text](https://github.com/ajinkyaingole30/ceph-blogadta/blob/master/Screenshot%20(345).png?raw=true)


_The following demonstrates and example of binding the above PersistentVolumeClaim to a Pod resource as a mounted file system:_
```javascript
---
apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod
spec:
  containers:
    - name: web-server
      image: nginx
      volumeMounts:
        - name: mypvc
          mountPath: /var/lib/www/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

kubectl apply -f pod.yaml
```
_Check the pod if it is running or not_
```javascript
kubectl get pods
```
![alt text](https://github.com/ajinkyaingole30/ceph-manual/blob/master/1.pod.png?raw=true)
_Get into the pod to check if the volume is attached_ 
```javascript
kubectl exec -it ceph-pod -- df -hT | grep /dev/rbd0
```
![alt text](https://github.com/ajinkyaingole30/ceph-manual/blob/master/1.podexec.png?raw=true)

___Using this you can integrate ceph with kubernetes___
![alt text](https://github.com/ajinkyaingole30/ceph-manual/blob/master/1.exec.png?raw=true)

I hope you enjoyed the article and found the information useful. Please share your feedback.

 Happy Cephing!