Kubernetes: Storage & Security
Tags: kubernetes, security, devops, kcdchennai
Published: 2023-05-18 10:48:47



## Introduction✍️

It's time to share Secrets!🤫

Obviously not mine.😜

In this blog let's talk about the **storage**, storage **classes**, **security** policies, **network** policies, security layers, and everything that comes under the store room and the security room of the Kubernetes. These topics play a very crucial role in the making of k8s clusters. Let's see how it does!

## Storage🛢️
### Volumes🎚️
- In Kubernetes, the pods are not permanent. Once a pod is created, it will be destroyed when the requirement finishes and one comes up in place of it. Just like docker containers, the data processed inside the pod also get deleted when the pod destroys. This will occur data loss which is a big problem.

- To resolve this problem, volumes come up

- We attach a volume to the pod. So now, whenever a pod processed some data inside it, it gives also gets stored in the volume. If now pod is going to be deleted we have still our data alive.

- The data generated by the pod is now stored in the volume!

Let's create a simple single node k8s cluster `volume-specific` file:

```
apiVersion: v1
kind: Pod
metadata:
  name: randon-number-generator
spec:
  container:
  - image: alpine
    name: alpine
    command: ["/bin/sh","-c"]
    args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"]
    volumeMounts:
    - mountPath: /opt
      name: data-volume
  volumes:
  - name: data-volume
    hostPath:
      path: /data
      type: Directory

```
In the above yaml file, we are generating a random number in a given range so that we can save that number in our volumes. `volumeMounts` are used to get the same data within the pod. `hostPath` is simply giving the `path directory` where the data will be stored.

This is how even after pod deletion we can still have our data in `/data dir`.

_Note: This is only for single-node k8s clusters._

- When you will work on a large-scale project, there will not be a single node cluster. Hundreds of Pods running at a time and giving the volume path as /data to all the pod's data is not recommended in the multi-node cluster for obvious reasons.

- This is because the PODs would use the /data directory on all the nodes.

- So we need proper storage solutions. Kubernetes supports different standard storage solutions like NFS, glusterFS and many more.

You will get a basic idea of how to define them in the further topics.

#### Persistent Volumes📻
- Whenever we create volumes, every time we need to configure them in a pod-definition file. So every configuration information is required to configure storage within the file.

- Now imagine we are running hundreds of pods and each time when a user wants to deploy the pods, they would have to configure storage every time for each pod in their environment.

- So doing this every time is not a best practice for us.

- Here, **Persistent Volumes** come up. It is a cluster-wide room of storage volumes configured by an administrator to make use of the users deploying applications on the cluster.

- Now we can use the storage using persisting volume claims

Let's create a definition yaml file named pv-definition.yaml for this:

```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv1
spec:
  accessModes:
    - ReadWriteOnce
  capacity: 
    storage: 1Gi
  awsElasticBlockStore:
    volumeID: <volume-id>
    fsType: ext4

```
There can be different kinds of `accessMode` like,

- ReadOnlyMany

- ReadWriteOnce

- ReadWriteMany

`awsElasicBlockStore` is one of the supported storage solutions we talked about above for the multi-node cluster. This will provide specific `volume-id` and `type` instead of just `/data` dir path to differentiate better.

Now run,

```
kubectl create -f pv-definition.yaml
```
To check the created persistent volumes, you know what command to use now:

`kubectl get persistentvolume`

#### Persistent Volume Claims📑

- After creating the persistentvolume, it's the time to create a persistent volume claim to make the storage available to a node.

- Persistent volumes and Persistent volume claims are two separate objects in k8s.

- An administrator creates a persistent volume and a user creates a persistent volume claim to use the storage.

- Kubernetes binds the persistent volume of the claims as per the requests and required properties set on the volumes.

- Kubernetes checks the sufficient capacity while binding volumes to the claims.

- It also uses labels & selectors to bind on a specific persistent in case of multiple possible matches to the right volumes.

Let's create a claim definition file named `pvc-definition.yaml` now,

```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources: 
    requests:
      storage: 500Mi

```
When this file is created, Kubernetes looks for the `volume` file which was created above.

`kubectl create -f pvc-definition.yaml`

The `accessModes` match, then it checks for the storage capacity and here to store `500Mi` in `1Gi` is the perfect match for them since there is no other available.

So the claim is bound to the volume.

To check the claims file use the get command:

`kubectl get persistentvolumeclaim`

### Storage Classes🗑️

- We have created the persistent volumes and persistent volume claims but before creating this volume you must have created a disk on google cloud.

- You have to manually provision the disk whenever your application needs storage on `gc`. And then manually create a persistent volume file using the same name defined during the creation of disk specifications.

- This whole process is called Static Provisioning Volumes.

To automate this process fully, we have Storage Class.

- You just have to define a provisioner like Google Storage and then everything will be seen by the provisioner. It automatically provisions the storage and attaches that to the pod when a claim is made.

- This is called Dynamic Provisioning Volumes.

- Now you don't have to create persistent volume manually.

- There are many storage provisioners such as **AWSEBS**, **AzrueFile**, **AzureDisk**, **CephFS** and many more

Take a look at a storage class definition file as `sc-definition.yaml`:

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: google-storage
provisioner: kubernetes.io/gce-pd

```
After defining this file, you have to give the storage class name in `pvc` file as the same as you have defined in the `sv` file as `google-storage` in this case.

You can create different kinds of classes in storage using different types of disks like

- Silver🔘 Storage Class with the standard disk,

- Gold🟠 Storage Class with SSD drives and

- Platinum⚪ Storage Class with SSD drives and replication.

### StatefulSets🏗️

- It creates pods based on the templates.

- It can scale up and scale down as per the requirements. And can perform rolling updates and rollbacks.

- Like, you want to deploy the master-node first then the worker-node-1 up completely then starts running and after that worker-node-2 will be up and run itself into the k8s cluster. StatefulSet can help you to achieve this.

- In this, when a pod goes down and a new pod comes up, it will be the same name that you have specifically defined for that pod.

- It maintains an identity for each of its pods that helps to maintain the order of the deployment of your pods.

- If you want to use StatefulSets just for the sake of identity purposes for the pods and not for the sequential deploying order then you can manually remove the commands of the order maintenance, just you have to make some changes in the YAML file.

> Note: If you have already learned this topic in my previous blog then feel free to skip this topic😇. Otherwise, continue learning!✊

Though you don't need to use this StatefulSet. As it totally depends on your need for the application. If you have servers that require an order to run or you need a stable naming convention for your pods. Then this is the right choice to use.

You can create a `StatefulSet` yaml file just like the deployment file with some changes like the main one is kind as `StatefulSet`. Take a look:

```
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  labels:
        app: msql
spec:
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql
  replicas: 3
  selector:
      matchLabels:
          app: mysql
  serviceName: mysql-h

```
than run:

`kubectl create -f statefulset-definition.yml`

To scale up or down use the `scale` command with the numbers you wanted to scale:

`kubectl scale statefulset mysql --replicas=5`

This is how you can work with **StatefulSets**.

## Security🕵️
### RBAC (Role-Based Access Control)🛂

-As the name suggests itself the working, it is used to define roles to users or a group and bind them who granted those permissions.

- Simply, granting the permissions for who will do what.

Let's quickly see how to create a `role-definition` file:

```
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""] # "" indicates the core API group
  resources: ["ConfigMap"]
  verbs: ["create"]

```
In the above file, we are creating the role for a developer who has resources as pods in which they can modify the pods and can allow to configure them.

Now run `create` command to create the role:

`kubectl create -f developer-role.yaml`

Now, link the user to that role. For this create another object file called **RoleBinding**:

```
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: devuser-developer-binding
subjects:
- kind: User # "" indicates the core API group
  name: dev-user
  apiGroups: rbac.authorization.k8s.io
roleRef: 
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io

```
Now run `create` command to bind this:

`kubectl ceate -f devuser-developer-binding.yaml`

To view the created roles, run the `get` command:

`kubectl get roles`

To see the bindings you have created run:

`kubectl get rolebindings`

To view the role in detail run `describe` command:

`kubectl describe role developer`

To check your accessibility for a particular object, run the:

`kubectl auth can-i create deployments`

By this, you can check access that you want to know like `delete nodes`

### Pod Security Policies (PSPs)📦

- It is a security policy defined for the users or a group to access the pods in the Kubernetes cluster.

- Yes! You guessed it right. It is one of the RBAC to grant permissions and bind them with whom granted with.

- It defines a set of security policies that are applied to the pods based on their labels and annotations.

- When a pod is created, Kubernetes checks for the specific labels and annotations which we have defined above in the file to proceed it from creating to running state.

> Note: Removed feature
> PodSecurityPolicy was deprecated in Kubernetes v1.21, and removed from Kubernetes in v1.25.

Instead of using PodSecurityPolicy, you can enforce similar restrictions on Pods using either or both:

- [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/)

- a 3rd party admission plugin, that you deploy and configure yourself

These new Pod Security Standards define three different policies to broadly cover the security spectrum.

These policies are _cumulative_ and range from **highly-permissive to highly-restrictive**.

Learn more [here](https://kubernetes.io/docs/concepts/security/pod-security-standards/)!

### Secrets🧑‍💻

Finally, it's time to decrypt the secrets in Kubernetes.

Never tell your secrets to anyone. Ok!

But what is meant by secrets in k8s?

- Let's say, you have a simple web application that is connected to a database that displays a successful message on the screen while getting connected.

- First, you have coded your user name and passwords into the source code. Then you understand that it's not a good idea to provide credentials like this.

- Then you create an object file called ConfigMap and put those values inside the yaml file. ConfigMap stores data in a plain text format. And that's again not a good idea.

So here, **Secrets** come in. It is used to store sensitive information which you don't want to share it with anybody.

It is similar to `configMaps` but it stores the information in encoded format instead of plain text.

So first encode your data. To do so, run:

```
$ echo -n 'mysql' | base64
bXlzcWw=

```
vice-versa for decoding the text:

```
$ echo -n 'bXlzcWw=' | base64 --decode
mysql

```
To check secrets, run:

`kubectl get secrets`

To get detailed information on secrets, run:

`kubectl describe secrets`

Now, create a `secret-definition.yaml` file:

```
apiVersion: v1
kind: Secret
metadat:
  name: myapp-secret
data:
  DB_Host: bXlzcWw=
  DB_User: cm9vdA==
  DB_Password: cGFzd3Jk

```
Now configure this secret with a `pod-definition.yaml` file:

```
apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
  labels:
    name: simple-webapp-color
spec:
  containers:
  - name: simple-webapp-color
    image: simple-webapp-color
    ports:
      - containerPort: 8080
    envFrom:
      - secretRef: 
          name: myapp-secret

```
Now run:

`kubectl create -f pod-definition.yaml`

This is how you can create your own secret file and configure it in the `pod-definition` file by adding the `envFrom` section.

And list as many variables as you want as per your need. The name should be matched with the one you have created in the `secret-definition` file. In this case, it is `myapp-secret`

### Network Policies📋

- In Kubernetes, every routing network traffic instruction is set by the network policies.

- A set of tools that defines the communication between the pods in a cluster.

- It is a powerful tool used for the security of network traffic in k8s clusters.

- Allowance of traffics from one specific pod to another one.

- Restricting traffic to a specific set of ports and protocols.

- Implementation is done by Network APIs.

- It can be applied to namespaces or individual pods.

A simple network policy yaml file:

```

    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: demo-network-policy
    spec:
      podSelector:
        matchLabels:
          app: my-app
      policyTypes:
      - Ingress
      ingress:
      - from:
        - podSelector:
            matchLabels:
              app: allowed-app
        ports:
        - protocol: TCP
          port: 80

```

For the deployment of the Network Policy YAML file, use the `kubectl` apply command:

`kubectl apply -f demo-network-policy.yaml`

This is how you can define your own network policies.

### TLS (Transport Layer Security)🌐

- Every communication over the internet must be secure. Otherwise, there is a high risk of hackers hacking the data which we are transferring through it.

- Likewise in Kubernetes clusters, there is a set of master nodes and worker nodes who communicates thoroughly.

- The communications between all the pods, nodes and API servers must be secured and encrypted.

- Administrators trying to communicate with the master node via kubelet or through APIs directly, everything must be fully secure.

- The communication between the servers and the clients must be secure and encrypted.

- To fulfill all these requirements we need security certificates. We all know about TLS certificates and how it works.

- It works with an asymmetric key format where every end has its own key-value pair lock-key to decrypt the data.

Now let's understand how TLS can be defined in Kubernetes.

In Kubernetes, it has two sides, one is the server side and another is the client side. Both sides must have security certificates signed by CA (Certificate Authority) to verify their identity. Let's look at both of them.

#### Server Certificates of the Servers🔏

1. **KUBE-API Server** - This helps the k8s to expose the HTTP service that other components and external users use to manage the cluster. So it is an important component that must be secured enough.
 It has a certificate and a `key-pair` named `apiserver.cert` 
 and `apiserver.key`

2. **ETCD Server** - It stores all the information about the clusters so generate a certificate and key-value pair for this also. The naming conventions are
 `etcdserver.cert` and `etcdserver.key`

3. **KUBELET Server** - It belongs in the worker node which also exposes the HTTP API endpoints to interact with others. Their cert and key-value are

`kubelet.cert` and `kubelet.key`

#### Client Certificate of the Clients🔐

1. **The Administrator** - The clients who access the services are the admins. It also requires a certificate and a `key-value` pair for access. Naming them as
`admin.cert` and `admin.key`

2. **The Schedulers** - It is another client who communicates with the `kube-api` server to schedule the objects as per the requirements. So it also needs verification to talk to the server.
Naming them as `scheduler.cert` and `scheduler.key`

3. **KUBE-CONTROLLER MANAGER** - It also communicates with the `kube-api` server and for the authentication it requires the same security checks.
 `controller-manager.cert` and `controller-manage.key`

4. **KUBE-PROXY** - Another client-side component has the 
 certificates and key
 `kube-proxy.cert` and `kube-proxy.key`

Now it's time to generate a certificate for our cluster. There are many tools to do so like **EASYRSA**, **OpenSSL**, **CFSSL** and many more. We will be using **OpenSSL** in this.

Create a private key using the `openssl` command:

`$ openssl genrsa -out apiserver.key 2048`

It will generate a key called `apiserver.key`

Now use the `request` command to generate a certificate signing request for the previous one

```
$ openssl req -new -key apiserver.key -out apiserver.csr -subj "/CN=kube-apiserver"
  $ openssl x509 -req -in apiserver.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out apiserver.crt -days 365

```
Now, create a secret to store the private key and certificates

```
$ kubectl create secret tls apiserver-certs --key=apiserver.key --cert=apiserver.crt -n kube-system

```
Modify the Kubernetes API server configuration to use the **TLS certificates**:

Now edit the file of api server config to use `tls` cert:

```
  $ vi /etc/kubernetes/manifests/kube-apiserver.yaml

  spec:
    containers:
    - name: kube-apiserver
      volumeMounts:
      - mountPath: /etc/kubernetes/pki/apiserver
        name: apiserver-certs
        readOnly: true
    volumes:
    - name: apiserver-certs
      secret:
        secretName: apiserver-certs

```

Restart the k8s api server to apply the new config

`$ systemctl restart kubelet`

This is how you can generate a certificate and configure TLS for the Kubernetes API server.

---

Thank you!🖤