Kubernetes. Replication and self-healing
Tags: docker, devops, kubernetes
Published: 2020-06-14 18:17:09

To start with, I would like to explain what does self-healing means in terms of Kubernetes. Self-healing is a fantastic feature of Kubernetes to recover from service or node failure automatically. In the following article, we will consider the benefit of using replication for your micro-services and how the Kubernetes cluster can automatically recover from a service failure. 

#Prerequisite
One of the great features of Kubernetes is the ability to replicate pods and their underlying containers across the cluster. So, before we set up our self-healing feature please make sure you have managed replication and here is a simple example of a deployment file that will deploy `nginx` container with replication factor `3`:
```
apiVersion: apps/v1
kind: Deployment-example
metadata:
  name: nginx-deployment-example
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.15.4
        ports:
        - containerPort: 80
 ```
All right, so let's create the deployment:
```
kubectl create -f deployment.yaml
```
Now, let's check whether our `nginx-deployment-example` was created:
```
 kubectl get deployments -n default
```
You should see your `nginx-deployment-example` deployment in the default namespace. If we want to see more details about those pods, please run the following command:
```
kubectl get pods -n default
```
We will see our 3 `nginx-deployment-example` pods:
```
NAME                                     READY   STATUS    RESTARTS   AGE
nginx-deployment-example-f4cd8584-f494x   1/1     Running   0          94s
nginx-deployment-example-f4cd8584-qvkbg   1/1     Running   0          94s
nginx-deployment-example-f4cd8584-z2bzb   1/1     Running   0          94s
```

#Self-healing
Kubernetes ensures that the actual state of the cluster and the desired statue of the cluster are always in-sync.  This is made possible through continuous monitoring within the Kubernetes cluster. Whenever the state of a cluster changes from what has been defined, the various components of Kubernetes work to bring it back to its defined state. This automated recovery is often referred to as self-healing.
So, let's copy one of the pods mentioned in the prerequisite and see what happens when we delete it:
```
kubectl delete pod nginx-deployment-example-f4cd8584-f494x
```
And after a few seconds, we see that our pod was deleted: 
`pod "nginx-deployment-example-f4cd8584-f494x" deleted`
Let's go ahead and list the pods one more time:
```
kubectl get pods -n default
NAME                                     READY   STATUS    RESTARTS   AGE
nginx-deployment-example-f4cd8584-qvkbg   1/1     Running   0          109s
nginx-deployment-example-f4cd8584-sgfqq   1/1     Running   0          5s
nginx-deployment-example-f4cd8584-z2bzb   1/1     Running   0          109s
```
And we see that the pod `nginx-deployment-example-f4cd8584-sgfqq` was automatically created to replace our deleted pod `nginx-deployment-example-f4cd8584-f494x`. And the reason is that `nginx` deployment is set to have 3 replicas. So, even though one of these was deleted, our **Kubernetes cluster works to make sure that the desired state is the actual state that we have**. 
So, now let's consider the case when there is an actual node failure in your cluster. First, let's check our nodes:
```
kubectl get nodes
```
You will see your Master and Worker nodes. Now, let's figure out what server our pods are running on. To do that, we have to `describe` it:
```
kubectl describe pod nginx-deployment-example-f4cd8584-qvkbg
```
Under the Events, you can see to which server the pod was assigned. We can also scroll up and under Node will also see where it has been assigned, which is of course will be the same server. Once you've identified all servers that are pods running on, you have to pick one server and simulate a node failure by `shutting down` the server.
Once the node has been shut down let us head back to the Master and check on the status of the nodes:
```
kubectl get nodes
```
We see that the cluster knows that one node is down. Let's also list our pods:
```
kubectl get pods -n default
```
You see that the state of the pod that was running on `"failed"` node is Unknown, but you see that another pod took its place. Let's go ahead and list the deployments:
```
kubectl get deployments -n default
```
And as we expected we have 3 pods available which are now in sync with our Desired amount of pods.
Now, let's try to describe the Unknown pod:
```
kubectl describe pod <pod_in_unknown_state>
```
You see that the Status of Unknown pod is `Terminating`, `Termination Grace Period` will be the `30s` by default and the reason for this is `NodeLost`. Also, you will see the messages that specifies that our node which was running our pod is unresponsive. 
Now, let's `start` our `"failed"` node and wait till it successfully rejoins the cluster.
Alright, once `"failed"` node is up and running, let's go ahead and check the status of our deployment:
```
kubectl get deployments -n default
```
We will see that our pod count is in sync. So, let's list our pods out:
```
kubectl get pods -n default
```
We will see that our Kubernetes cluster has finally terminated the old pod, and we are left with our desired count of `3` pods. 

#Conclusion#

As we see, Kubernetes takes a self-healing approach to infrastructure that reduces the criticality of failures, making fire drills less common. Kubernetes heals itself when there is a discrepancy and ensures the cluster always matches the declarative state. In other words, Kubernetes kicks in and fixes a deviation, if detected. For example, if a pod goes down, a new one will be deployed to match the desired state.