Pod Disruption Budget in Kubernetes
Tags: kubernetes, cloudnative, pods
Published: 2023-07-03 14:32:49

Maintaining highly available applications in a Kubernetes cluster can be hard, especially when nodes go under maintenance or fail. The application pods get terminated or may reschedule on other nodes, which can cause downtime or data loss. 

To make the applications available **24*7** even during system disruptions, a feature was introduced in Kubernetes known as [Pod Disruption Budget](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets). But before that, let’s dive into the different disruptions a system can face.

# What are Disruptions?

[Disruption](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/), in general, means to break a process, and in terms of Kubernetes, pod disruption means to terminate the pod running on a cluster node if a node fails/upgrades or someone destroys it. 

There are two types of disruptions:

- Involuntary Disruptions
- Voluntary Disruptions

## Involuntary Disruptions
These disruptions are unavoidable and occur mainly due to hardware or software errors. Some of its examples are:

- Hardware failure of the node.
- The cluster admin deletes the node accidentally.
- Kernel-related problem.
- Cloud provider or hypervisor-related failure makes VM disappear.
- Node gets disappear due to cluster network partitions.
- Not enough resources left on a node.
 
## Voluntary Disruptions
These disruptions occur by the application owner or cluster administrator.

An application owner can do the following:

- Deletes the deployment/controller managing the pods.
- Updating the deployment pod’s template causes a restart.
- Accidentally deleting a pod.

A cluster administrator can do the following:

- [Draining a node](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/) for upgrade and maintenance.
- Drain the node to scale down the cluster.
- Removing pod from node to schedule some other pod on it.

These are some of the disruptions which take applications down and give users downtime. But let’s see how to deal with these disruptions.

# How to Deal with Disruptions?
To deal with involuntary disruptions, one can follow the below solutions:

- Make sure your pod [requests the resources](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) it needs and not more.
- Create enough replicas of your application to make it more available(HA).
- Even in HA, use [anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) to have pods on all cluster nodes or zone if using multi-zone clusters.  

Similarly, to deal with voluntary disruptions mainly caused by cluster admin actions such as draining a node for maintenance and scaling down a cluster. One can use [Pod Disruption Budget (PDB)](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets) to make applications always available.

# Pod Disruption Budget
_[Pod Disruption Budget (PDB) ](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets)is an object created by the application owner that defines the minimum number of application replicas that should run during voluntary disruptions (node upgrade/maintenance) to make it highly available._

Let’s understand with an example, Let’s say you are running a deployment with 6 replicas and have created a PDB which should have 4 replicas to run always in case of voluntary disruptions. Then the eviction API will allow the disruption of two pods at a time. 
## Features of  Pod Disruption Budget
Below are the following features of PDB:

- The application owner creates the PDBs.
- It helps the operations team to manage the cluster while the application is always available.
- Provides an interface between cluster admin and application owner to work smoothly.
- [Eviction API](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#create-eviction-pod-v1-core) respects it.
- It defines the availability requirements.
- It works on Deployment, ReplicaSet, ReplicationController, and StatefulSet objects.

## Pod Disruption Budget Fields
There are three main fields in PDB:

- `.spec.selector` in PDB denotes the set of pods on which it is applied. It is the same as the application controller's label selectors. 
- `.spec.minAvailable` denotes the number of pods that must be available after eviction. It can be an absolute number or percentage.
- `.spec.maxUnavailable` denotes the number of pods that can be unavailable after eviction. It can be an absolute number or percentage.

**One can only specify the minAvailable or maxUnavailable field in a single pod disruption budget, not both.**

Let’s see in more detail about PDB by creating a deployment and PDB on a multi-node cluster and draining one node.

## Example of Pod Disruption Budget

- You can deploy a local multi-node cluster with Kind or use managed Kubernetes services. I have used the EKS cluster for this demo.

```
kubectl get nodes
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sp05cdmo5xdte0sf0mfz.JPG)

- PDB is a **namespaced-scope** resource and belongs to api group **policy** with the **v1** version.
```
kubectl api-resources | grep pdb
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g7zm1lremp6eoyigzsbj.JPG)

- Below is the nginx-deployment yaml configuration with 8 replicas and **app:nginx** as label selector.
```
# nginx-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deploy
  labels:
    app: nginx
spec:
  replicas: 8
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
```
```
kubectl apply -f nginx-deploy.yaml
kubectl get deploy
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lxnkhcbxeayvbgyfsgh6.JPG)

- Verify pods are scheduled on both nodes

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u0w8zdfdn3cz1zdg4fn5.JPG)

### Use Case: 1

- Create a PDB for the above deployment with the **minAvailable field set to 5** and **app:nginx as the label selector**.
```
# pdb1.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pdb-minavail
spec:
  minAvailable: 5
  selector:
    matchLabels:
      app: nginx
```
```
kubectl apply -f pdb1.yaml
kubectl get pdb
kubectl describe pdb pdb-minavail
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dow1hokp3fxcoj89kwwg.JPG)
As we have set **minAvailable=5**, which means out of 8 replicas, 5 replicas will always run even if any voluntary disruption occurs.

- Now, let’s drain the node and see pdb working
```
kubectl drain <node_name> --ignore-daemonsets --delete-emptydir-data
```
It tries to evict the pod and retries until all get evicted. Also, my node is now **SchedulingDisabled**. All the pods running on it are drained.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n8phjkdwpo1v265ytpn4.JPG)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8f4267uv7w7el21k9lp3.JPG)

- Let’s check the pod status and verify whether they have been rescheduled.
```
kubectl get pods -o wide
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hc7z5rav3nf0z8va3xl7.png)
Here 4 pods were running on each node, and when I drained the node, pods got evicted and started rescheduling. One of the pods got rescheduled on other node, and three were not as insufficient resources left on the node and get into pending state.

But my node draining gets completed successfully as the PDB requirement was fulfilled to run 5 application pods. The eviction API respected PDB in this voluntary disruption.

- Now uncordon the node to make the rest of the pods schedulable. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u0b8ezc7955odlw3vfo2.JPG)

### Use Case: 2

- Now, increase the existing pdb **minAvailable field to 6** and drain the node to see what happens.
```
# pdb1.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pdb-minavail
spec:
  minAvailable: 6
  selector:
    matchLabels:
      app: nginx
```
```
kubectl apply -f pdb1.yaml
kubectl get pdb
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gb3q420c599pmx2tykwz.JPG)

- Once again, drain the cluster node.
```
k drain <node_name> --ignore-daemonsets  --delete-emptydir-data
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dqjnxsp9lhe2vh55y2dv.JPG)
You will observe that the drain will not be completed, and eviction API will retry to evict pods until it can reschedule on another node and throws an **error: cannot evict pod as it would violate the pod’s disruption budget.**

But why has this happened? **Because the minimum available pods in pdb are 6 and the other node can only schedule 5 pods according to its resource capacity.** As mentioned, eviction API gives PDB priority, so to make a minimum of 6 pods available, it will not drain the node and run the pods on it.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ti51t8tinngqh3q3og5l.JPG)

Although my node will mark SchedulingDisabled, it's not drained.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w4ntfxn3lrwy07bpzr5j.JPG)

- Delete the pdb and uncordon the node.
```
kubectl delete pdb pdb-minavail
```
### Use Case : 3

- Now, create another pdb resource with **maxUnavailable set to 3**.
```
# pdb2.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pdb-maxunavail
spec:
  maxUnavailable: 3
  selector:
    matchLabels:
      app: nginx
```
```
kubectl apply -f pdb2.yaml
kubectl describe pdb pdb-maxunavail
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bx6v9kv0142bbarj7vp4.JPG)

- Follow the same steps of draining a node to check pdb working
```
k drain <node_name>  --ignore-daemonsets  --delete-emptydir-data
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/enmfeuxw1g0frd8o92kv.JPG)
This time draining works completely as the pdb requirement got satisfied.

- The rest of the pods remain unschedulable.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9e4qcq8odzzodvftvcff.JPG)
Uncordon the node now.

### Use Case: 4
Reduce the **maxUnavailable field to 2**, which makes 6 pods run all the time. Now if you drain the node, **use case 2** scenario will happen. Node draining will be incomplete, and pods will not be evicted completely by giving weightage to PDB.

### Use Case: 5
Now, what if I set **maxUnavailable to 0?** This is equal to the setting of **minAvailable to 100%**. It will be ensured that none of your pods will be disrupted when voluntary disruptions occur.

# When not to use PDB?
There are certain cases when pdb cannot be used, such as:

- It doesn’t work for involuntary disruptions
- In voluntary disruptions, it will not work when pods or deployments get deleted.
- Two PDBs can not work together on the same resource.
- PDBs don’t work on a single pod or replica of deployment.

# Summary
PDBs are useful when we want applications always available, even at cluster maintenance and upgrade times.

# References

- [What is Disruption?](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)
- [PDB in Pods](https://kubernetes.io/docs/tasks/run-application/configure-pdb/)
- [Kubecon lightning talk](https://www.youtube.com/watch?v=0AGZ5no6-yo)