Pod rebalancing and allocations in Kubernetes
Tags: kubernetes, devops
Published: 2023-04-03 12:43:38

**Does Kubernetes rebalance your Pods?**

**If there's a node that has more space, does Kubernetes recompute and balance the workloads?**

_Let's have a look at an example._

You have a cluster with a single node that can host 2 Pods.

If the node crashes, you will experience downtime.

You could have a second node with one Pod each to prevent this.

![A Kubernetes cluster with a single node](https://res.cloudinary.com/learnk8s/image/upload/v1680499135/threads/rebalancing-2_twahia.png)

You provision a second node.

_What happens next?_

_Does Kubernetes notice that there's a space for your Pod?_

_Does it move the second Pod and rebalance the cluster?_

![Does Kubernetes move the pods to the lower utilized node?](https://res.cloudinary.com/learnk8s/image/upload/v1680499135/threads/rebalancing-3_rshygg.png)

**Unfortunately, it does not.**

_But why?_

When you define a Deployment, you specify:

- The template for the Pod.
- The number of copies (replicas).

![A Kubernetes deployment](https://res.cloudinary.com/learnk8s/image/upload/v1680499135/threads/rebalancing-4_ggcubm.png)

**But nowhere in that file you said you want one replica for each node!**

The ReplicaSet counts 2 Pods, and that matches the desired state.

Kubernetes won't take any further action.

In other words, Kubernetes **does not** rebalance your pods automatically.

_But you can fix this._

There are three popular options:

1. Pod (anti-)affinity.
2. Pod topology spread constraints.
3. The Descheduler.

The first option is to [use pod anti-affinity.](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)

**With pod anti-affinity, your Pods repel other pods with the same label, forcing them to be on different nodes.**

[You can read more about pod anti-affinity here.](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)

![Example of a pod anti-affinity](https://res.cloudinary.com/learnk8s/image/upload/v1680499135/threads/rebalancing-7_ayklzo.png)

**Notice how pod affinity is evaluated when the scheduler allocates the pods.**

It is not applied retroactively, so you might need to delete a few pods to force the scheduler to recompute the allocations.

![Kubernetes does not rebalance two pods that have pod anti-affinity and are already alloacated to the same node](https://res.cloudinary.com/learnk8s/image/upload/v1680499135/threads/rebalancing-8_qgh6ro.png)

**Alternatively, you can use [topology spread constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) to control how Pods are spread across your cluster among failure domains such as regions, zones, nodes, etc.**

This is similar to pod affinity but more powerful.

![Spreading posts across failure domains](https://res.cloudinary.com/learnk8s/image/upload/v1680499136/threads/rebalancing-10_prmxfn.png)

With topology spread constraints, you can pick the topology and choose the pod distribution (skew), what happens when the constraint is unfulfillable (schedule anyway vs don't) and the interaction with pod affinity and taints.

![Example of pod topology spread constraints](https://res.cloudinary.com/learnk8s/image/upload/v1680499136/threads/rebalancing-11_pvdcec.png)

However, even in this case, the scheduler evaluates topology spread constraints when the pod is allocated.

**It does not apply retroactively — you can still delete the pods and force the scheduler to reallocate them.**

![Kubernetes does not rebalance two pods that have pod topology spread constraints and are already alloacated to the same node](https://res.cloudinary.com/learnk8s/image/upload/v1680499136/threads/rebalancing-12_zgmynh.png)

**If you want to rebalance your pods dynamically (not just when the scheduler allocates them), you should check out [the Descheduler.](https://github.com/kubernetes-sigs/descheduler)**

The Descheduler scans your cluster at regular intervals, and if it finds a node that is more utilized than others, it deletes a pod in that node.

![The Descheduler deletes pods](https://res.cloudinary.com/learnk8s/image/upload/v1680499136/threads/rebalancing-13_ntjgy8.png)

_What happens when a Pod is deleted?_

**The ReplicaSet will create a new Pod, and the scheduler will likely place it in a less utilized node.**

If your pod has topology spread constraints or pod affinity, it will be allocated accordingly.

![The Kubernetes scheduler will allocate pods efficiently](https://res.cloudinary.com/learnk8s/image/upload/v1680499136/threads/rebalancing-14_m8dtu6.png)

The Descheduler can evict pods based on policies such as:

- Node utilization.
- Pod age.
- Failed pods.
- Duplicates.
- Affinity or taints violations.

![](https://res.cloudinary.com/learnk8s/image/upload/v1680499136/threads/rebalancing-15_pduiuo.png)

**If your cluster has been running long, the resource utilization could be more balanced.**

The following two strategies can be used to rebalance your cluster based on CPU, memory or number of pods.

![Descheduler configuration to rebalance under and overtilized nodes](https://res.cloudinary.com/learnk8s/image/upload/v1680499136/threads/rebalancing-16_fupkpj.png)

Another practical policy is preventing developers and operators from treating pods like virtual machines.

**You can use the descheduler to ensure pods only run for a fixed time (e.g. seven days).**

![Deleting pods after 7 days of utilization](https://res.cloudinary.com/learnk8s/image/upload/v1680499549/threads/rebalancing-17_gsn5ym.png)

**Lastly, you can combine the Descheduler with Node Problem Detector and Cluster Autoscaler to automatically remove Nodes with problems.**

[The Descheduler can be used to descheduler workloads from those Nodes.](https://github.com/kubernetes-sigs/descheduler/blob/master/docs/user-guide.md#autoheal-node-problems)

The Descheduler is an excellent choice to keep your cluster efficiency in check, but it isn't installed by default.

[It can be deployed as a Job, CronJob or Deployment.](https://github.com/kubernetes-sigs/descheduler)

And finally, if you've enjoyed this thread, you might also like:

- The Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training
- This collection of past threads https://twitter.com/danielepolencic/status/1298543151901155330
- The Kubernetes newsletter I publish every week https://learnk8s.io/learn-kubernetes-weekly