Kubernetes scheduler deep dive
Tags: kubernetes, devops
Published: 2023-05-23 12:44:34

The scheduler is in charge of deciding where your pods are deployed in the cluster.

It might sound like an easy job, but it's rather complicated!

Let's start with the basic.

When you submit a deployment with kubectl, the API server receives the request, and the resource is stored in etcd.

_Who creates the pods?_

![A pod resource is stored in Etcd](https://res.cloudinary.com/learnk8s/image/upload/v1676265277/threads/scheduler-2_iffurf.png)

It's a common misconception that it's the scheduler's job to create the pods.

**Instead, the controller manager creates them (and the associated ReplicaSet).**

![The controller manager creates the pods](https://res.cloudinary.com/learnk8s/image/upload/v1676265277/threads/scheduler-3_fzbzfx.png)

At this point, the pods are stored as "Pending" in the etcd and are not assigned to any node.

They are also added to the scheduler's queue, ready to be assigned.

![Pods are added to the scheduler queue.](https://res.cloudinary.com/learnk8s/image/upload/v1676265277/threads/scheduler-4_sqbnrp.png)

The scheduler process Pods 1 by 1 through two phases:

1. Scheduling phase (what node should I choose?).
1. Binding phase (let's write to the database that this pod belongs to that node).

![Pods are allocated one at the time](https://res.cloudinary.com/learnk8s/image/upload/v1676265277/threads/scheduler-5_ezr7yh.png)

The Scheduler phase is divided into two parts. The Scheduler:

1. Filters relevant nodes (using a list of functions called predicates)
1. Ranks the remaining nodes (using a list of functions called priorities)

Let's have a look at an example.

![The scheduler filters and scores nodes](https://res.cloudinary.com/learnk8s/image/upload/v1676265277/threads/scheduler-6_fr8vbf.png)

Consider the following cluster with nodes with and without GPU.

Also, a few nodes are already running at total capacity.

![A collection of Kubernetes nodes](https://res.cloudinary.com/learnk8s/image/upload/v1676265277/threads/scheduler-7_rkriei.png)

You want to deploy a Pod that requires some GPU.

You submit the pod to the cluster, and it's added to the scheduler queue.

The scheduler discards all nodes that don't have GPU (filter phase).

![All non-GPU nodes are discarded](https://res.cloudinary.com/learnk8s/image/upload/v1676265277/threads/scheduler-8_ayzkxe.png)

Next, the scheduler scores the remaining nodes.

In this example, the fully utilized nodes are scored lower.

In the end, the empty node is selected.

![The remaining nodes are scored](https://res.cloudinary.com/learnk8s/image/upload/v1676265277/threads/scheduler-9_juxik4.png)

What are some examples of filters?

- `NodeUnschedulable` prevents pods from landing on nodes marked as unschedulable.
- `VolumeBinding` checks if the node can bind the requested volume.

The default filtering phase has 13 predicates.

![Default predicates in the Kubernetes scheduler](https://res.cloudinary.com/learnk8s/image/upload/v1676265278/threads/scheduler-10_ndfbhj.png)

Here are some examples of scoring:

- `ImageLocality` prefers nodes that already have the container image downloaded locally.
- `NodeResourcesBalancedAllocation` prefers underutilized nodes.

There are 13 functions to decide how to score and rank nodes.

![Default functions to score nodes in Kubernetes](https://res.cloudinary.com/learnk8s/image/upload/v1676265278/threads/scheduler-11_tzyuys.png)

How can you influence the scheduler's decisions?

- `nodeSelector`
- Node affinity
- Pod affinity/anti-affinity
- Taints and tolerations
- Topology constraints
- Scheduler profiles

`nodeSelector` is the most straightforward mechanism.

You assign a label to a node and add that label to the pod.

The pod can only be deployed on nodes with that label.

![Assigning pods to nodes with the nodeSelector](https://res.cloudinary.com/learnk8s/image/upload/v1676265278/threads/scheduler-13_z5z8v9.png)

Node affinity extends nodeSelector with a more flexible interface.

You can still tell the scheduler where the Pod should be deployed, but you can also have soft and hard constraints.

![Assigning pods to nodes with node affinity](https://res.cloudinary.com/learnk8s/image/upload/v1676265278/threads/scheduler-14_hvm2hf.png)

With Pod affinity/anti-affinity, you can ask the scheduler to place a pod next to a specific pod.

Or not.

For example, you could have a deployment with anti-affinity on itself to force spreading pods.

![Scheduling pods with pod affinity and anti-affinity](https://res.cloudinary.com/learnk8s/image/upload/v1676265278/threads/scheduler-15_ifw2qs.png)

With taints and tolerations, pods are tainted, and nodes repel (or tolerate) pods.

This is similar to node affinity, but there's a notable difference: with Node affinity, Pods are attracted to nodes.

Taints are the opposite - they allow a node to repel pods.

![Scheduling pods with taints and tolerations](https://res.cloudinary.com/learnk8s/image/upload/v1676265278/threads/scheduler-16_bxlqml.png)

Moreover, tolerations can repel pods with three effects: evict, "don't schedule", and "prefer don't schedule".

Personal note: this is one of the most difficult APIs I worked with.

I always (and consistently) get it wrong as it's hard (for me) to reason in double negatives.

You can use topology spread constraints to control how Pods are spread across your cluster.

This is convenient when you want to ensure that all pods aren't landing on the same node.

![Pod topology constraints](https://res.cloudinary.com/learnk8s/image/upload/v1676265279/threads/scheduler-17_nq2fip.png)

And finally, you can use Scheduler policies to customize how the scheduler uses filters and predicates to assign nodes to pods.

This relatively new feature (>1.25) allows you to turn off or add new logic to the scheduler.

![Scheduler policies in Kubernetes](https://res.cloudinary.com/learnk8s/image/upload/v1676265279/threads/scheduler-18_iyvamp.png)

You can learn more about the scheduler here:

- Kubernetes scheduler https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/
- Scheduling framework https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/
- Scheduler policies https://kubernetes.io/docs/reference/scheduling/config/

And finally, if you've enjoyed this thread, you might also like:

- The Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training
- This collection of past threads https://twitter.com/danielepolencic/status/1298543151901155330
- The Kubernetes newsletter I publish every week "Learn Kubernetes weekly" https://learnk8s.io/learn-kubernetes-weekly