Reserved CPU and memory in Kubernetes nodes
Tags: kubernetes
Published: 2022-10-25 12:54:39

_In Kubernetes, are there hidden costs to running many cluster nodes?_

Yes, since [not all CPU and memory in your Kubernetes nodes can be used to run Pods.](https://learnk8s.io/allocatable-resources)

In a Kubernetes node, CPU and memory are divided into:

1. Operating system.
1. Kubelet, CNI, CRI, CSI (+ system daemons).
1. Pods.
1. Eviction threshold.

![Allocatable resources in Kubernetes nodes](https://res.cloudinary.com/uasabi/image/upload/v1666701295/threads/reserved-resources-2.png)

Those reservations depend on the size of the instance and could add up to a sizeable overhead.

_Let's give a quick example._

Imagine you have a cluster with a single 1GiB / 1vCPU node.

The following resources are reserved for the kubelet and operating system:

- 255MiB of memory.
- 60m of CPU.

On top of that, 100MB is reserved for the eviction threshold.

In total, **that's 25% of memory and 6% of CPU that you can't use.**

![In a 1GiB / 1vCPU Kubernetes node, 25% of the memory is reserved by the kubelet](https://res.cloudinary.com/uasabi/image/upload/v1666701296/threads/reserved-resources-3.png)

_How does this work in a cloud provider?_

EKS has some (interesting?) limits.

Let's pick an m5.large with 2vCPU and 8GiB of memory.

AWS reserves the following for the kubelet and operating system:

- 574MiB of memory.
- 70m of CPU.

_This time, you are lucky._

**You can use ~93% of the available memory.**

![If you use EKS with m5.large nodes, you can use 93% of the available memory to run pods](https://res.cloudinary.com/uasabi/image/upload/v1666701295/threads/reserved-resources-4.png)

_But where do those numbers come from?_

Every cloud provider has its way of defining limits, but for the CPU, they seem to all agree on the following values:

- 6% of the first core.
- 1% of the next core (up to 2 cores).
- 0.5% of the next 2 cores (up to 4 cores).
- 0.25% of any cores above four cores.

As for the memory limits, this varies a lot between providers.

Azure is the most conservative, and AWS the least.

The [reserved memory for the kubelet in Azure is:](https://docs.microsoft.com/en-us/azure/aks/concepts-clusters-workloads#resource-reservations)

- 25% of the first 4 GB of memory.
- 20% of the following 4 GB of memory (up to 8 GB).
- 10% of the following 8 GB of memory (up to 16 GB).
- 6% of the next 112 GB of memory (up to 128 GB).
- 2% of any memory above 128 GB.

That's [the same for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu) except for one value: the eviction threshold is 100MB in GKE and 750MiB in AKS.

[In EKS, the memory is allocated using the following formula:](https://github.com/awslabs/amazon-eks-ami/blob/4b54ee95d42df8a2715add2a32f5150db097fde8/files/bootstrap.sh#L247-L251)

```
255MiB + (11MiB * MAX_NUMBER OF PODS)
```

_This formula raises a few questions, though._

In the previous example, the m5.large reserved 574MiB of memory.

_Does that mean the VM can have at most (574 - 255) / 11 = 29 pods?_

This is correct if you don't [enable the prefix assignment mode in your VPC-CNI.](https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html)

**If you do, the results are vastly different.**

With up to 110 pods, AWS reserves:

- 1.4GiB of memory.
- (still) 70m of CPU.

Which sounds more reasonable and in line with the rest of the cloud providers.

![If you use EKS with m5.large nodes and prefix assignment mode, 1.4GiB of memory are reserved by the kubelet](https://res.cloudinary.com/uasabi/image/upload/v1666701296/threads/reserved-resources-7.png)

Let's have a look at GKE for a comparison.

With a similar instance type (i.e. n1-standard-2 with 7.5GB memory, 2vCPU), the reservations for the kubelet are as follows:

- 1.7GB of memory.
- 70m of CPU.

![If you use GKE with n1-standard-2 nodes, 1.7GiB of memory are reserved by the kubelet](https://res.cloudinary.com/uasabi/image/upload/v1666701296/threads/reserved-resources-8.png)

In other words, 23% of the instance memory cannot be allocated to run pods.

> If the instance costs $48.54 per month, **you spend $11.16 to run the kubelet.**

_What about the other cloud providers?_

_How do you inspect the values?_

We've built a simple tool that inspects the config of the kubelet and pulls the relevant details.

You can find it here: https://github.com/learnk8s/kubernetes-resource-inspector.

![Inspecting resources allocation in Kubernetes nodes](https://res.cloudinary.com/uasabi/image/upload/v1666701297/threads/reserved-resources-demo.gif)

If you're interested in exploring more about node sizes, we also built a simple instance calculator where you can define the size of the workloads, and it will show all instances that can fit that (and their price).

https://learnk8s.io/kubernetes-instance-calculator

![The Kubernetes instance calculator from Learnk8s](https://res.cloudinary.com/uasabi/image/upload/v1666701296/threads/reserved-resources-9.png)

I hope you enjoyed this short article on Kubernetes resource reservations; here, you can find more links to explore the subject further.

- [Kubernetes instance calculator](https://learnk8s.io/kubernetes-instance-calculator).
- [Allocatable memory and CPU in Kubernetes Nodes](https://learnk8s.io/allocatable-resources)
- Allocatable [memory and CPU resources on GKE.](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu)
- AWS EKS AMI [reserved CPU](https://github.com/awslabs/amazon-eks-ami/blob/4b54ee95d42df8a2715add2a32f5150db097fde8/files/bootstrap.sh#L253-L262) and [reserved memory](https://github.com/awslabs/amazon-eks-ami/blob/4b54ee95d42df8a2715add2a32f5150db097fde8/files/bootstrap.sh#L247-L251).
- [AKS resource reservations.](https://docs.microsoft.com/en-us/azure/aks/concepts-clusters-workloads#resource-reservations)
- Official docs https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/)
- [Enabling prefix assignment in EKS and VPC CNI.](https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html)

And finally, if you've enjoyed this thread, you might also like the Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training or this collection of past Twitter threads https://twitter.com/danielepolencic/status/1298543151901155330