Understanding Kubernetes Evicted Pods
Tags: kubernetes, memory, pods, evicted
Published: 2022-09-20 15:55:44

---
title: Understanding Kubernetes Evicted Pods
published: true
description: What does it mean that Kubernetes Pods are evicted? They are terminated, usually due to a lack of resources. But, why does this happen?
tags: kubernetes, memory, pods, evicted
cover_image: https://sysdig.com/wp-content/uploads/Blog_Images-SecureKubernetesDeployment-featured.png
canonical_url: https://sysdig.com/blog/kubernetes-pod-evicted/
# Use a ratio of 100:42 for best results.
---

<p>
What does it mean that Kubernetes Pods are evicted? They are terminated, usually the result of not having enough resources. But why does this happen?
</p>
<p>
<strong>Eviction</strong> is a process where a <strong>Pod</strong> assigned to a Node is <strong>asked for termination</strong>. One of the most common cases in Kubernetes is <strong>Preemption</strong>, where in order to schedule a new Pod in a Node with limited resources, another Pod needs to be terminated to leave resources to the first one. 
</p><p>
Also, Kubernetes constantly checks resources and evicts Pods if needed, a process called <strong>Node-pressure eviction</strong>.
</p>
<p>
<img src="https://sysdig.com/wp-content/uploads/Blog_Images-SecureKubernetesDeployment-featured.png" alt="Understanding Kubernetes Evicted Pods main image" width="1200" height="600" class="alignnone size-medium wp-image-54171" />
</a> 
<em>Every day, thousands of Pods are evicted from their homes. Stranded and confused, they have to abandon their previous lifestyle. Some of them even become nodeless. The current society, imposing higher demands of CPU and memory, is part of the problem.</em>
</p>
<p>
</p>
<p>
During this article, you will discover:
</p>
<ul>
<li><a href="#reasonswhypodsareevicted">Reasons why Pods are evicted: Preemption and Node-pressure</a>
</li><li><a href="#preemptioneviction">Preemption eviction</a>
</li><li><a href="#podpriorityclasses">Pod Priority Classes</a>
</li><li><a href="#nodepressureeviction">Node-pressure eviction</a>
</li><li><a href="#qosclasses">Quality of Service Classes</a>
</li><li><a href="#othertypeseviction">Other types of eviction</a>
</li><li><a href="#podevictionprometheus">Kubernetes Pod eviction monitoring in Prometheus</a>
</li>
</ul>


<h2 id="reasonswhypodsareevicted">Reasons why Pods are evicted: Preemption and Node-pressure</h2>
<p>
There are several reasons why Pod eviction can happen in Kubernetes. The most important ones are:
</p>
<ul>
<li>Preemption
</li><li>Node-pressure eviction
</li>
</ul>


<h2 id="preemptioneviction">Preemption eviction</h2>
<p>
<strong>Preemption</strong> is the following process: if a new Pod needs to be scheduled but <strong>doesn’t have any suitable Node with enough resources</strong>, then kube-scheduler will check if by <strong>evicting</strong> (terminating) some Pods with <strong>lower priority</strong> the new Pod can be part of that Node.
</p>
<p>
Let’s first understand how Kubernetes scheduling works.
</p>


<h3>Pod Scheduling</h3>
<p>
Kubernetes <strong>Scheduling</strong> is the process where <strong>Pods are assigned to nodes</strong>.
</p>
<p>
By default, there’s a Kubernetes entity responsible for scheduling, called <code>kube-scheduler</code> which will be running in the control plane. The Pod will start in the <a target="_blank" href="https://sysdig.com/blog/kubernetes-pod-pending-problems/" rel="noopener">Pending state</a> until a matching node is found.
</p>
<p>
The process of assigning a Pod to a Node follows this sequence:
</p>
<ol>
<li>Filtering
</li><li>Scoring
</li>
</ol>


<h4>Filtering</h4>
<p>
During the <strong>Filtering step,</strong> <code>kube-scheduler</code> will <strong>select</strong> all Nodes <strong>where the current Pod might be placed</strong>. Features like Taints and Tolerations will be taken into account here. Once finished, it will have a list of suitable Nodes for that Pod.
</p>


<h4>Scoring</h4>
<p>
During the <strong>Scoring step,</strong> <code>kube-scheduler</code> will take the resulting list from the previous step and <strong>assign a score to each of the nodes</strong>. This way, candidate nodes are ordered from most suitable to least. In case two nodes have the same score, kube-scheduler orders them randomly.
</p>
<p>
<a href="https://sysdig.com/wp-content/uploads/Troubleshooting-Evicted-Pods-02.png"><img src="https://sysdig.com/wp-content/uploads/Troubleshooting-Evicted-Pods-02.png" alt="Filtering and Scoring process" class="aligncenter size-full wp-image-53161 u-no-shadow" /></a>
</p>
<p>
But, what happens if there are no suitable Nodes for a Pod to run? When that’s the case, Kubernetes will start the <strong>preemption</strong> process, trying to <strong>evict</strong> lower priority Pods in order for the new one to be assigned.
</p>


<h3 id="podpriorityclasses">Pod Priority Classes</h3>
<p>
How can I prevent a particular Pod from being evicted in case of a preemption process? Chances are, a specific Pod is critical for you and should never be terminated.
</p>
<p>
That’s why Kubernetes features <strong>Priority Classes</strong>.
</p>
<p>
A <strong>Priority Class</strong> is a Kubernetes object that allows us to map <strong>numerical priority values</strong> to specific Pods. Those with a higher value are classified as more important and less likely to be evicted.
</p>
<p>
You can query current Priority Classes using:
</p>
<pre class="prettyprint" style="margin: 1em 0">kubectl get priorityclasses
kubectl get pc

NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-cluster-critical   2000000000   false            2d
system-node-critical      2000001000   false            2d
</pre>


<h3>Priority Class example</h3>
<p>
Let’s do a practical example using the <a href="https://tapas.io/episode/220482" target="_blank" rel="noopener">Berry Club comic</a> from <a href="https://www.mrlovenstein.com/">Mr. Lovenstein</a>:
</p>

<div class="embed-responsive"><div class="twitter-tweet twitter-tweet-rendered" style="display: flex; max-width: 550px; width: 100%; margin-top: 10px; margin-bottom: 10px;"><iframe id="twitter-widget-0" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" class="" style="position: static; visibility: visible; width: 550px; height: 802px; display: block; flex-grow: 1;" title="Twitter Tweet" src="https://platform.twitter.com/embed/Tweet.html?creatorScreenName=sysdig&amp;dnt=true&amp;embedId=twitter-widget-0&amp;features=eyJ0ZndfdGltZWxpbmVfbGlzdCI6eyJidWNrZXQiOlsibGlua3RyLmVlIiwidHIuZWUiXSwidmVyc2lvbiI6bnVsbH0sInRmd19ob3Jpem9uX3RpbWVsaW5lXzEyMDM0Ijp7ImJ1Y2tldCI6InRyZWF0bWVudCIsInZlcnNpb24iOm51bGx9LCJ0ZndfdHdlZXRfZWRpdF9iYWNrZW5kIjp7ImJ1Y2tldCI6Im9uIiwidmVyc2lvbiI6bnVsbH0sInRmd19yZWZzcmNfc2Vzc2lvbiI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfY2hpbl9waWxsc18xNDc0MSI6eyJidWNrZXQiOiJjb2xvcl9pY29ucyIsInZlcnNpb24iOm51bGx9LCJ0ZndfdHdlZXRfcmVzdWx0X21pZ3JhdGlvbl8xMzk3OSI6eyJidWNrZXQiOiJ0d2VldF9yZXN1bHQiLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X3NlbnNpdGl2ZV9tZWRpYV9pbnRlcnN0aXRpYWxfMTM5NjMiOnsiYnVja2V0IjoiaW50ZXJzdGl0aWFsIiwidmVyc2lvbiI6bnVsbH0sInRmd19leHBlcmltZW50c19jb29raWVfZXhwaXJhdGlvbiI6eyJidWNrZXQiOjEyMDk2MDAsInZlcnNpb24iOm51bGx9LCJ0ZndfZHVwbGljYXRlX3NjcmliZXNfdG9fc2V0dGluZ3MiOnsiYnVja2V0Ijoib24iLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X3R3ZWV0X2VkaXRfZnJvbnRlbmQiOnsiYnVja2V0Ijoib2ZmIiwidmVyc2lvbiI6bnVsbH19&amp;frame=false&amp;hideCard=false&amp;hideThread=false&amp;id=1386299194013585414&amp;lang=en&amp;origin=https%3A%2F%2Fsysdig.com%2F%3Fp%3D54152%26preview%3Dtrue%26preview_id%3D54152&amp;sessionId=8fae47ff009ec6c685ad92f8a5eb65c73c4c18bf&amp;siteScreenName=sysdig&amp;theme=light&amp;widgetsVersion=1bfeb5c3714e8%3A1661975971032&amp;width=550px" data-tweet-id="1386299194013585414"></iframe></div><script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></div>
<p>
There are three Pods representing blueberry, raspberry and strawberry:
</p>
<pre class="prettyprint" style="margin: 1em 0">NAME         READY   STATUS             RESTARTS   AGE
blueberry    1/1     Running            0          4h41m
raspberry    1/1     Running            0          58m
strawberry   1/1     Running            0          5h22m
</pre>
<p>
And there are two Priority Classes: trueberry and falseberry. The first one will have a higher value indicating higher priority.
</p>
<pre class="prettyprint" style="margin: 1em 0">apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: trueberry
value: 1000000
globalDefault: false
description: "This fruit is a true berry"

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: falseberry
value: 5000
globalDefault: false
description: "This fruit is a false berry"
</pre>
<ul>
<li>Blueberry will have the trueberry priority class (value = 1000000)
</li><li>raspberry and strawberry will both have the falseberry priority class (value = 5000)
</li>
</ul>
<p>
This will mean that in case of a preemption, raspberry and strawberry are more likely to be evicted to make room for higher priority Pods.
</p>

<p>
Then assign the Priority Classes to Pods by adding this to the Pod definition:
</p>
<pre class="prettyprint" style="margin: 1em 0"> priorityClassName: trueberry
</pre>
<p>
Let’s now try to add three more fruits, but with a twist. All of the new fruits will contain the higher Priority Class called <code>trueberry</code>.
</p>
<p>
Since the three new fruits have memory or CPU requirements that the node can’t satisfy, <code>kubelet</code> evicts all Pods with lower priority than the new fruits. Blueberry stays running as it has the higher priority class.
</p>
<pre class="prettyprint" style="margin: 1em 0">NAME         READY   STATUS             RESTARTS   AGE
banana       0/1     ContainerCreating  0          2s
blueberry    1/1     Running            0          4h42m
raspberry    0/1     Terminating        0          59m
strawberry   0/1     Terminating        0          5h23m
tomato       0/1     ContainerCreating  0          2s
watermelon   0/1     ContainerCreating  0          2s
</pre>
<p>
<a href="https://sysdig.com/wp-content/uploads/Blog_Images-TrobleshootingEvicted-diagram4.png"><img src="https://sysdig.com/wp-content/uploads/Blog_Images-TrobleshootingEvicted-diagram4.png" alt="Kubernetes Priority Classes - Live example" width="1200" height="600" class="aligncenter size-full wp-image-53161 u-no-shadow"/></a>
</p>

<p>
This is the end result:
</p>
<pre class="prettyprint" style="margin: 1em 0">NAME         READY   STATUS             RESTARTS   AGE
banana       1/1     Running            0          3s
blueberry    1/1     Running            0          4h43m
tomato       1/1     Running            0          3s
watermelon   1/1     Running            0          3s
</pre>

<p>
These are strange times for berry club...
</p>


<h2 id="nodepressureeviction">Node-pressure eviction</h2>
<p>
Apart from preemption, Kubernetes also constantly checks node resources, like disk pressure, CPU or Out of Memory (OOM).
</p>
<p>
In case a resource (like <strong>CPU</strong> or <strong>memory</strong>) consumption in the node <strong>reaches a certain threshold</strong>, <code>kubelet</code> will start evicting Pods in order to <strong>free up the resource</strong>. Quality of Service (QoS) will be taken into account to determine the eviction order.
</p>


<h3 id="qosclasses">Quality of Service Classes</h3>
<p>
In Kubernetes, Pods are giving one of three <strong>QoS Classes</strong>, which will define how likely they are going to be evicted in case of lack of resources, from less likely to more likely:
</p>
<ul>
<li><strong>Guaranteed</strong>
</li><li><strong>Burstable</strong>
</li><li><strong>BestEffort</strong>
</li>
</ul>
<p>
How are these QoS Classes assigned to Pods? This is based on <strong>limits and requests</strong> for <strong>CPU</strong> and <strong>memory</strong>. As a reminder:
</p>
<ul>
<li><strong>Limits</strong>: maximum amount of a resource that a container can use.
</li><li><strong>Requests</strong>: minimum desired amount of resources for a container to run.
</li>
</ul>
<p>
For more information about limits and requests, please check <a target="_blank" href="https://sysdig.com/blog/kubernetes-limits-requests/" rel="noopener">Understanding Kubernetes limits and requests by example</a>.
</p>
<p>
<a href="https://sysdig.com/wp-content/uploads/Troubleshooting-Evicted-Pods-03.png"><img src="https://sysdig.com/wp-content/uploads/Troubleshooting-Evicted-Pods-03.png" alt="QoS Classes in Kubernetes" width="1200" height="450" class="aligncenter size-full wp-image-53161 u-no-shadow" /></a>
</p>


<h4>Guaranteed</h4>
<p>
A Pod is assigned with a QoS Class of <strong>Guaranteed</strong> if:
</p>
<ul>
<li>All containers in the Pod have <strong>both Limits and Requests set</strong> for CPU and memory.
</li><li>All containers in the Pod have <strong>the same value</strong> for CPU Limit and CPU Request.
</li><li>All containers in the Pod have <strong>the same value</strong> for memory Limit and memory Request.
</li>
</ul>
<p>
A Guaranteed Pod won’t be evicted in normal circumstances to allocate another Pod in the node.
</p>


<h4>Burstable</h4>
<p>
A Pod is assigned with a QoS Class of <strong>Burstable</strong> if:
</p>
<ul>
<li>It doesn’t have QoS Class of Guaranteed.
</li><li><strong>Either Limits or Requests</strong> have been set for a container in the Pod.
</li>
</ul>
<p>
A Burstable Pod can be evicted, but less likely than the next category.
</p>


<h4>BestEffort</h4>
<p>
A Pod will be assigned with a QoS Class of <strong>BestEffort</strong> if:
</p>
<ul>
<li><strong>No Limits and Requests</strong> are set for any container in the Pod.
</li>
</ul>
<p>
BestEffort Pods have the highest chance of eviction in case of a node-pressure process happening in the node.
</p>
<p><span style="text-size: smaller; font-style: italic; border: 1px solid #FF9E1B; display: block; border-radius: 25px; padding: 25px; max-width: 96%; margin: 2em auto;"><em>
Important: there may be other available resources in Limits and Requests, like ephemeral-storage, but they are not used for QoS Class calculation.
</em></span></p>
<p>


<a href="https://sysdig.com/wp-content/uploads/Blog_Images-TrobleshootingEvicted-diagram3.png"><img src="https://sysdig.com/wp-content/uploads/Blog_Images-TrobleshootingEvicted-diagram3.png" alt="Quality of Service cheatsheet" width="1200" height="600" class="aligncenter size-full wp-image-53161 u-no-shadow" /></a>
</p>
<p>
As mentioned, QoS Classes will be taken into account for node-pressure eviction. Here’s the process that happens internally.
</p>
<p>
The kubelet ranks the Pods to be evicted in the following order:
</p>
<ol>
<li><code>BestEffort</code> Pods or <code>Burstable</code> Pods where usage exceeds requests
</li><li><code>Burstable</code> Pods where usage is below requests or <code>Guaranteed</code> Pods
</li>
</ol>
<p>Kubernetes will try to evict Pods from group 1 before group 2.
</p>
<p>
Some takeaways from the above:
</p>
<ul>
<li>If you add very low requests in your containers, their Pod is likely going to be assigned group 1, which means it's more likely to be evicted.
</li><li>You can’t tell which specific Pod is going to be evicted, just that Kubernetes will try to evict ones from group 1 before group 2.
</li><li><code>Guaranteed</code> Pods are usually safe from eviction: <code>kubelet</code> won’t evict them in order to schedule other Pods. But if some system services need more resources, the kubelet will terminate <code>Guaranteed</code> Pods if necessary, always with the lowest priority.
</li>
</ul>


<h2 id="othertypeseviction">Other types of eviction</h2>
<p>
This article is focused on preemption and node-pressure eviction, but Pods can be evicted in other ways as well. Examples include:
</p>


<h3>API-initiated eviction</h3>
<p>
You can request an on-demand eviction of a Pod in one of your nodes by using <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#create-eviction-pod-v1-core">Kubernetes Eviction API</a>.
</p>


<h3>Taint-based eviction</h3>
<p>
With Kubernetes Taints and Tolerations you can guide how your Pods should be assigned to Nodes. But if you apply a <code>NoExecute</code> taint to an existing Node, all Pods which are not tolerating it will be immediately evicted.
</p>


<h3>Node drain</h3>
<p>
There are times when Nodes become unusable or you don't want to work on them anymore. The command <code>kubectl cordon</code> prevents new Pods to be scheduled on it, but there’s also the possibility to completely empty all current Pods at once. If you run <code>kubectl drain nodename</code>, all Pods in the node will be evicted, respecting its graceful termination period.
</p>


<h2 id="podevictionprometheus">Kubernetes Pod eviction monitoring in Prometheus</h2>
<p>
In your cloud solution, you can use Prometheus to easily monitor Pod evictions by doing:
</p>
<pre class="prettyprint" style="margin: 1em 0">
kube_pod_status_reason{reason="Evicted"} > 0
</pre>
<a href="https://sysdig.com/wp-content/uploads/Troubleshooting-Evicted-Pods-6.png"><img src="https://sysdig.com/wp-content/uploads/Troubleshooting-Evicted-Pods-6.png" alt="Monitor Evicted Pods with Prometheus" width="1999" height="652" class="aligncenter size-full wp-image-53161 u-no-shadow"  /></a>

<p>
This will display all evicted Pods in your cluster. You can also pair this with <code>kube_pod_status_phase{phase="Failed"}</code> in order to alert on those who were evicted after a failure in the Pod.
</p>
<p>
If you want to dig deeper, check the following articles for monitoring resources in Prometheus:
</p>
<ul>
<li>
<a href="https://sysdig.com/blog/kubernetes-resource-limits/" rel="noopener" target="_blank">How to rightsize the Kubernetes resource limits</a>
</li>
<li>
<a href="https://sysdig.com/blog/kubernetes-capacity-planning/" rel="noopener" target="_blank">Kubernetes capacity planning: How to rightsize the requests of your cluster </a>
</li>
</ul>

<h2 id="conclusion">Conclusion</h2>
<p>
As you can see, eviction is just another feature from Kubernetes which allows you to control limited resources: in this case, the nodes that Pods will be using.
</p>
<p>
During <strong>preemption</strong>, Kubernetes will try to free up resources by evicting less priority Pods to schedule a new one. With Priority Classes you can control which Pods are more likely to keep running after preemption since there’s less chance that they will be evicted.
</p>
<p>
During execution, Kubernetes will check for <strong>Node-pressure</strong> and evict Pods if needed. With QoS classes you can control which Pods are more likely to be evicted in case of node-pressure.
</p>
<p>
Memory and CPU are important resources in your nodes, and you need to configure your Pods, containers and nodes to use the right amount of them. If you manage these resources accordingly, there could not only be a benefit in costs, but also you can ensure that the important processes will keep running, no matter how.
</p>
<hr>

<div class="sysdig-box" style="background-color: #f8f8f8; padding: 2em;">
<h2 style="margin-top: 0;">Get ahead of Pod eviction with Sysdig Monitor</h2>
<p>
With Sysdig Advisor, you can review cluster resource availability in order to prevent Pod eviction. Featuring:
</p>
<ul>

<li>CPU overcommitment metric

<li>CPU Capacity

<li>How to fix tips
</li>
</ul>
<a href="https://sysdig.com/wp-content/uploads/cpu-overcommit.png"><img src="https://sysdig.com/wp-content/uploads/cpu-overcommit.png" alt="Screenshot of Sysdig Monitor, with the CPU overcommit resource" width="2784" height="1804" class="aligncenter size-full wp-image-54357" /></a>
<p>
Sysdig Advisor accelerates mean time to resolution (MTTR) with live logs, performance data, and suggested remediation steps. It’s the easy button for Kubernetes troubleshooting!
</p>
<p>
<a target="_blank" href="https://sysdig.com/company/free-trial-monitor/" rel="noopener">Try it free for 30 days!</a>
</p>
</div>