Kubernetes OOM and CPU Throttling
Tags: kubernetes, devops, cloud, prometheus
Published: 2023-01-26 09:56:52

---
title: Kubernetes OOM and CPU Throttling
published: true
description: When working with Kubernetes, Out of Memory (OOM) errors and CPU throttling are the main headaches of resource handling in cloud applications. Why is that?

tags: kubernetes, devops, cloud, prometheus
cover_image: https://sysdig.com/wp-content/uploads/BlogImages-TroubleshootKubernetesOOM-featured-anim.gif
canonical_url: https://sysdig.com/blog/troubleshoot-kubernetes-oom/
# Use a ratio of 100:42 for best results.
---

<!-- wp:heading -->
<h2 id="introduction">Introduction</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>When working with Kubernetes, Out of Memory (OOM) errors and CPU throttling are the main headaches of resource handling in cloud applications. Why is that?</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>CPU and Memory requirements in cloud applications are ever more important, since they are tied directly to your cloud costs.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>With <a href="https://sysdig.com/blog/kubernetes-limits-requests/" target="_blank" rel="noreferrer noopener">limits and requests</a>, you can configure how your pods should allocate memory and CPU resources in order to prevent resource starvation and adjust cloud costs.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In case a Node doesn’t have enough resources, <a href="https://sysdig.com/blog/kubernetes-pod-evicted/" target="_blank" rel="noopener">Pods might get evicted</a> via preemption or node-pressure.<br>When a process runs Out Of Memory (OOM), it’s killed since it doesn’t have the required resources.<br>In case CPU consumption is higher than the actual limits, the process will start to be throttled.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>But, how can you actively monitor how close your Kubernetes Pods to OOM and CPU throttling?</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="kubernetes-oom">Kubernetes OOM</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Every container in a Pod needs memory to run.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Kubernetes limits are set per container in either a Pod definition or a Deployment definition.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>All modern Unix systems have a way to kill processes in case they need to reclaim memory. This will be marked as Error 137 or <code>OOMKilled.</code></p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">   State:          Running
      Started:      Thu, 10 Oct 2019 11:14:13 +0200
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 10 Oct 2019 11:04:03 +0200
      Finished:     Thu, 10 Oct 2019 11:14:11 +0200
</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>This Exit Code 137 means that the process used more memory than the allowed amount and had to be terminated.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This is a feature present in Linux, where the kernel sets an <code>oom_score</code> value for the process running in the system. Additionally, it allows setting a value called <code>oom_score_adj</code>, which is used by Kubernetes to allow Quality of Service. It also features an <code>OOM Killer,</code> which will review the process and terminate those that are using more memory than they should.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Note that in Kubernetes, a process can reach any of these limits:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>A Kubernetes Limit set on the container.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>A Kubernetes ResourceQuota set on the namespace.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>The node’s actual Memory size.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:image {"id":64792,"width":721,"height":435,"sizeSlug":"full","linkDestination":"media"} -->
<figure class="wp-block-image size-full is-resized"><a href="https://sysdig.com/wp-content/uploads/BlogImages-TroubleshootKubernetesOOM-1.png"><img src="https://sysdig.com/wp-content/uploads/BlogImages-TroubleshootKubernetesOOM-1.png" alt="Kubernetes OOM graph" class="wp-image-64792" width="721" height="435" title="image_tooltip"/></a></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3>Memory overcommitment</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Limits can be higher than requests, so the sum of all limits can be higher than node capacity. This is called overcommit and it is very common. In practice, if all containers use more memory than requested, it can exhaust the memory in the node. This usually causes the <a target="_blank" href="https://sysdig.com/blog/kubernetes-pod-evicted/" rel="noopener">death of some pods</a> in order to free some memory.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Monitoring Kubernetes OOM</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>When using node exporter in Prometheus, there’s one metric called <code>node_vmstat_oom_kill</code>. It’s important to track when an OOM kill happens, but you might want to get ahead and have visibility of such an event before it happens.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Instead, you can check how close a process is to the Kubernetes limits:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">(sum by (namespace,pod,container)
(rate(container_cpu_usage_seconds_total{container!=""}[5m])) / sum by 
(namespace,pod,container)
(kube_pod_container_resource_limits{resource="cpu"})) &gt; 0.8
</pre>
<!-- /wp:preformatted -->

<!-- wp:heading -->
<h2 id="kubernetes-cpu-throttling">Kubernetes CPU throttling</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>CPU Throttling</strong> is a behavior where processes are slowed when they are about to reach some resource limits.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Similar to the memory case, these limits could be:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>A Kubernetes Limit set on the container.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>A Kubernetes ResourceQuota set on the namespace.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>The node’s actual Memory size.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Think of the following analogy. We have a highway with some traffic where:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>CPU is the road.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Vehicles represent the process, where each one has a different size.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Multiple lanes represent having several cores.</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>A request would be an exclusive road, like a bike lane.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Throttling here is represented as a traffic jam: eventually, all processes will run, but everything will be slower.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>CPU process in Kubernetes</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>CPU is handled in Kubernetes with <strong>shares</strong>. Each CPU core is divided into 1024 shares, then divided between all processes running by using the cgroups (control groups) feature of the Linux kernel.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":64825,"width":721,"height":352,"sizeSlug":"full","linkDestination":"media"} -->
<figure class="wp-block-image size-full is-resized"><a href="https://sysdig.com/wp-content/uploads/BlogImages-TroubleshootKubernetesOOM-4.png"><img src="https://sysdig.com/wp-content/uploads/BlogImages-TroubleshootKubernetesOOM-4.png" alt="Kubernetes shares system for CPU" class="wp-image-64825" width="721" height="352"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>If the CPU can handle all current processes, then no action is needed. If processes are using more than 100% of the CPU, then shares come into place. As any Linux Kernel, Kubernetes uses the CFS (Completely Fair Scheduler) mechanism, so the processes with more shares will get more CPU time.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Unlike memory, a Kubernetes won't kill Pods because of throttling.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":64794,"width":721,"height":329,"sizeSlug":"full","linkDestination":"media"} -->
<figure class="wp-block-image size-full is-resized"><a href="https://sysdig.com/wp-content/uploads/BlogImages-TroubleshootKubernetesOOM-2.png"><img src="https://sysdig.com/wp-content/uploads/BlogImages-TroubleshootKubernetesOOM-2.png" alt="Kubernetes Throttling graph" class="wp-image-64794" width="721" height="329" title="image_tooltip"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><span style="text-size: smaller; font-style: italic; border: 1px solid #FF9E1B; display: block; border-radius: 25px; padding: 25px; max-width: 96%; margin: 2em auto;"><em>You can check CPU stats in /sys/fs/cgroup/cpu/cpu.stat</em></span></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>CPU overcommitment</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>As we saw in the <a target="_blank" href="https://sysdig.com/blog/kubernetes-limits-requests/" rel="noopener">limits and requests article</a>, it’s important to set limits or requests when we want to restrict the resource consumption of our processes. Nevertheless, beware of setting up total requests larger than the actual CPU size, as this means that every container should have a guaranteed amount of CPU.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Monitoring Kubernetes CPU throttling</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>You can check how close a process is to the Kubernetes limits:</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">(sum by (namespace,pod,container)(rate(container_cpu_usage_seconds_total
{container!=""}[5m])) / sum by (namespace,pod,container)
(kube_pod_container_resource_limits{resource="cpu"})) &gt; 0.8</pre>
<!-- /wp:preformatted -->

<!-- wp:paragraph -->
<p>In case we want to track the amount of throttling happening in our cluster, cadvisor provides <code>container_cpu_cfs_throttled_periods_total</code> and <code>container_cpu_cfs_periods_total</code>. With these two, you can easily calculate the % of throttling in all CPU periods.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="best-practices">Best practices</h2>
<!-- /wp:heading -->

<!-- wp:heading {"level":3} -->
<h3>Beware of limits and requests</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Limits are a way to set up a maximum cap on resources in your node, but these need to be treated carefully, as you might end up with a process throttled or killed.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Prepare against eviction</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>By setting very low requests, you might think this will grant a minimum of either CPU or Memory to your process. But <code>kubelet</code> will evict first those Pods with usage higher than requests first, so you’re marking those as the first to be killed!</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In case you need to protect specific Pods against preemption (when <code>kube-scheduler</code> needs to allocate a new Pod), assign Priority Classes to your most important processes.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Throttling is a silent enemy</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>By setting unrealistic limits or overcommitting, you might not be aware that your processes are being throttled, and performance impacted. Proactively monitor your CPU usage and know your actual limits in both containers and namespaces.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="wrapping-up">Wrapping up</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here’s a cheat sheet on Kubernetes resource management for CPU and Memory. This summarizes the current article plus these ones which are part of the same series:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li><a target="_blank" href="https://sysdig.com/blog/kubernetes-pod-evicted/" rel="noopener">https://sysdig.com/blog/kubernetes-pod-evicted/</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="https://sysdig.com/blog/kubernetes-limits-requests/" target="_blank" rel="noopener">https://sysdig.com/blog/kubernetes-limits-requests/</a></li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:image {"id":64790,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image size-large"><a href="https://sysdig.com/wp-content/uploads/BlogImages-TroubleshootKubernetesOOM-diagram-1170x664.png"><img src="https://sysdig.com/wp-content/uploads/BlogImages-TroubleshootKubernetesOOM-diagram-1170x664.png" alt="Kubernetes CPU and Memory cheatsheet" class="wp-image-64790"/></a></figure>
<!-- /wp:image -->

<!-- wp:separator -->
<hr class="wp-block-separator has-alpha-channel-opacity"/>
<!-- /wp:separator -->

<!-- wp:heading -->
<h2>Rightsize your Kubernetes Resources with Sysdig Monitor</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>With Sysdig Monitor’s new feature, Cost Advisor, you can optimize your Kubernetes costs:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Memory requests</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>CPU requests</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>With our out-of-the-box Kubernetes Dashboards, you can&nbsp;<a href="https://sysdig.com/blog/kubernetes-capacity-planning/" target="_blank" rel="noreferrer noopener">discover underutilized resources</a><br>in a couple of clicks.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":38348,"sizeSlug":"full","linkDestination":"custom"} -->
<figure class="wp-block-image size-full"><img src="https://sysdig.com/wp-content/uploads/Blog-How-to-do-capacity-planning-for-Kubernetes-Image-14.png" alt="Capacity planning Kubernetes Sysdig Monitor" class="wp-image-38348"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><a href="https://sysdig.com/company/free-trial-monitor/">Try it free for 30 days!</a></p>
<!-- /wp:paragraph -->