Kubernetes ErrImagePull and ImagePullBackOff in detail
Tags: kubernetes, container, docker, orchestration
Published: 2022-10-05 19:34:01

---
title: Kubernetes ErrImagePull and ImagePullBackOff in detail
published: true
description: Learn more about Kubernetes statuses ErrImagePull and ImagePullBackOff. When do they happen? What do they mean? 
tags: kubernetes, container, docker, orchestration
cover_image: https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-00.png
canonical_url: https://sysdig.com/blog/kubernetes-errimagepull-imagepullbackoff/
# Use a ratio of 100:42 for best results.
---

<p>
Pod statuses like ImagePullBackOff or ErrImagePull are common when working with containers.
</p>
<p>
<strong>ErrImagePull</strong> is an error happening when the image specified for a container can’t be retrieved or pulled.
</p>
<p>
<strong>ImagePullBackOff</strong> is the waiting grace period while the image pull is fixed.
</p>
<p>
<a href="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-00.png"><img src="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-00.png" alt="ErrImagePull and ImagePullBackOff cover" width="1200" height="600" class="aligncenter size-full wp-image-53161 u-no-shadow"/></a>
</p>
<p>
In this article, we will take a look at:
</p>
<ul>
<li><a href="#containerimages">Container Images</a>
</li><li><a href="#pullingimages">Pulling Images</a>
</li><li><a href="#imagepullpolicy">Image Pull Policy</a>
</li><li><a href="#errimagepull">ErrImagePull</a>
</li><li><a href="#debuggingerrimagepull">Debugging ErrImagePull</a>
</li><li><a href="#monitoringimagepullerrors">Monitoring Image Pull Errors</a>
</li><li><a href="#otherimageerrors">Other Image Errors</a>
</li>
</ul>
<h2 id="containerimages">Container Images</h2>
<p>
One of the greatest strengths of containerization is the ability to run any particular image in seconds. A <b>container</b> is a group of processes executing in isolation from the underlying system. A<b> container image</b> contains all the resources needed to run those processes: the binaries, libraries, and any necessary configuration.
</p>
<p>
A <b>container registry</b> is a repository for container images, where there are two basic actions:
</p>
<ul>
<li><strong>Push</strong>: upload an image so it’s available in the repo
</li><li><strong>Pull</strong>: download an image to use it in a container
</li>
</ul>

<p><span style="text-size: smaller; font-style: italic; border: 1px solid #FF9E1B; display: block; border-radius: 25px; padding: 25px; max-width: 96%; margin: 2em auto;"><em>
The docker CLI will be used in the examples for this article, but you can use any tool that implements the Open Container Initiative Distribution specs for all the container registry interactions.
</em></span></p>

<h2 id="pullingimages">Pulling images</h2>
<p>
Images can be defined by name. In addition, a particular version of an image can be labeled with a specific <strong>name</strong> or <strong>tag</strong>. It can also be identified by its <strong>digest</strong>, a hash of the content.
</p>
<p>
The tag <code>latest</code> refers to the most recent version of a given image.
</p>


<h3>Pull images by name</h3>
<p>
By only providing the name for the image, the image with tag <code>latest</code> will be pulled
</p>
<pre class="prettyprint" style="margin: 1em 0">docker pull nginx
kubectl run mypod nginx
</pre>


<h3>Pull images by name + tag</h3>
<p>
If you don’t want to pull the <code>latest</code> image, you can provide a specific release tag:
</p>
<pre class="prettyprint" style="margin: 1em 0">docker pull nginx:1.23.1-alpine
kubectl run mypod nginx:1.23.1-alpine
</pre>
<p>
For more information, you can check this <a target="_blank" href="https://sysdig.es/blog/toctou-tag-mutability/" rel="noopener">article about tag mutability</a>.
</p>


<h3>Pull images by digest</h3>
<p>
A digest is sha256 hash of the actual image. You can pull images using this digest, then verify its authenticity and integrity with the downloaded file.
</p>
<pre class="prettyprint" style="margin: 1em 0">docker pull sha256:d164f755e525e8baee113987bdc70298da4c6f48fdc0bbd395817edf17cf7c2b
kubectl run mypod --image=nginx:sha25645b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5
</pre>


<h2 id="imagepullpolicy">Image Pull Policy</h2>
<p>
Kubernetes features the ability to set an <strong>Image Pull Policy</strong> (<strong>imagePullPolicy</strong> field) for each container. Based on this, the way the <code>kubelet</code> retrieves the container image will differ.
</p>
<p>
There are three different values for imagePullPolicy:
</p>
<ul>
<li>Always
</li><li>IfNotPresent
</li><li>Never
</li>
</ul>


<h3>Always</h3>
<p>
With imagePullPolicy set to <strong>Always</strong>, kubelet <strong>will check the repository every time</strong> when pulling images for this container.
</p>


<h3>IfNotPresent</h3>
<p>
With imagePullPolicy set to <strong>IfNotPresent</strong>, kubelet will only pull images from the repository<strong> if it doesn’t exist</strong> in the node locally.
</p>


<h3>Never</h3>
<p>
With imagePullPolicy set to <strong>Never</strong>, kubelet <strong>will never try to pull</strong> images from the image registry. If there’s an image cached locally (pre-pulled), it will be used to start the container.
</p>
<p>
If the image is not present locally, Pod creation will fail with <code>ErrImageNeverPull</code> error.
</p>
<p>
<a href="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-01.png"><img src="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-01.png" alt="ImagePullPolicy description: always, never and ifnotpresent" width="1200" height="600" class="aligncenter size-full wp-image-53161 u-no-shadow"/></a>
<p>
Note that you can modify the entire image pull policy of your cluster by using the AlwaysPullImages <a target="_blank" href="https://sysdig.com/blog/kubernetes-admission-controllers/" rel="noopener">admission controller</a>.
</p>


<h3>Default Image Pull Policy</h3>
<ul>
<li>If you omit the imagePullPolicy and the tag is <code>latest</code>, imagePullPolicy is set to <code>Always.</code>
</li><li>If you omit the imagePullPolicy and the tag for the image, imagePullPolicy is set to <code>Always</code>.
</li><li>If you omit the imagePullPolicy and the tag is set to a value different than <code>latest</code>, imagePullPolicy is set to <code>IfNotPresent.</code>
</li>
</ul>


<h2 id="errimagepull">ErrImagePull</h2>
<p>
When Kubernetes tries to pull an image for a container in a Pod, things might go wrong. The status <strong>ErrImagePull</strong> is displayed when <code>kubelet</code> tried to start a container in the Pod, but something was wrong with the image specified in your Pod, Deployment, or ReplicaSet manifest.
</p>
<p>
Imagine that you are using kubectl to retrieve information about the Pods in your cluster:
</p>
<pre class="prettyprint" style="margin: 1em 0">$ kubectl get pods
NAME    READY   STATUS             RESTARTS   AGE
goodpod 1/1     Running            0          21h
mypod   0/1     ErrImagePull       0          4s
</pre>
<p>
Which means:
</p>
<ul>
<li>Pod is not in <code>READY</code> status
</li><li>Status is <code>ErrImagePull</code>
</li>
</ul>
<p>
Additionally, you can check the logs for containers in your Pod:
</p>
<pre class="prettyprint" style="margin: 1em 0">$ kubectl logs mypod --all-containers
Error from server (BadRequest): container "mycontainer" in pod "mypod" is waiting to start: trying and failing to pull image
</pre>
<p>
In this case, this is pointing to a 400 Error (BadRequest), since probably the image indicated is not available or doesn’t exist.
</p>


<h2 id="imagepullbackoff">ImagePullBackOff</h2>
<p>
ImagePullBackOff is a Kubernetes waiting status, a grace period with an increased back-off between retries. After the back-off period expires, kubelet will try to pull the image again.
</p>
<p>
This is similar to the <a target="_blank" href="https://sysdig.com/blog/debug-kubernetes-crashloopbackoff/" rel="noopener">CrashLoopBackOff status</a>, which is also a grace period between retries after an error in a container. Back-off time is increased each retry, up to a maximum of five minutes.
</p>
<p>
Note that ImagePullBackOff is not an error. As mentioned, it’s just a status reason that is caused by a problem when pulling the image.
</p>
<pre class="prettyprint" style="margin: 1em 0">$ kubectl get pods
NAME    READY   STATUS             RESTARTS   AGE
goodpod 1/1     Running            0          21h
mypod   0/1     ImagePullBackOff   0          84s
</pre>
<p>
Which means:
</p>
<ul>
<li>Pod is not in <code>READY</code> status
</li><li>Status is <code>ImagePullBackOff</code>
</li><li>Unlike CrashLoopBackOff, there are no restarts (technically Pod hasn’t even started)
</li>
</ul>
<pre class="prettyprint" style="margin: 1em 0">$ kubectl describe pod mypod
State:          Waiting
Reason:       ImagePullBackOff
...
Warning  Failed     3m57s (x4 over 5m28s)  kubelet            Error: ErrImagePull
Warning  Failed     3m42s (x6 over 5m28s)  kubelet            Error: ImagePullBackOff
Normal   BackOff    18s (x20 over 5m28s)   kubelet            Back-off pulling image "failed-image"
</pre>
<p>
<a href="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-02.png"><img src="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-02.png" alt="ErrImagePull and ImagePullBackOff timeline" width="1200" height="600" class="aligncenter size-full wp-image-53161 u-no-shadow"/></a>


<h2 id="debuggingerrimagepull">Debugging ErrImagePull and ImagePullBackOff</h2>
<p>
There are several potential causes of why you might encounter an Image Pull Error. Here are some examples:
</p>
<ul>
<li>Wrong image name
</li><li>Wrong image tag
</li><li>Wrong image digest
</li><li>Network problem or image repo not available
</li><li>Pulling from a private registry but not imagePullSecret was provided
</li>
</ul>
<p>
This is just a list of possible causes, but it’s important to note that there might be many others based on your solution. The best course of action would be to check:
</p>
<pre class="prettyprint" style="margin: 1em 0">
$ kubectl describe pod podname

$ kubect logs podname –all-containers

$ kubectl get events --field-selector involvedObject.name=podname
</pre>
<p>
In the following example you can see how to dig into the logs, where an image error is found.

<a href="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-03.png"><img src="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-03.png" alt="Three terminals with debugging options for ErrImagePull and ImagePullBackOff" width="1200" height="600" class="aligncenter size-full wp-image-53161 u-no-shadow"/></a>

<h2 id="otherimageerrors">Other image errors</h2>


<h3>ErrImageNeverPull</h3>
<p>
This error appears when kubelet fails to pull an image in the node and the imagePullPolicy is set to Never. In order to fix it, either change the Pull Policy to allow images to be pulled externally or add the correct image locally.
</p>


<h3>Pending</h3>
<p>
Remember that an ErrImagePull and the associated ImagePullBackOff may be different from a Pending status on your Pod.
</p>
<p>
Pending status, most likely, is the result of kube-scheduler not being able to assign your Pod to a working or eligible Node.
</p>


<h2 id="monitoringimagepullerrors">Monitoring Image Pull Errors in Prometheus</h2>
<p>
Using Prometheus and Kube State Metrics (KSM), we can easily track our Pods with containers in ImagePullBackOff or ErrImagePull statuses.
</p>
<pre class="prettyprint" style="margin: 1em 0">kube_pod_container_status_waiting_reason{reason="ErrImagePull"}
kube_pod_container_status_waiting_reason{reason="ImagePullBackOff"}
</pre>
<p>
In fact, these two metrics are complementary, as we can see in the following Prometheus queries:
</p>
<p>
<a href="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-04-1.png"><img src="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-04-1.png" alt="Monitoring ErrImagePull and ImagePullBackOff in Prometheus" width="1200" height="600" class="aligncenter size-full wp-image-53161 u-no-shadow"/></a>
<p>
The Pod is shifting between the waiting period in ImagePullBackOff and the image pull retrial returning an ErrImagePull.
</p>
<p>
Also, if you’re using containers with <code>ImagePullPolicy</code> set to <code>Never</code>, remember that you need to track the error as <code>ErrImageNeverPull</code>.
</p>
<pre class="prettyprint" style="margin: 1em 0">kube_pod_container_status_waiting_reason{reason="ErrImageNeverPull"}
</pre>


<h2 id="conclusion">Conclusion</h2>
<p>
Container images are a great way to kickstart your cloud application needs. Thanks to them, you have access to thousands of curated applications that are ready to be started and scaled.
</p>
<p>
However, due to misconfiguration, misalignments, or repository problems, image errors might start appearing. A container can’t start properly if the image definition is malformed or there are errors on the setup.
</p>
<p>
Kubernetes provides a graceful period in case of an image pull error. This Image Pull Backoff is quite useful, as it gives you time to fix the problem in the image definition. But you need to be aware when this happens in your cluster and what does it mean each time.
</p>
<hr>


<h2>Troubleshoot Image Pull Errors with Sysdig Monitor</h2>
<p>
With the Advisor feature of Sysdig Monitor, you can easily review Image Pull Errors happening in your Kubernetes cluster.
</p>
<p>
Sysdig Advisor accelerates mean time to resolution (MTTR) with live logs, performance data, and suggested remediation steps. It’s the easy button for Kubernetes troubleshooting!
</p>
<p>
<a href="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-05.png"><img src="https://sysdig.com/wp-content/uploads/ErrImagePull-ImagePullbackOff-05.png" alt="Troubleshooting ErrImagePull and ImagePullBackOff in Sysdig Monitor" width="1200" height="600" class="aligncenter size-full wp-image-53161 u-no-shadow"/></a>
<p>
<a target="_blank" href="https://sysdig.com/company/free-trial-monitor/" rel="noopener">Try it free for 30 days!</a>
</p>