Repository: kubernetes
Issue: Pending image downloads of deleted containers are never interrupted, leading to possible DoS
Labels: priority/backlog,sig/node,kind/feature,triage/accepted

### What happened?

During a research project on Kubernetes, we discovered a potential bug on any reasonably recent Kubernetes version (we tested in 1.25 to 1.29) that affects all clusters regardless of the container runtime used. By exploiting the fact that the CRI-API does not maintain the container download status, we created a denial of service via resource exhaustion.

----

When an user deploys a pod, the deployed container(s) may rely on image(s) that are not present on the target node, triggering a download by the container runtime. However, if that pod is deleted (either with or without the `--force` flag) before the container runtime completes the download of the image, this download will be carried on anyway. Kubernetes tracks no information at all about the download, and there is no way of interrupting it unless the container runtime is forcibly terminated and recreated.

This (useless, as the images will never be used) sequence of downloads will waste the node’s CPU and network bandwidth and block the download queue, preventing the deployment of other containers with images yet to be downloaded. We experimented on a wide variety of settings, including but not limited to private clouds, public clouds and edge devices, with different container runtimes (containerd and CRI-O) and different Kubernetes providers (kubeadm, microk8s, and GKE).

The impact this bug has on the cluster is twofold. First, by the way the download queue is implemented in the container runtime, queuing up multiple image download requests prevents legitimate users from deploying workloads that require an image download to be run. Since deleting a container from Kubernetes does not affect the container runtime download queue, this queue can grow uncontrolled **even if the requests are canceled before the download starts**. This starvation can continue for as long as the attacker likes, as long as they have access to the cluster and can request to download fresh images not present on the node.

Secondly, this bug has a significant impact on the target node's resources. This uncontrolled resource usage is untied from any service account or quota and will potentially skew the information available to the scheduler. Furthermore, even a quota of exactly one Pod does not hamper the attack as employing the `--force` flag allows attackers to rapidly deploy downloads and wipe the Pod from the API, never exceeding it.


### What did you expect to happen?

As a Pod is deleted but its containers have not been instantiated yet and their image is still being downloaded or is queued for download, such download should be aborted or evicted from the queue as soon as the target node is informed of the deletion.

### How can we reproduce it (as minimally and precisely as possible)?

1. Install a fresh Kubernetes cluster using kubeadm. We used versions 1.25 to 1.29. No modifications are needed to any configuration of the cluster. Service Accounts, Quotas, RBAC/ABAC configurations, etc… are not relevant to the attack’s success.
2. Identify a node to attack in the cluster if there is more than one. Add a NodeSelector to any Pod deployed in subsequent steps so that they can only be deployed on that node.
3. Identify one or more Docker images that are able to be downloaded from the cluster and are not present in the node’s image cache. The content, configuration, etc.. of the image is irrelevant.
4. Repeat until there are non-downloaded Docker images available:
  - Deploy a Pod (with the NodeSelector) of a certain image i;
  - Wait 1-2 seconds;
  - Request the Pod’s deletion (possibly with --force)
5. The victim node’s image download queue will now be filled up with bogus requests, preventing any user's download of further images in the cluster until the attack is stopped and the downloads complete or the node’s containerd process is restarted.
6. On clusters with k8s version >= 1.27 and MaxParallelImagePulls set to > 1, the attack will still be successful, so long as the attacker manages to request enough images to fill up the parallelism provided by MaxParallelImagePulls. 

### Anything else we need to know?

We contacted the Kubernetes Security Team in late 2023, and we were informed that this issue is not considered a vulnerability and we could proceed with making it public at our own discretion.

We extensively studied this issue, its impact, limitations and a proof of concept for blocking it in a publication. The paper is currently under evaluation for a submission in a journal, and its preprint is available at this link: [https://arxiv.org/abs/2401.10582](https://arxiv.org/abs/2401.10582)

### Kubernetes version

<details>

This details refer to a specific set of tests we made in the past, but we replicated the attack in several other scenarios.

```console
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"93e0d7146fb9c3e9f68aa41b2b4265b2fcdb0a4c", GitTreeState:"clean", BuildDate:"2023-08-24T00:42:11Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
  Private cloud, GKE
</details>


### OS version

<details>

```console
$ uname -a
Linux master 5.4.0-146-generic #163-Ubuntu SMP Fri Mar 17 18:26:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
Distributor ID:    Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:    20.04
Codename:    focal
```

</details>


### Install tools

_No response_

### Container runtime (CRI) and version (if applicable)

_No response_

### Related plugins (CNI, CSI, ...) and versions (if applicable)

_No response_