Repository: kubernetes
Issue: Kubelet removes non-Kubernetes images.
Labels: kind/bug,sig/node,priority/important-longterm,lifecycle/rotten,needs-triage

### What happened?

We also hit by this issue. We're using Flatcar Linux which is quite barebone so we are using a few containers to augment it. Unfortunately kubelet removes the images when they are not in active use. For example we a have a cronjob which uses an image we pulled and specially tagged. That images isn't used by any pods at all but kubelet choose to remove it under disk pressure. It is very annoying.

It would be nice to have a setting which says that "you can touch images only if the tag is starting with this prefix" or similar.

Even if you are using another Linux distribution then you might want to install something which isn't available in the distribution itself. Using a container is a very likely way to go and you don't expect that something which never belonged to the Kubernetes cluster will be removed by kubelet. It's very unintuitive.

### What did you expect to happen?

Kubelet won't remove images which aren't belongs to it.

### How can we reproduce it (as minimally and precisely as possible)?

* Pull some images from the command line.
* Fill up the local disk to create a DiskPressure "situation".
* Wait until kubelet removes the images - by the documentation it does in every five minutes.

### Anything else we need to know?

I created this new ticket because the old one (#103255) I found gets closed by k8s-ci-robot.

We are still using Kubernetes 1.21 as because of the faulty etcd-client lib used in 1.22 and 1.23 we cannot upgrade our clusters. The fix appeared in 1.24 but never backported to 1.22 or 1.23 which left us no possible upgrade paths.

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
Kubernetes cluster running on EC2 instances in AWS.
</details>


### OS version

<details>

```console
$ cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3033.2.3
VERSION_ID=3033.2.3
BUILD_ID=2022-03-02-2032
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3033.2.3 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"
$ uname -a
Linux ip-10-8-74-48.ec2.internal 5.10.102-flatcar #1 SMP Wed Mar 2 19:40:13 -00 2022 x86_64 AMD EPYC 7571 AuthenticAMD GNU/Linux
```

</details>


### Install tools

<details>
None.
</details>


### Container runtime (CRI) and version (if applicable)

<details>
N/A
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
N/A
</details>