Repository: kubernetes
Issue: The image of a running container was deleted by the image garbage collection
Labels: kind/bug,sig/node,priority/important-longterm,triage/accepted

### What happened?

The image of the container was deleted while pod was running.
When I noticed it, it was already almost two weeks after the pod was created, and by prometheus metrics, I could see at the time that the pod was created the kubelet performed image garbage collection on the node and images were deleted.

It might be worth mentioning that nothing other than the kubelet could have deleted the image from the node.

This situation happen a while ago, and I couldn't deal with it for too long while it happened, So unfortunately I don't have data to share. But by looking in the code the following scenario seems to make sense:
1. At the start of the garbage collection process the image was already in the node (for a while), but not in used yet. 
2. The garbage collection get the current images and pods on that node from the runtime, because this image is not in use by any pod, it is considered unused.  Ref: https://github.com/kubernetes/kubernetes/blob/b340ef2e29b1694464bee60487160654d4bf8bbc/pkg/kubelet/images/image_gc_manager.go#L232-L247
3. Before the image is actually deleted, A pod using this image is scheduling at this node. since the image is already on the node and the `imagePullPolicy` is `IfNotPresent` there is no need for pulling and the container is start to running. There is no knowledge about that the image is about to be deleted. Ref: https://github.com/kubernetes/kubernetes/blob/b340ef2e29b1694464bee60487160654d4bf8bbc/pkg/kubelet/images/image_manager.go#L126-L143
4. The garbage collection is still running, and it eventually comes to delete this image. There is no knowledge about that the image is in use now.

In short, I think that in the time between pods are detected from runtime to find the unused image to the time those images are  actually deleted (the time between step 2 to step 4), new pods could be scheduling on that node, and make use of the images that will be deleted by the garbage collection immediately afterwards. Although the chances of this scenario are rare, the troublesome time I described would be bigger when the node could contain a lot of images and pods, and the difference between `HighThresholdPercent` and `LowThresholdPercent` have an effect as well.



### What did you expect to happen?

The image garbage collection should not delete used images. 

### How can we reproduce it (as minimally and precisely as possible)?

Suppose an unused image X has already been on the node for some time.
The timeline is designed as following,
![image](https://github.com/kubernetes/kubernetes/assets/34262680/01a8992a-95d8-44cb-bbca-8bcf249496a8)
Any pod that uses image X and is scheduling on the node within the orange zone will cause this issue.


### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:36:28Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
local
</details>


### OS version

_No response_

### Install tools

_No response_

### Container runtime (CRI) and version (if applicable)

<details>
CRI-O v1.27.3 
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

_No response_