Repository: kubernetes
Issue: Pods not consistently being cleaned up 
Labels: kind/bug,priority/important-soon,priority/backlog,sig/node,lifecycle/frozen,needs-triage

### What happened?

We have a K8S 1.22.3 environment using containerd (no docker) where pods are not being consistently cleaned up. The basic idea is that we have a process that spawns several pods every couple of minutes, and those pods are subsequently deleted. They do not show up doing a 'kubectl get pods'. However, running 'crictl pods' shows that these pods sometimes stick around in a NotReady state. For example:

```
[root@node1 ~]# crictl -r unix:///run/containerd/containerd.sock pods | grep prepull
57d34e9d15bc5       8 seconds ago        Ready               prepull-ds-hlavcfws-2mzqq                             delme1              0                   (default)
2da9a12e06252       29 seconds ago       Ready               prepull-ds-gejesdmn-mt59g                             delme1              0                   (default)
0d5798229c2cb       About a minute ago   NotReady            prepull-ds-sdowkjqg-g4mgd                             delme2              0                   (default)
bdb0bd23b39ad       8 minutes ago        NotReady            prepull-ds-yyqznjxk-vwct7                             delme2              0                   (default)
f0a780e01801e       10 minutes ago       NotReady            prepull-ds-dkeknuby-q85f2                             delme2              0                   (default)
c83fab4498c4f       11 minutes ago       NotReady            prepull-ds-rdwxyosg-d9f2l                             delme1              0                   (default)
1308e12003c7d       13 minutes ago       NotReady            prepull-ds-sspwnvgl-gcrcd                             delme2              0                   (default)
4563e0f9bb8c7       17 minutes ago       NotReady            prepull-ds-nasjpdvl-lcbzp                             delme1              0                   (default)
c77561daacc75       19 minutes ago       NotReady            prepull-ds-furbleho-v7rmk                             delme2              0                   (default)
15995bb3ab38f       23 minutes ago       NotReady            prepull-ds-batbenhc-ckrs8                             delme2              0                   (default)
a07d5264768d7       24 minutes ago       NotReady            prepull-ds-mzydnjji-cc6dt                             delme1              0                   (default)
a980810199ae9       43 minutes ago       NotReady            prepull-ds-bjijjjlj-k64p4                             delme1              0                   (default)
3a48e17074d66       43 minutes ago       Ready               foo-prepull-5759955774-nbj92                          delme1              0                   (default)
141b7dc77b0b0       44 minutes ago       NotReady            prepull-ds-suzddozj-kfbrd                             delme2              0                   (default)
8eb8b1a3a217b       45 minutes ago       NotReady            prepull-ds-uqyvjtgb-bwbmr                             delme2              0                   (default)
```

If I examine one of those pods, such as **prepull-ds-nasjpdvl-lcbzp**, then kubectl doesn't show it existing:

```
[root@node1 ~]# kubectl -n delme1 get pod prepull-ds-nasjpdvl-lcbzp
Error from server (NotFound): pods "prepull-ds-nasjpdvl-lcbzp" not found
```

The kubelet log output related to that pod, if of any value:

```
[root@node1 ~]# journalctl -u kubelet -n 100000 | grep prepull-ds-nasjpdvl-lcbzp
Feb 10 11:31:47 node1 kubelet[3123]: I0210 11:31:47.892981    3123 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce16f07-3fb3-42c5-ab0b-7b9c4b5c5672-tmp\") pod \"prepull-ds-nasjpdvl-lcbzp\" (UID: \"8ce16f07-3fb3-42c5-ab0b-7b9c4b5c5672\") "
Feb 10 11:31:47 node1 kubelet[3123]: I0210 11:31:47.893042    3123 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjb5n\" (UniqueName: \"kubernetes.io/projected/8ce16f07-3fb3-42c5-ab0b-7b9c4b5c5672-kube-api-access-pjb5n\") pod \"prepull-ds-nasjpdvl-lcbzp\" (UID: \"8ce16f07-3fb3-42c5-ab0b-7b9c4b5c5672\") "
Feb 10 11:32:40 node1 kubelet[3123]: E0210 11:32:40.168208    3123 kuberuntime_container.go:719] "Container termination failed with gracePeriod" err="rpc error: code = NotFound desc = an error occurred when try to find container \"997e867c6b77c4a53dd24ed82b5f0ff4522832563b6ec37750e81c25071c61f7\": not found" pod="delme1/prepull-ds-nasjpdvl-lcbzp" podUID=8ce16f07-3fb3-42c5-ab0b-7b9c4b5c5672 containerName="prepull-pod-mkxjvcaj" containerID="containerd://997e867c6b77c4a53dd24ed82b5f0ff4522832563b6ec37750e81c25071c61f7" gracePeriod=1
Feb 10 11:32:40 node1 kubelet[3123]: E0210 11:32:40.168255    3123 kuberuntime_container.go:744] "Kill container failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"997e867c6b77c4a53dd24ed82b5f0ff4522832563b6ec37750e81c25071c61f7\": not found" pod="delme1/prepull-ds-nasjpdvl-lcbzp" podUID=8ce16f07-3fb3-42c5-ab0b-7b9c4b5c5672 containerName="prepull-pod-mkxjvcaj" containerID={Type:containerd ID:997e867c6b77c4a53dd24ed82b5f0ff4522832563b6ec37750e81c25071c61f7}
Feb 10 11:32:40 node1 kubelet[3123]: E0210 11:32:40.191724    3123 pod_workers.go:836] "Error syncing pod, skipping" err="failed to \"KillContainer\" for \"prepull-pod-mkxjvcaj\" with KillContainerError: \"rpc error: code = NotFound desc = an error occurred when try to find container \\\"997e867c6b77c4a53dd24ed82b5f0ff4522832563b6ec37750e81c25071c61f7\\\": not found\"" pod="delme1/prepull-ds-nasjpdvl-lcbzp" podUID=8ce16f07-3fb3-42c5-ab0b-7b9c4b5c5672
```

Normally I wouldn't care about these NotReady pods sticking around, but the bigger problem is that they continue to accumulate, until eventually kubelet fails with an error such as:

`"GenericPLEG: Unable to retrieve pods" err="rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (16784723 vs. 16777216)"`

Once we get into that state we can run 'crictl pods' and get back a similar error:

```
[root@node1 tmp]# crictl pods
FATA[0002] listing pod sandboxes: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (16783177 vs. 16777216)
```

Basically the number of pods returned by 'crictl pods' grows so large that it exceeds the message size. Note that we do not see this problem on a 1.20.11 cluster that is configured very similarly. In that case no NotReady pods stick around long term like we're seeing on 1.22.3. In case it's relevant, we also see a lot of the sandbox containers persisting:

```
[root@node1 ~]# nerdctl --namespace k8s.io ps -a | grep prepull | grep Created
1308e12003c7    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  22 minutes ago        Created             k8s://delme2/prepull-ds-sspwnvgl-gcrcd
141b7dc77b0b    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  53 minutes ago        Created             k8s://delme2/prepull-ds-suzddozj-kfbrd
15995bb3ab38    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  32 minutes ago        Created             k8s://delme2/prepull-ds-batbenhc-ckrs8
229f355b1f68    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  About a minute ago    Created             k8s://delme2/prepull-ds-ykrjmsge-kxdmf
3bdce493a500    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  7 minutes ago         Created             k8s://delme2/prepull-ds-wgnvtvhy-2n5cc
4563e0f9bb8c    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  26 minutes ago        Created             k8s://delme1/prepull-ds-nasjpdvl-lcbzp
49aa4c6e93ed    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  5 minutes ago         Created             k8s://delme1/prepull-ds-rwjoifro-p6x9d
8eb8b1a3a217    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  54 minutes ago        Created             k8s://delme2/prepull-ds-uqyvjtgb-bwbmr
a07d5264768d    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  32 minutes ago        Created             k8s://delme1/prepull-ds-mzydnjji-cc6dt
a980810199ae    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  52 minutes ago        Created             k8s://delme1/prepull-ds-bjijjjlj-k64p4
bdb0bd23b39a    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  16 minutes ago        Created             k8s://delme2/prepull-ds-yyqznjxk-vwct7
c77561daacc7    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  28 minutes ago        Created             k8s://delme2/prepull-ds-furbleho-v7rmk
c83fab4498c4    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  20 minutes ago        Created             k8s://delme1/prepull-ds-rdwxyosg-d9f2l
ddf619213c49    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  8 minutes ago         Created             k8s://delme1/prepull-ds-bclemxxo-ktmb6
f0a780e01801    k8s.gcr.io/pause:3.5                                                                                                        "/pause"                  19 minutes ago        Created             k8s://delme2/prepull-ds-dkeknuby-q85f2
```

Perhaps the containers sticking around keeps the pods alive? I'm just hunting for clues at this point. Thanks for any assistance.

### What did you expect to happen?

Deleted pods to be totally removed, not persisting indefinitely.

### How can we reproduce it (as minimally and precisely as possible)?

I don't know. We have an in-house process to create these clusters using OpenSource K8S. It's easy to reproduce in our environment, but I don't have a suggestion on reproducing elsewhere.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
None. This is on physical hardware.
</details>

### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

$ uname -a
Linux node1 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
```
</details>


### Install tools

<details>
kubeadm created cluster
</details>


### Container runtime (CRI) and and version (if applicable)

<details>
Originally observed using containerd 1.5.0. Upgraded to 1.5.9 with no obvious change in behavior.

Note that the 1.20.11 cluster that does not exhibit these problems also runs with 1.5.0 with an identical containerd configuration.
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>