Repository: kubernetes
Issue: Kubelet reporting 137 exit code for a container whose exit code 0 was previously known
Labels: kind/bug,area/kubelet,sig/node,needs-triage

#### What happened:
I have pods with a `RestartPolicy: Never` that perform a task, wait for a sigterm, clean up and then exit. These pods have only one container inside them. I have observed on kubernetes > 1.20.0, that after issuing a graceful delete, sometimes these pods will be stuck in `Terminating`. Upon describing the pod, I can see the following state:

```
    State:          Waiting
      Reason:       ContainerCreating
    Last State:     Terminated
      Reason:       ContainerStatusUnknown
      Message:      The container could not be located when the pod was deleted.  The container used to be Running
      Exit Code:    137
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
```

I have confirmed from the `containerd` logs (the container runtime I am using) and with `strace`, that the pods are receiving a single `sigterm` and then exiting, with an exitCode of 0. I have been patching log statements into kubelet and I am now confident I understanding _what_ is happening, though I do not know why. All of the following statements relate specifically to commit `4aa451e8458a7cbf78ed464e9e47e87d424541ce`, which I believe is the sha for kubernetes `1.21.1`.

In https://github.com/kubernetes/kubernetes/blob/4aa451e8458a7cbf78ed464e9e47e87d424541ce/pkg/kubelet/kubelet.go#L1960-L1982, upon receiving a `pleg.ContainerDied` event for the pod sandbox, which for `containerd` is a container, https://github.com/kubernetes/kubernetes/blob/4aa451e8458a7cbf78ed464e9e47e87d424541ce/pkg/kubelet/kubelet.go#L1980 is called. This leads to us calling https://github.com/kubernetes/kubernetes/blob/4aa451e8458a7cbf78ed464e9e47e87d424541ce/pkg/kubelet/kubelet.go#L2268-L2269, and the returned `podStatus` is passed to https://github.com/kubernetes/kubernetes/blob/4aa451e8458a7cbf78ed464e9e47e87d424541ce/pkg/kubelet/kubelet.go#L2273. This takes us to https://github.com/kubernetes/kubernetes/blob/4aa451e8458a7cbf78ed464e9e47e87d424541ce/pkg/kubelet/kubelet_pods.go#L1519-L1522 where we immediately call `convertStatusToAPIStatus` with the `podStatus` arg. This function calls https://github.com/kubernetes/kubernetes/blob/4aa451e8458a7cbf78ed464e9e47e87d424541ce/pkg/kubelet/kubelet_pods.go#L1621-L1627, and we end up setting the ExitCode to 137 https://github.com/kubernetes/kubernetes/blob/4aa451e8458a7cbf78ed464e9e47e87d424541ce/pkg/kubelet/kubelet_pods.go#L1810-L1817.  Why does that happen? Well, it turns out that `podStatus.ContainerStatuses` is an empty slice at this point in time. 

Knowing that we got that `podStatus` from `kl.PodCache.Get`, I trace backwards and find that the slice of `ContainerStatuses` is empty because the last time this cache was updated, at `pleg/generic.go#updatecache`, we call `container/runtime.go#GetPodStatus` which is implemented in `kuberuntime_manager.go#GetPodStatus`, which calls `kuberuntime_container.go#getPodContainerStatuses`, which calls `cri/remote_riuntime.go#ListContainers`, which is getting an empty list of containers back from `containerd`, thus is ends up putting an empty list of container statuses in the cache. Just to confirm, I can see that the `ListContainers` request kubelet is sending seems correct, and has the correct pod UID in the filter.

#### What you expected to happen:

I've compared the above situation to a pod that cleans up correctly. Ultimately, I can only find one difference. For a "good" pod, inside `syncLoopIteration` in `kubelet.go`, we see a `pleg.ContainerDied` for the container, then a `pleg.ContainerDied` for the pod sandbox, then a `pleg.ContainerRemoved` for the container.

For a "bad" pod, the order is changed. We see the `pleg.ContainerDied` for the container, then a `pleg.ContainerRemoved` for the container, then the `pleg.ContainerDied` for the pod sandbox. In no case do I ever see a `pleg.ContainerRemoved` for the pod sandbox.

I would expect, given that the only container is exiting with 0, and `containerd` does know that at some point, that it would be reflected correctly in kubelet. 

#### How to reproduce it (as minimally and precisely as possible):

I am very sorry, but I have not been able to write a test that reproduces this. If that's a blocker, I can try to write a pod similar enough to [my pod spec](https://github.com/DataDog/chaos-controller/blob/8c64ef47a9e14cc79af287d85b380b0d791666eb/controllers/disruption_controller.go#L679-L831) to reproduce. I am happy to share any logs, run any patches, assist in any way, etc. If anyone has guidance on what _could_ be wrong, I can look into that on my own as well.

I have tested many times and am completely unable to reproduce this on the kubernetes 1.19 line. I am able to reproduce it very consistently on all tested kubernetes 1.20 and above, but have only debugged in 1.21.1 and only care about a fix there. 

We have found that the more pods on a single node, and the more pods being deleted at once, increase the odds of reproduction. Beyond that, I have no reproduction steps beyond creating many pods with a `RestartPolicy: Never` and then deleting them after they're Ready/Running.

#### Anything else we need to know?:

I have noticed that all of this occurs while the pod is gracefully terminating. Inside `kubelet.go#syncPod`, there is a check if a pod is currently pending termination. Locally, I have found that the map being checked never has anything in it, and we always return `false` when checking if the pod is pending termination, even when it is. I do not know if this is related to the behavior, but it was notable during debugging.

I can provide kubelet or containerd logs at any verbosity requested. I am comfortable patching kubelet with more logging if there's something specific I can test for you.

#### Environment:
- Kubernetes version (use `kubectl version`): 1.21.1
- Cloud provider or hardware configuration: MBP 13" 2020
- OS (e.g: `cat /etc/os-release`): Underlying host OS is OS X 11.15
- Kernel (e.g. `uname -a`): Linux minikube 4.19.157 # 1 SMP Mon Jan 18 19:46:48 UTC 2021 x86_64 GNU/Linux
- Install tools:
- Network plugin and version (if this is a network-related bug):
- Others: Running minikube 1.22.0 with containerd 1.4.3