Repository: kubernetes
Issue: kubelet evented panic when use generic pleg relisting 
Labels: kind/bug,sig/node,triage/needs-information,needs-triage

### What happened?

kubelet evented panic when use generic pleg relisting，the log is like:
```
Oct 21 06:03:47  kubelet[1530]: E1021 06:03:47.241522    1530 remote_runtime.go:550] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="&ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},}"
Oct 21 06:03:48  kubelet[1530]: E1021 06:03:48.844680    1530 remote_runtime.go:550] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="&ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},}"
Oct 21 06:03:48  kubelet[1530]: E1021 06:03:48.844719    1530 resource_metrics.go:118] "Error getting summary for resourceMetric prometheus endpoint" err="failed to list pod stats: failed to list all containers: rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Oct 21 06:03:51  kubelet[1530]: E1021 06:03:51.478833    1530 evented.go:356] "Evented PLEG: Get cache" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" podID=3c55a6f7-4909-4cc8-a48c-c534a6b94304
Oct 21 06:03:53  kubelet[1530]: E1021 06:03:51.580766    1530 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Oct 21 06:03:53  kubelet[1530]: goroutine 369 [running]:
Oct 21 06:03:53  kubelet[1530]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x43ea9e0, 0x77e53c0)
Oct 21 06:03:53  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
Oct 21 06:03:53  kubelet[1530]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
Oct 21 06:03:53  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86
Oct 21 06:03:53  kubelet[1530]: panic(0x43ea9e0, 0x77e53c0)
Oct 21 06:03:53  kubelet[1530]: /usr/local/go/src/runtime/panic.go:965 +0x1b9
Oct 21 06:03:54  kubelet[1530]: k8s.io/kubernetes/pkg/kubelet/pleg.(*EventedPLEG).updateRunningPodMetric(0xc000a18a00, 0xc0011af950)
Oct 21 06:03:54  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/evented.go:359 +0x7d
Oct 21 06:03:54  kubelet[1530]: k8s.io/kubernetes/pkg/kubelet/pleg.(*EventedPLEG).processCRIEvents(0xc000a18a00, 0xc0010ba840)
Oct 21 06:03:54  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/evented.go:251 +0x3b2
Oct 21 06:03:54  kubelet[1530]: k8s.io/kubernetes/pkg/kubelet/pleg.(*EventedPLEG).watchEventsChannel(0xc000a18a00)
Oct 21 06:03:54  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/evented.go:206 +0xec
Oct 21 06:03:54  kubelet[1530]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00181e0a0)
Oct 21 06:03:54  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
Oct 21 06:03:54  kubelet[1530]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00181e0a0, 0x53f5b20, 0xc0014981e0, 0x1, 0xc00212a120)
Oct 21 06:03:54  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
Oct 21 06:03:54  kubelet[1530]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00181e0a0, 0x0, 0x0, 0x1, 0xc00212a120)
Oct 21 06:03:54  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
Oct 21 06:03:54  kubelet[1530]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00181e0a0, 0x0, 0xc00212a120)
Oct 21 06:03:54  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
Oct 21 06:03:54  kubelet[1530]: created by k8s.io/kubernetes/pkg/kubelet/pleg.(*EventedPLEG).Start
Oct 21 06:03:54  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/evented.go:124 +0x148
Oct 21 06:04:48  kubelet[1530]: panic: send on closed channel
Oct 21 06:04:48  kubelet[1530]: goroutine 403 [running]:
Oct 21 06:04:49  kubelet[1530]: k8s.io/kubernetes/pkg/kubelet/cri/remote.(*remoteRuntimeService).GetContainerEvents(0xc00081ea20, 0xc0010ba840, 0x0, 0x1)
Oct 21 06:04:49  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/cri/remote/remote_runtime.go:1230 +0x198
Oct 21 06:04:49  kubelet[1530]: k8s.io/kubernetes/pkg/kubelet/pleg.(*EventedPLEG).watchEventsChannel.func1(0xc000a18a00, 0xc0010ba840)
Oct 21 06:04:49  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/evented.go:195 +0x69
Oct 21 06:04:49  kubelet[1530]: created by k8s.io/kubernetes/pkg/kubelet/pleg.(*EventedPLEG).watchEventsChannel
Oct 21 06:04:49  kubelet[1530]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/evented.go:180 +0xac
Oct 21 06:04:48  systemd[1]: kubelet.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 21 06:04:48  systemd[1]: Unit kubelet.service entered failed state.
Oct 21 06:04:48  systemd[1]: kubelet.service failed.
```

### What did you expect to happen?

kubelet not panic

### How can we reproduce it (as minimally and precisely as possible)?

It's hard to reproduce the rpc call "context deadline exceeded" timeout situation, So I added some codes in kubelet to construct this.
1. add the patch for kubelet
```
 pkg/kubelet/cri/remote/remote_runtime.go       | 8 ++++++++
 pkg/kubelet/kuberuntime/kuberuntime_manager.go | 5 +++++
 2 files changed, 13 insertions(+)

diff --git a/pkg/kubelet/cri/remote/remote_runtime.go b/pkg/kubelet/cri/remote/remote_runtime.go
index 424824467b5..d48d6cffb7a 100644
--- a/pkg/kubelet/cri/remote/remote_runtime.go
+++ b/pkg/kubelet/cri/remote/remote_runtime.go
@@ -1206,6 +1206,7 @@ func (r *remoteRuntimeService) CheckpointContainer(options *runtimeapi.Checkpoin
 	return nil
 }

+var eventErrored bool = false
 func (r *remoteRuntimeService) GetContainerEvents(containerEventsCh chan *runtimeapi.ContainerEventResponse) error {
 	containerEventsStreamingClient, err := r.runtimeClient.GetContainerEvents(context.Background(), &runtimeapi.GetEventsRequest{})
 	if err != nil {
@@ -1227,6 +1228,13 @@ func (r *remoteRuntimeService) GetContainerEvents(containerEventsCh chan *runtim
 			return err
 		}
 		if resp != nil {
+			if !eventErrored {
+				if resp.PodSandboxStatus != nil && resp.PodSandboxStatus.Metadata != nil &&
+					resp.PodSandboxStatus.Metadata.Name == "centos7-pod" {
+					eventErrored = true
+					return errors.New("test error")
+				}
+			}
 			containerEventsCh <- resp
 			klog.V(4).InfoS("container event received", "resp", resp)
 		}
diff --git a/pkg/kubelet/kuberuntime/kuberuntime_manager.go b/pkg/kubelet/kuberuntime/kuberuntime_manager.go
index 1c5e545fa9d..f10263c43ce 100644
--- a/pkg/kubelet/kuberuntime/kuberuntime_manager.go
+++ b/pkg/kubelet/kuberuntime/kuberuntime_manager.go
@@ -1040,6 +1040,7 @@ func (m *kubeGenericRuntimeManager) GeneratePodStatus(event *runtimeapi.Containe
 	}, nil
 }

+var errorGetPod bool = false
 // GetPodStatus retrieves the status of the pod, including the
 // information of all containers in the pod that are visible in Runtime.
 func (m *kubeGenericRuntimeManager) GetPodStatus(uid kubetypes.UID, name, namespace string) (*kubecontainer.PodStatus, error) {
@@ -1073,6 +1074,10 @@ func (m *kubeGenericRuntimeManager) GetPodStatus(uid kubetypes.UID, name, namesp

 	klog.V(4).InfoS("getSandboxIDByPodUID got sandbox IDs for pod", "podSandboxID", podSandboxIDs, "pod", klog.KObj(pod))

+	if pod.Name == "centos7-pod" && errorGetPod == false {
+		errorGetPod = true
+		return nil, errors.New("test2 error")
+	}
 	sandboxStatuses := []*runtimeapi.PodSandboxStatus{}
 	containerStatuses := []*kubecontainer.Status{}
 	timestamp := time.Now()
--
2.39.3 (Apple Git-146)
```
2. create a pod, then the kubelet panic
```

```

### Anything else we need to know?

_No response_

### Kubernetes version

kubelet 1.21 with latest evented pleg.


### Cloud provider

None


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>
 Centos7

### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
containerd 1.7
</details>