Repository: kubernetes
Issue: [EventedPLEG] EventedPLEG should be more tolerant of runtime outage
Labels: kind/bug,sig/node,triage/accepted

### What would you like to be added?

It would be better to improve the retry mechanism of EventedPLEG. EventedPLEG should be more tolerant of the outage of runtime such as restarting the runtime.




### Why is this needed?

I tried restarting containerd when EventedPLEG is enabeld. 
```
# time systemctl restart containerd

real	0m0.401s
user	0m0.008s
sys 	0m0.007s
```

Then, Evented PLEG fell back to GenericPLEG immediately.
```
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.152929     698 evented.go:206] "Evented PLEG: Failed to get container events, retrying: " err="rpc error: code = Unavailable desc = error reading from server: EOF"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.152970     698 log.go:32] "GetContainerEvents failed to get streaming client" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.152985     698 generic.go:242] "GenericPLEG: Relisting"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153033     698 log.go:32] "ListPodSandbox with filter from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\"" filter="nil"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153064     698 kuberuntime_sandbox.go:305] "Failed to list pod sandboxes" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153076     698 http2_client.go:993] "[transport] [client-transport 0xc000bc38c8] Closing: connection error: desc = \"error reading from server: EOF\"\n"
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153156     698 logging.go:39] "[core] [Channel #9 SubChannel #10]Subchannel Connectivity change to IDLE\n"
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153192     698 controlbuf.go:544] "[transport] [client-transport 0xc000bc38c8] loopyWriter exiting with error: connection error: desc = \"error reading from server: EOF\"\n"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153084     698 generic.go:256] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153280     698 evented.go:206] "Evented PLEG: Failed to get container events, retrying: " err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153327     698 log.go:32] "GetContainerEvents failed to get streaming client" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153344     698 generic.go:242] "GenericPLEG: Relisting"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153376     698 log.go:32] "ListPodSandbox with filter from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\"" filter="nil"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153407     698 kuberuntime_sandbox.go:305] "Failed to list pod sandboxes" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153422     698 generic.go:256] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153436     698 evented.go:206] "Evented PLEG: Failed to get container events, retrying: " err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153460     698 log.go:32] "GetContainerEvents failed to get streaming client" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153474     698 generic.go:242] "GenericPLEG: Relisting"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153513     698 log.go:32] "ListPodSandbox with filter from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\"" filter="nil"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153533     698 kuberuntime_sandbox.go:305] "Failed to list pod sandboxes" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153581     698 generic.go:256] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153589     698 evented.go:206] "Evented PLEG: Failed to get container events, retrying: " err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no su
ch file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153609     698 pickfirst.go:208] "[pick-first-lb] [pick-first-lb 0xc000b404e0] Received SubConn state update: 0xc000b40900, {ConnectivityState:IDLE ConnectionError:<nil>}\n"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153615     698 log.go:32] "GetContainerEvents failed to get streaming client" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153630     698 generic.go:242] "GenericPLEG: Relisting"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153655     698 log.go:32] "ListPodSandbox with filter from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\"" filter="nil"
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153680     698 kuberuntime_sandbox.go:305] "Failed to list pod sandboxes" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or direc
tory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: E0126 13:04:10.153695     698 generic.go:256] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153737     698 evented.go:206] "Evented PLEG: Failed to get container events, retrying: " err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: no such file or directory\""
Jan 26 13:04:10 kind-control-plane kubelet[698]: I0126 13:04:10.153766     698 evented.go:190] "Fall back to Generic PLEG relisting since Evented PLEG is not working"
```

EventedPLEG should keep working in this case because GenericPLEG also cannot work.