Repository: kubernetes
Issue: GracefulNodeShutdown not evacuating all pods
Labels: kind/bug,priority/backlog,sig/node,priority/important-longterm,triage/accepted

### What happened?

Node is being shutdown, then according to logs all pods are being terminated, howewer, some remain scheduled/running on the node.

### What did you expect to happen?

Expected that all pods being evacuated/rescheduled to another running node.

### How can we reproduce it (as minimally and precisely as possible)?

```
# kubectl create ns shutdown-test
# kubectl -n shutdown-test create deploy pause --image k8s.gcr.io/pause --replicas=20
```

After it, ensure that all pods are scheduled to a specific node.

i.e.:

```
# kubectl -n shutdown-test get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE                      NOMINATED NODE   READINESS GATES
pause-655c56c859-4tn6d   1/1     Running   0          54s   10.112.10.209   node-4.................   <none>           <none>
pause-655c56c859-5dwd5   1/1     Running   0          54s   10.112.10.213   node-4.................   <none>           <none>
pause-655c56c859-8d5vf   1/1     Running   0          54s   10.112.10.202   node-4.................   <none>           <none>
pause-655c56c859-8p4hj   1/1     Running   0          55s   10.112.10.194   node-4.................   <none>           <none>
pause-655c56c859-9vsq5   1/1     Running   0          55s   10.112.10.195   node-4.................   <none>           <none>
pause-655c56c859-bplfn   1/1     Running   0          54s   10.112.10.207   node-4.................   <none>           <none>
pause-655c56c859-bv4m2   1/1     Running   0          54s   10.112.10.201   node-4.................   <none>           <none>
pause-655c56c859-bz7xp   1/1     Running   0          54s   10.112.10.198   node-4.................   <none>           <none>
pause-655c56c859-dztww   1/1     Running   0          54s   10.112.10.210   node-4.................   <none>           <none>
pause-655c56c859-f6wdz   1/1     Running   0          54s   10.112.10.211   node-4.................   <none>           <none>
pause-655c56c859-h78zl   1/1     Running   0          54s   10.112.10.197   node-4.................   <none>           <none>
pause-655c56c859-hkczs   1/1     Running   0          54s   10.112.10.206   node-4.................   <none>           <none>
pause-655c56c859-jp5tc   1/1     Running   0          54s   10.112.10.204   node-4.................   <none>           <none>
pause-655c56c859-jqcxx   1/1     Running   0          54s   10.112.10.208   node-4.................   <none>           <none>
pause-655c56c859-l9lrh   1/1     Running   0          54s   10.112.10.205   node-4.................   <none>           <none>
pause-655c56c859-ql492   1/1     Running   0          54s   10.112.10.199   node-4.................   <none>           <none>
pause-655c56c859-qmwxx   1/1     Running   0          54s   10.112.10.212   node-4.................   <none>           <none>
pause-655c56c859-rs8jd   1/1     Running   0          54s   10.112.10.196   node-4.................   <none>           <none>
pause-655c56c859-rvdvb   1/1     Running   0          54s   10.112.10.203   node-4.................   <none>           <none>
pause-655c56c859-xdtv6   1/1     Running   0          54s   10.112.10.200   node-4.................   <none>           <none>
```

Then shutdown node gracefully:

```
root@node-4:~# systemctl reboot
```

After node is rebooted, nothing is changed:

```
# kubectl -n shutdown-test get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP              NODE                      NOMINATED NODE   READINESS GATES
pause-655c56c859-4tn6d   1/1     Running   1          4m14s   10.112.10.227   node-4.................   <none>           <none>
pause-655c56c859-5dwd5   1/1     Running   1          4m14s   10.112.10.225   node-4.................   <none>           <none>
pause-655c56c859-8d5vf   1/1     Running   1          4m14s   10.112.10.222   node-4.................   <none>           <none>
pause-655c56c859-8p4hj   1/1     Running   1          4m15s   10.112.10.223   node-4.................   <none>           <none>
pause-655c56c859-9vsq5   1/1     Running   1          4m15s   10.112.10.226   node-4.................   <none>           <none>
pause-655c56c859-bplfn   1/1     Running   1          4m14s   10.112.10.215   node-4.................   <none>           <none>
pause-655c56c859-bv4m2   1/1     Running   1          4m14s   10.112.10.231   node-4.................   <none>           <none>
pause-655c56c859-bz7xp   1/1     Running   1          4m14s   10.112.10.214   node-4.................   <none>           <none>
pause-655c56c859-dztww   1/1     Running   1          4m14s   10.112.10.221   node-4.................   <none>           <none>
pause-655c56c859-f6wdz   1/1     Running   1          4m14s   10.112.10.217   node-4.................   <none>           <none>
pause-655c56c859-h78zl   1/1     Running   1          4m14s   10.112.10.230   node-4.................   <none>           <none>
pause-655c56c859-hkczs   1/1     Running   1          4m14s   10.112.10.233   node-4.................   <none>           <none>
pause-655c56c859-jp5tc   1/1     Running   1          4m14s   10.112.10.232   node-4.................   <none>           <none>
pause-655c56c859-jqcxx   1/1     Running   1          4m14s   10.112.10.216   node-4.................   <none>           <none>
pause-655c56c859-l9lrh   1/1     Running   1          4m14s   10.112.10.218   node-4.................   <none>           <none>
pause-655c56c859-ql492   1/1     Running   1          4m14s   10.112.10.228   node-4.................   <none>           <none>
pause-655c56c859-qmwxx   1/1     Running   1          4m14s   10.112.10.219   node-4.................   <none>           <none>
pause-655c56c859-rs8jd   1/1     Running   1          4m14s   10.112.10.224   node-4.................   <none>           <none>
pause-655c56c859-rvdvb   1/1     Running   1          4m14s   10.112.10.220   node-4.................   <none>           <none>
pause-655c56c859-xdtv6   1/1     Running   1          4m14s   10.112.10.229   node-4.................   <none>           <none>
```

Sometimes a few pods are being evacuated as expected, but usually not all. This time none was.


### Anything else we need to know?

GracefulNodeShutdown is enabled in kubelet:

```
# grep Grace /var/lib/kubelet/config.yaml 
  GracefulNodeShutdown: true
shutdownGracePeriod: 45s
shutdownGracePeriodCriticalPods: 30s
```

[kubelet.log](https://github.com/kubernetes/kubernetes/files/9564927/kubelet.log)


### Kubernetes version

<details>

```console
$ kubectl version
# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:15:38Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

```

</details>


### Cloud provider

<details>
n/a
</details>


### OS version

<details>

```console
# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
# uname -a
Linux node-4 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64 GNU/Linux
```

</details>


### Install tools

<details>
kubeadm
</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>