Repository: kubernetes
Issue: Graceful Node Shutdown does not update endpoints for terminating pods
Labels: kind/bug,sig/node,priority/important-longterm,triage/accepted

### What happened?

When graceful node shutdown is enabled, I expect that services are drained appropriately during pod shutdown (i.e. the same way that `kubectl delete pod` would work). However this doesn't seem to be the case - the pod never enters the `Terminating` state according to `kubectl` and doesn't get removed from the service endpoints until it enters the `Completed` or `Failed` state. This causes connection errors and lost requests in `Cluster`-routed services.

### What did you expect to happen?

After pod termination starts, the EndpointSlice should be updated to have the pod that is terminating removed or marked terminating, even before the pod is fully shut down. This is what happens when a pod gets removed for any other reason (e.g. `kubectl delete pod`).

### How can we reproduce it (as minimally and precisely as possible)?

You can see the repro at https://github.com/mikekap/vagrant-kubeadm-kubernetes/tree/graceful-shutdown-repro . The repro sets up a 2 node cluster with graceful shutdown enabled and a simple deployment/service.

<details>
<summary>Here is the deployment/service yaml pasted again</summary>

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-deployment
spec:
  selector:
    matchLabels:
      app: echo
  replicas: 2
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
      - name: echo
        image: mendhak/http-https-echo
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "echo 'Begin shutdown' > /proc/1/fd/1; sleep 20; echo 'End shutdown' > /proc/1/fd/1; exit 0"]
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: echo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
```
</details>

To reproduce:
 * Run the deployment, preferably one pod on each node.
 * Run `watch -n 1 'kubectl  describe endpointslice my-service-5j2nw'` in a separate terminal (you'll need to figure out the right name unfortunately)
 * Use the virtualbox UI to send an ACPI shutdown to one of the nodes.

Within 20 seconds (the preStop hook duration), the EndpointSlice should be updated to have the pod that is terminating removed. This is what happens when a pod gets removed for any other reason (e.g. `kubectl delete pod`).

To ensure that everything else worked, you can reboot the node and see the journal logs (via something like `journalctl -b -1`) of the shutdown sequence triggering the kubelet graceful shutdown logic.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
None
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux master-node 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>
kubeadm
</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>