Repository: kubernetes
Issue: A pod with a still running postStart lifecycle hook that is deleted is not terminated even after terminationGracePeriod
Labels: kind/bug,priority/backlog,sig/node,lifecycle/rotten,triage/accepted

### What happened?

We introduced a postStart hook to one of our pods. Eventually, we noticed these pods hanging in "Terminating" state indefinitely when we try to delete them.
Trying `kubectl logs`yielded 
`Error from server (BadRequest): container "reproducer" in pod "reproducer" is waiting to start: ContainerCreating`

That was the first hint that the postStart hook was the problem (since a pod is considered PodInitializing until its postStart hook exits).

We can exec into the container just fine, there we found that indeed our postStart hook was stuck. Killing all processes of the postStart hook made the pod terminate.
Looking at the logs of our PID 1 process (luckily, we also write that to a file...) showed no sign of it receiving a SIGTERM (we have a shutdown hook in that application that would log if it were invoked).

From our observations, we conclude that while the pod is still not done with the postStart hook, K8s does not send a SIGTERM to the container's PID 1 process. To make matters worse, even after the terminationGracePeriod expired, no kill happens, leading to the observed behaviour.

### What did you expect to happen?

The pod terminates - if not with a SIGTERM to the PID 1 process, then at least with a kill after the terminationGracePeriod expired.

### How can we reproduce it (as minimally and precisely as possible)?

Apply the following pod spec:
```
apiVersion: v1
kind: Pod
metadata:
  name: reproducer
spec:
  containers:
    - command:
        - /bin/bash
        - -c
        - |
          trap "echo SIGTERM received, setting exit file | tee -a /log.txt; touch /exit" SIGTERM;
          while [ ! -f "/exit" ]; do
            echo "$(date) Faking work" | tee -a /log.txt;
            sleep 10;
          done

      image: ubuntu
      imagePullPolicy: IfNotPresent
      lifecycle:
        postStart:
          exec:
            command:
              - /bin/bash
              - -c
              - | 
                while [ ! -f "/postStartDone" ]; do 
                  sleep 5;  
                done; 
                echo "poststart hook done" | tee -a /log.txt

      name: reproducer
  terminationGracePeriodSeconds: 30
```


(Note that we explicitly set the terminationGracePeriodSeconds to the default 30 for clarity.)
In one terminal, run `watch -d kubectl get pods`.
After applying, the pod will show up, but since the postStart hook will not exit, the pod will stay in ContainerCreating.
Confirm that you cannot see the logs with `kubectl logs reproducer`  (error message `Error from server (BadRequest): container "reproducer" in pod "reproducer" is waiting to start: ContainerCreating`)

Exec into the pod with `kubectl exec -it reproducer -- /bin/bash`
Confirm that both the root process and the post start hook are running with `ps aux`:
```
root@reproducer:/# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   4360  3320 ?        Ss   09:55   0:00 /bin/bash -c trap "echo SIGTERM received, setting exit file | tee -a /log.txt; touch /exit" SIGTERM; while [ ! -f "/exit"
root          11  0.0  0.0   4360  3364 ?        Ss   09:55   0:00 /bin/bash -c while [ ! -f "/postStartDone" ]; do    sleep 5;   done;  echo "poststart hook done" | tee -a /log.txt
root          19  0.0  0.0   4628  3796 pts/0    Ss   09:55   0:00 /bin/bash
root          31  0.0  0.0   2788  1096 ?        S    09:55   0:00 sleep 10
root          32  0.0  0.0   2788  1108 ?        S    09:55   0:00 sleep 5
root          33  0.0  0.0   7060  1592 pts/0    R+   09:55   0:00 ps aux
```

Exit.
Delete the pod with  `kubectl delete reproducer`.
Observe that the pod goes from Container Creating to Terminating and stays in that state indefinitely (and stays that way indefinitely, even after the termination grace period of 30secs).
Exec into the pod again, run `tail -f log.txt`, observe the log output... you should only see lines `Faking work`.
In another shell, create the file that will make the postStart hook exit successfully with
`kubectl exec reproducer -- /bin/touch /postStartDone` 
`poststart hook done`will be logged and eventually, the pod will terminate. 




### Anything else we need to know?

_No response_

### Kubernetes version

Originally observed on EKS v1.24.8-eks-ffeb93d, reproduced on AKS v1.23.8 and v1.25.2, Minikube v1.24.10 and v1.25.6  

### Cloud provider

See above AWS ( EKS v1.24.8-eks-ffeb93d) and Azure AKS v1.23.8 and v1.25.2

### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>