Repository: kubernetes
Issue: Service account token gets invalidated while terminating pod is re-deleted
Labels: kind/bug,sig/auth,needs-triage

### What happened?

Attempting to kill a pod while it's already terminating causes its bound service account token to be invalidated.

This can happen, for example, when pod termination takes a long time, and an impatient user will re-attempt to delete it, this time with a short grace-period (e.g., via `kubectl delete pod --now`, which sets a grace-period=1s).



### What did you expect to happen?

I expect the service account token to remain valid throughout the entire lifetime of the pod, and for k8s API calls from within the pod to succeed (well, at least to authenticate successfully).

### How can we reproduce it (as minimally and precisely as possible)?

1. Create a role binding that allows the default service account to read pods:
```sh
kubectl create rolebinding --clusterrole view --serviceaccount default:default test
``` 
2. Create the following test pod.
The PreStop hook of the pod repeatedly runs `kubectl get pods`, and logs any error encountered:
```yaml
kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  terminationGracePeriodSeconds: 100000
  containers:
  - name: test
    image: bitnami/kubectl
    command:
    - "bin/sh"
    - "-c"
    - "sleep 100000000"
    lifecycle:
      preStop:
        exec:
          command:
          - "bin/sh"
          - "-c"
          - |
            echo "PreStop started at $(date +%T)" >> /tmp/prestop.log
            while true; do
              resp=$(kubectl get pods 2>&1)
              if [ $? -ne 0 ]; then
                echo "kubectl failed at $(date +%T)" >> /tmp/prestop.log
                echo $resp >> /tmp/prestop.log
              fi
              sleep 1
            done
EOF
```
3. Wait for the pod to become ready
```sh
kubectl get pod test --watch
```
4. Gracefully delete the pod, and immediately delete it again with gracePeriod=1 (== `--now`):
```sh
kubectl delete pod test --wait=false && kubectl delete pod test --wait=false --grace-period=1
```
5. Tail the logs of the PreStop hook:
```sh
kubectl exec -it test -- tail -f /tmp/prestop.log
```

The observed output is:
```
PreStop started at 07:14:25
kubectl failed at 07:15:30
error: You must be logged in to the server (Unauthorized)
kubectl failed at 07:15:31
error: You must be logged in to the server (Unauthorized)
kubectl failed at 07:15:32
error: You must be logged in to the server (Unauthorized)
kubectl failed at 07:15:33
error: You must be logged in to the server (Unauthorized)
```
6. Finally, cleanup by force deleting the pod.
(note - this only cleanups the pod, but the container remains dangling in the background):
```sh
kubectl delete pod test --force
```

### Anything else we need to know?

Based on several attempts to reproduce it, it seems that the token gets invalidated after about [60 seconds] + [grace-period seconds] + [a few seconds] - relative to when the pod was initially deleted.

Hence, different grace-period values yielded different token invalidation times:
- grace-period=1 --> invalidated after ~63s from initial pod deletion
- grace-period=30 --> ~95s
- grace-period=120 --> ~187s

On the other hand, if the 2nd delete command is issued at some later time (e.g., delete, sleep, delete --grace-period), then the token seems to be invalidated as soon as the 2nd deleted command is issued (+ [a few seconds]):
- sleep 120 + grace-period=1 --> ~127s
- sleep 120 + grace-period=30 --> ~126s

And yet on another hand (?!?!):
-sleep 120 + grace-period=120 --> ~3m
- sleep 90 + grace-period=300 --> ~6m

So, the invalidation timing seems to depend on multiple factors.
Hopefully this yields some useful clues.

### Kubernetes version

<details>

Tested on K8s v1.27.3(GKE), but also reproduced on v1.28.X (GKE + vanilla/kOps)
```console
$ kubectl version
Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3-gke.100
```

</details>


### Cloud provider

<details>
GKE, but also reproduced on vanilla/kOps.
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>