Repository: kubernetes
Issue: Readiness probe stops too early at eviction
Labels: kind/bug,priority/backlog,sig/node,triage/accepted

### What happened?

When kubelet evicts a pod, the ready condition doesn’t get `NotReady` during the pod termination even if a `readinessProbe` is configured.

### What did you expect to happen?

A readiness probe works during a pod termination so that the pod gets `NotReady` as early as possible.


### How can we reproduce it (as minimally and precisely as possible)?

Use this `readiness.yaml`:
```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: readiness-script-configmap
data:
  readiness-script.sh: |
    #!/bin/sh
    handler() {
      rm /tmp/ready
      sleep 20
    }
    touch /tmp/ready
    trap handler SIGTERM
    while true; do
      sleep 1
    done
---
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: readiness-container
  name: readiness-test
spec:
  containers:
  - command:
    - sh
    - /script/readiness-script.sh
    image: busybox
    name: readiness
    readinessProbe:
      exec:
        command:
        - cat
        - /tmp/ready
      initialDelaySeconds: 3
      periodSeconds: 3
    volumeMounts:
    - name: readiness-script
      mountPath: /script
  - command:
    - sh
    - -c
    - for i in `seq 100`; do dd if=/dev/random of=file${i} bs=1048576 count=1 2>/dev/null; sleep .1; done; while true; do sleep 5; done
    name: disk-consumer
    image: busybox
    resources:
      limits:
        ephemeral-storage: "100Mi"
      requests:
        ephemeral-storage: "100Mi"
  volumes:
  - name: readiness-script
    configMap:
      name: readiness-script-configmap
```

Create resources and wait an eviction happens:
```
$ kubectl create -f readiness.yaml; kubectl get pods readiness-test -w
configmap/readiness-script-configmap created
pod/readiness-test created
NAME         	READY   STATUS          	RESTARTS   AGE
readiness-test   0/2 	ContainerCreating   0      	0s
readiness-test   1/2 	Running         	0      	3s
readiness-test   2/2 	Running         	0      	7s
readiness-test   0/2 	Error           	0      	46s
readiness-test   0/2 	Error           	0      	47s

```

When deleting this pod, the readiness probe works during termination:
```
$ kubectl create -f readiness.yaml; (sleep 15; kubectl delete pod readiness-test) & kubectl get pods -w
configmap/readiness-script-configmap created
pod/readiness-test created
[1] 137999
NAME             READY   STATUS              RESTARTS   AGE
readiness-test   0/2     ContainerCreating   0          0s
readiness-test   1/2     Running             0          2s
readiness-test   2/2     Running             0          6s
pod "readiness-test" deleted
readiness-test   2/2     Terminating         0          15s
readiness-test   1/2     Terminating         0          21s
readiness-test   0/2     Terminating         0          45s
readiness-test   0/2     Terminating         0          45s
readiness-test   0/2     Terminating         0          45s
readiness-test   0/2     Terminating         0          45s

```


### Anything else we need to know?

I guess this issue is caused as follows:

At eviction, a pod phase is set to `PodFailed` internally in `podStatusFn` before stopping containers in the pod:
https://github.com/kubernetes/kubernetes/blob/0d8f996aa9a1667d5994f10e03de5be21bf205e0/pkg/kubelet/kubelet.go#L2026-L2029
https://github.com/kubernetes/kubernetes/blob/0d8f996aa9a1667d5994f10e03de5be21bf205e0/pkg/kubelet/eviction/eviction_manager.go#L605-L608

Because the internal pod phase is `PodFailed`, the probe worker finishes working without probing containers at termination:
https://github.com/kubernetes/kubernetes/blob/dd68c5f2409fec7176e6172d6f9d97bd6447c4da/pkg/kubelet/prober/worker.go#L203-L216


### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2
</details>


### Cloud provider

<details>
None.
I reproduced on my laptop.
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>
kind
</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>