Repository: kubernetes
Issue: NoExecute taint is added with extra 5s delay when a Node's ready condition becomes Unknown
Labels: kind/bug,sig/node,triage/accepted

### What happened?

When investigating a delay of Pod eviction on unreachable Node, I found it always took 5s longer than the desired duration.
Looking at the timestamps, I found an odd one: the timestamp when `node.kubernetes.io/unreachable` NoExecute taint was added.

```
# Node taints:
taints:
- effect: NoSchedule
  key: node.kubernetes.io/unreachable
  timeAdded: "2023-09-21T06:32:41Z"
- effect: NoExecute
  key: node.kubernetes.io/unreachable
  timeAdded: "2023-09-21T06:32:46Z"    # This one is 5s later than all other timestamps

# Node conditions:
conditions:
- lastHeartbeatTime: "2023-09-21T06:31:55Z"
  lastTransitionTime: "2023-09-21T06:32:41Z"
  message: Kubelet stopped posting node status.
  reason: NodeStatusUnknown
  status: Unknown
  type: MemoryPressure
- lastHeartbeatTime: "2023-09-21T06:31:55Z"
  lastTransitionTime: "2023-09-21T06:32:41Z"
  message: Kubelet stopped posting node status.
  reason: NodeStatusUnknown
  status: Unknown
  type: DiskPressure
- lastHeartbeatTime: "2023-09-21T06:31:55Z"
  lastTransitionTime: "2023-09-21T06:32:41Z"
  message: Kubelet stopped posting node status.
  reason: NodeStatusUnknown
  status: Unknown
  type: PIDPressure
- lastHeartbeatTime: "2023-09-21T06:31:55Z"
  lastTransitionTime: "2023-09-21T06:32:41Z"
  message: Kubelet stopped posting node status.
  reason: NodeStatusUnknown
  status: Unknown
  type: Ready

# Pod conditions:
- lastProbeTime: null
  lastTransitionTime: "2023-09-21T06:32:41Z"
  status: "False"
  type: Ready
```

This can be reproducd consistently by terminating kubelet on a Node. The 5s delay of tainting adds 5s delay to the recovery time of Pods which are supposed to be evicted by the taint and recreated on other Nodes sooner. And I can't think of a reason why this taint alone should be added 5s later after the other taint and all other status changes. After investigating the relevant code for how the 5s is introduced, I think it should be an unwanted effect:

`processTaintBaseEviction()` uses the last observed ready condition of the Node instead of the current one to determine whether it should add the Node to the taint queue. When a Node is set to unreachable due to missing heartbeats, the last observed ready condition is still `true` and the current ready condition is `unknown`, we should use the latter for `processTaintBaseEviction()`. Otherwise the taint can only be added when `monitorNodeHealth()` runs another time (5s later) during which the current ready condition becomes the last observed ready condition, like the following logs show:
https://github.com/kubernetes/kubernetes/blob/cde6a46a483a0cc7d3f2404db506e1c24ecd4e9d/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L741

```
# First round handling when the Node is detected as unreachable.
I0921 13:38:27.930549       1 node_lifecycle_controller.go:954] "Node hasn't been updated" node="kind-worker2" duration="40.006646215s" nodeConditionType=Ready currentCondition="&NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-09-21 13:35:27 +0000 UTC,LastTransitionTime:2023-09-21 13:35:27 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,}"
I0921 13:38:27.930777       1 node_lifecycle_controller.go:954] "Node hasn't been updated" node="kind-worker2" duration="40.006732434s" nodeConditionType=MemoryPressure currentCondition="&NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-09-21 13:35:27 +0000 UTC,LastTransitionTime:2023-09-21 13:35:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,}"
I0921 13:38:27.930881       1 node_lifecycle_controller.go:954] "Node hasn't been updated" node="kind-worker2" duration="40.006958941s" nodeConditionType=DiskPressure currentCondition="&NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-09-21 13:35:27 +0000 UTC,LastTransitionTime:2023-09-21 13:35:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,}"
I0921 13:38:27.930931       1 node_lifecycle_controller.go:954] "Node hasn't been updated" node="kind-worker2" duration="40.007072433s" nodeConditionType=PIDPressure currentCondition="&NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-09-21 13:35:27 +0000 UTC,LastTransitionTime:2023-09-21 13:35:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,}"
I0921 13:38:27.940640       1 controller_utils.go:186] "Recording status change event message for node" status="NodeNotReady" node="kind-worker2"
I0921 13:38:27.940700       1 controller_utils.go:123] "Update ready status of pods on node" node="kind-worker2"
I0921 13:38:27.940803       1 controller_utils.go:144] "Updating ready status of pod to false" pod="kube-system/antrea-agent-sw57p"
I0921 13:38:27.941226       1 event.go:307] "Event occurred" object="kind-worker2" fieldPath="" kind="Node" apiVersion="v1" type="Normal" reason="NodeNotReady" message="Node kind-worker2 status is now: NodeNotReady"
I0921 13:38:27.952517       1 controller_utils.go:144] "Updating ready status of pod to false" pod="kube-system/kube-proxy-ktsph"
I0921 13:38:27.952879       1 event.go:307] "Event occurred" object="kube-system/antrea-agent-sw57p" fieldPath="" kind="Pod" apiVersion="v1" type="Warning" reason="NodeNotReady" message="Node is not ready"
I0921 13:38:27.961405       1 event.go:307] "Event occurred" object="kube-system/kube-proxy-ktsph" fieldPath="" kind="Pod" apiVersion="v1" type="Warning" reason="NodeNotReady" message="Node is not ready"
# Second round handling after the Node is detected as unreachable.
I0921 13:38:32.962342       1 node_lifecycle_controller.go:954] "Node hasn't been updated" node="kind-worker2" duration="45.038461632s" nodeConditionType=Ready currentCondition="&NodeCondition{Type:Ready,Status:Unknown,LastHeartbeatTime:2023-09-21 13:35:27 +0000 UTC,LastTransitionTime:2023-09-21 13:38:27 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,}"
I0921 13:38:32.962379       1 node_lifecycle_controller.go:954] "Node hasn't been updated" node="kind-worker2" duration="45.038523697s" nodeConditionType=MemoryPressure currentCondition="&NodeCondition{Type:MemoryPressure,Status:Unknown,LastHeartbeatTime:2023-09-21 13:35:27 +0000 UTC,LastTransitionTime:2023-09-21 13:38:27 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,}"
I0921 13:38:32.962408       1 node_lifecycle_controller.go:954] "Node hasn't been updated" node="kind-worker2" duration="45.038548769s" nodeConditionType=DiskPressure currentCondition="&NodeCondition{Type:DiskPressure,Status:Unknown,LastHeartbeatTime:2023-09-21 13:35:27 +0000 UTC,LastTransitionTime:2023-09-21 13:38:27 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,}"
I0921 13:38:32.962431       1 node_lifecycle_controller.go:954] "Node hasn't been updated" node="kind-worker2" duration="45.03857838s" nodeConditionType=PIDPressure currentCondition="&NodeCondition{Type:PIDPressure,Status:Unknown,LastHeartbeatTime:2023-09-21 13:35:27 +0000 UTC,LastTransitionTime:2023-09-21 13:38:27 +0000 UTC,Reason:NodeStatusUnknown,Message:Kubelet stopped posting node status.,}"
# The node is added to the taint queue only in the second round.
I0921 13:38:32.962509       1 node_lifecycle_controller.go:793] "Node is unresponsive. Adding it to the Taint queue" node="kind-worker2" timeStamp="2023-09-21 13:38:32.962484274 +0000 UTC m=+71.367258065"
```

I will create a PR to fix it if the above analysis makes sense.

### What did you expect to happen?

`node.kubernetes.io/unreachable` NoExecute taint shouldn't be added with extra 5s delay, compared with when `node.kubernetes.io/unreachable` NoSchedule taint is added and the Node is detected as unhealthy.

### How can we reproduce it (as minimally and precisely as possible)?

1. Stop kubelet on one Node.
2. Check the timestamps of the taints and condition transition of the Node.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>