Repository: kubernetes
Issue: unavailable etcd leading to unexpected pod recreation
Labels: kind/support,sig/cluster-lifecycle,kind/feature,sig/architecture,needs-triage,sig/etcd

### What happened?

Recently, an incident in one of our clusters led to the deletion of approximately ~70% of the pods running on our worker nodes. While investigating what happened,  I found a series of etcd logs mentioning apply requests taking too long and a series of delete operations.

I tried to reproduce the scenario by simulating an issue with etcd (i. e. removing it from the kubelet reach inside the nodes) and this actually led to the recreation of unrelated pods (it's hard to find the sweet spot of pod deletion between a few pods and the cluster being totaly unreacheable) 

### What did you expect to happen?

The cluster shouldn't start deleting resources but instead "lock" itself to new operations, AFAIK.

### How can we reproduce it (as minimally and precisely as possible)?

Simply remove the etcd manifest from the /etc/kubernetes/manifests file for a relatively short time (~5mins)

### Anything else we need to know?

Some etcd pod logs I used to conduct the investigation

2024-10-21T21:30:02.5567686822 stderr F_{"level": "warn", "ts": "2024-10-21T21:30:02.556527Z","caller" : "etcdserver/util.go:170", "msg":"apply request took too long", "took":"109.535008ms", "expected-duration":"100ms","prefix":"","request": "header: <ID:11426074845274859338 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn: <compare: <target:MOD key:\"/registry/ pods/kube-system/cilium-knn9k\" mod_revision:551332032 > success:<request_put:<key:\"/registry/pods/kube-system/cilium-knn9k\" value_size:19461 >> failure:<>>","response":"size:22"} 

2024-10-21T21:30:02.657714974Z stderr F {"level": "info","ts": "2024-10-21T21:30:02.657359Z","caller" : "traceutil/trace.go:171", "msg":"trace[802572949] transaction", "detail":"{read_only:false; response_revision:603736321; number_of_response:1; }", "duration":"136.224602ms", "start": "2024-10-21T21:30:02.521053Z", "end":"2024-10-21T21:30:02.657278Z", "steps":["trace[80257 2949] 'process raft request' (duration: 134.140488ms)"], "step_count":1} 

2024-10-21T21:30:02.6582696652 stderr F {"level": "info", "ts": "2024-10-21T21:30:02.658004Z","caller" : "traceutil/trace.go:171", "msg":"trace[1759431508] linearizableReadLoop", "detail":"{readStateIndex:614487588; appliedIndex:614487486; }", "duration":"153.792181ms","start": "2024-10-21T21:30:02.5041922", "end": "2024-10-21T21:30:02.657984Z", "steps":["trace[1759431508] ' read index received" (duration: 36.17462ms)","trace [1759431508] 'applied index is now lower than readState. Index' (duration: 117.615976ms)"], "step_count":2} 

Start of generalized pod deletion (logs may be incomplete):

"msg":"apply request took too long", "took":"109.535008ms", "expected-duration":"100ms","prefix":"","request": "header: <ID:11426074845274859338 username sion:551332032 equest. :<key:\"/registry/pods/kube-system/cilium-knn9k\" value_size:19461 >> failure: <>>", "response":"size:22"}
1 trace.go:219] Trace [1248109794]: "Get" accept:application/vnd.kubernetes.protobuf, application/json, audit-id:1d03bc70-d072-43b8-97fc-3cbaf308b64e, client: 172.17.3.47, protocol:HTTP/2.0, resource:pods, scope: resource, url:/api/v1/namespaces/kube-system/pods/conntrack-adjuster-s5fkd, user-agent: kubelet/ v1.26.12 (linux/amd64) kubernetes/df63cd7, verb: GET (21-Oct-2024 21:30:02.734) (total time: 587ms): ration: 134.140488ms)"], "step_count":1} 

2024-10-21T21:30:03.329117818Z stderr F Trace[1248109794]: ---"About to write a response" 463ms (21:30:03.198) g":"trace[1759431508] linearizableReadLoop", "detail":"{readStateIndex:614487588; appliedIndex:614487486; }", "duration":"153.792181ms","start": "202 2024-10-21T21:30:03.329124819Z stderr F Trace[1248109794]: [587.053209ms] [587.053209ms] END "trace[1759431508] 'applied index is now lower than readState.Index' (duration: 117.615976ms)"], "step_count":2} 

2024-10-21T21:30:03.329130754Z stderr F 11021 21:30:03.321965 1 trace.go:219] Trace[1891213869]: "Delete" accept:application/vnd.kubernetes.protobuf, application/json, audit-id:20355a78-8d75-4e87-831d-07ba3156deab, client:172.17.3.47, protocol:HTTP/2.0, resource: pods, scope: resource, url:/api/v1/namespaces/apm/pods/apmXXXX, user-agent: kubelet/v1.26.12 (linux/amd64) kubernetes/df63cd7,verb: DELETE (21-Oct-2024 21:30:01.966) (total time: 1355ms):tion: 132.833274ms)"], "step_count":1} 

2024-10-21T21:30:03.329136245Z stderr F Trace [1891213869]: "Object deleted from database" 1024ms (21:30:03.198) msg":"trace[1031673685] transaction","detail":"{read_only: false; number_of_response:1; response_revision:603736317;}", "duration":"157.996656ms","star 2024-10-21T21:30:03.329140862Z stderr F Trace[1891213869]: ---"Writing http response done" 123ms (21:30:03.321) duration: 148.088092ms)"], "step_count":1} 

2024-10-21T21:30:03.329146014Z stderr F Trace[1891213869]: [1.355740157s] [1.355740157s] END ,"msg":"trace [1268363042] transaction", "detail":"{read_only:false; response_revision:603736318; number_of_response:1;}","duration":"157.490323ms", "st 2024-10-21T21:30:03.329151087Z stderr F 11021 21:30:03.322410 1 trace.go:219] Trace [1220059992]: "Delete" accept:*/*, audit-id:98ff45dc-7440-412e-b02d-411b772f4835, client: 127.0.0.1, protocol:HTTP/2.0, resource: pods, scope: resource, url:/api/v1/namespaces/XXXXXXXX/pods/XXXXXXXX, user-agent: node-fetch, verb: DELETE (21-Oct-2024 21:30:02.211) (total time: 1110ms): ,"msg":"trace[1611058272] transaction", "d '{read_only:false; response_revision:603736319; number_of_response:1; }", "duration":"156.926033ms"  2024-10-21T21:30:03.329156733Z stderr F Trace[1220059992]:---"Writing http response done" 123ms (21:30:03.322) 

Trace [1412908394]: "Delete" accept:*/*, audit-id:a9c4e508-08c0-46bd-9e7a-7a6231e2f813, client: 127.0.0.1, protocol:HTTP/2.0, resource: pods, scope: resource, url:/api/v1/namespaces/XXX-XXX-XXX/pods/XX-XX-XXX-XX-XXXXXXX, user-agent: node-fetch, verb: DELETE (21-Oct-2024 145981] sion:603736322;
2024-10-21T21:30:03.329171633Z stderr F Trace[1412908394]: ---"Writing http response done" 122ms (21:30:03.322) 2024-10-21T21:30:03.329175524Z stderr F Trace[1412908394]: [1.000495185s] [1.000495185s] END se:1; response_revision:603736314; }", "duration":"160.792537ms"," 


### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.28.13
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.2
WARNING: version difference between client (1.28) and server (1.30) exceeds the supported minor version skew of +/-1
```

</details>


### Cloud provider

<details>

ClusterAPI using Cluster API Provider Openstack

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux kmxdjklsd-control-plane-cdjmc 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

```

</details>


### Install tools

<details>

Kubeadm

</details>


### Container runtime (CRI) and version (if applicable)

<details>

Containerd v1.6.20 2806fc1057397dbaeefbea0e4e17bddfbd388f38

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

CNI: Cilium v1.15.3

</details>