Repository: kubernetes
Issue: The kube-apiserver (with 3 etcd endpints by --etcd-servers)still connect the unhealthy etcd member when we shut down one master node(which has one etcd static pods)
Labels: kind/bug,needs-sig,needs-triage

### What happened?

1. we use the kube-apiserver connect to etcd by three members as :
```
"etcd-servers": [
      "https://10.255.69.14:2379",
      "https://10.255.69.15:2379",
      "https://10.255.69.16:2379",
      "https://localhost:2379"
    ],
```

2. Shut down the master3 node(10.255.69.16), which has both etcd and apiserver static pods


3. Found the master1(10.255.69.14) apiserver and master2(10.255.69.15) apiserver still connect to the unhealthy 10.255.69.16 etcd endpoint

### What did you expect to happen?

When we shutdown one etcd member, the kube-apiserver should quickly switch to the healthy etcd endpoint

### How can we reproduce it (as minimally and precisely as possible)?

1 set the apiserver connect etcd by --etcd-servers ,which has 3 members
2 shut down the master(not reboot)

### Anything else we need to know?

- kube-apiserver logs show as follows:
```
apiserver 
2024-11-15T16:08:24.340356305+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:24.340Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc01402e000/10.255.69.14:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2024-11-15T16:08:24.340460558+08:00 stderr F I1115 08:08:24.340378      20 healthz.go:257] etcd check failed: readyz
2024-11-15T16:08:24.340460558+08:00 stderr F [-]etcd failed: error getting data from etcd: context deadline exceeded
2024-11-15T16:08:24.340541843+08:00 stderr F E1115 08:08:24.340479      20 timeout.go:141] post-timeout activity - time-elapsed: 1.001696115s, GET "/readyz" result: <nil>
2024-11-15T16:08:27.369028487+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:27.368Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00f856000/10.255.69.14:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2024-11-15T16:08:27.369028487+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:27.368Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00f856000/10.255.69.14:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2024-11-15T16:08:28.614097370+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:28.613Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00
20b5880/10.255.69.14:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2024-11-15T16:08:28.614097370+08:00 stderr F I1115 08:08:28.613845      20 trace.go:205] Trace[523486837]: "GuaranteedUpdate etcd3" audit-id:dbab0d30-2e4f-4604-b367-c84f000f1f86,key:/configmaps/kube-system/kube-controller-manag
er,type:*core.ConfigMap (15-Nov-2024 08:08:23.614) (total time: 4999ms):
2024-11-15T16:08:28.614097370+08:00 stderr F Trace[523486837]: ---"Txn call finished" err:context deadline exceeded 4998ms (08:08:28.613)
2024-11-15T16:08:28.614097370+08:00 stderr F Trace[523486837]: [4.999504982s] [4.999504982s] END
2024-11-15T16:08:28.616421774+08:00 stderr F I1115 08:08:28.616312      20 trace.go:205] Trace[1123415286]: "Update" url:/api/v1/namespaces/kube-system/configmaps/kube-controller-manager,user-agent:kube-controller-manager/v1.25.8 (linux/amd64) kubernetes/594da2b/leader-election,audit-id:dbab0d30-2e4f-4604-b367-c84f000f1f86,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-Nov-2024 08:08:23.614) (total time: 5002ms):
2024-11-15T16:08:28.616421774+08:00 stderr F Trace[1123415286]: ---"Write to database call finished" len:535,err:Timeout: request did not complete within requested timeout - context deadline exceeded 4999ms (08:08:28.613)      
2024-11-15T16:08:28.616421774+08:00 stderr F Trace[1123415286]: [5.002050748s] [5.002050748s] END
2024-11-15T16:08:28.616890678+08:00 stderr F E1115 08:08:28.616631      20 timeout.go:141] post-timeout activity - time-elapsed: 2.794599ms, PUT "/api/v1/namespaces/kube-system/configmaps/kube-controller-manager" result: <nil> 
2024-11-15T16:08:30.058932458+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:30.058Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc010bd0e00/10.255.69.14:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}

2024-11-15T16:08:32.011392068+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:32.011Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00bd06fc0/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:50894->10.255.69.16:2379: read: connection timed out"}
2024-11-15T16:08:32.012062393+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:32.011Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0a4efcc40/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:44404->10.255.69.16:2379: read: connection timed out"}
2024-11-15T16:08:32.012211954+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:32.011Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc010bd0e00/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed out"}
2024-11-15T16:08:32.012211954+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:32.012Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc010bd0e00/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed out"}
2024-11-15T16:08:32.012226796+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:32.012Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc010bd0e00/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed out"}
2024-11-15T16:08:32.012226796+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:32.012Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc010bd0e00/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed out"}
2024-11-15T16:08:32.012226796+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:32.012Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc010bd0e00/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed out"}
2024-11-15T16:08:32.012226796+08:00 stderr F I1115 08:08:32.012185      20 trace.go:205] Trace[1958868955]: "GuaranteedUpdate etcd3" audit-id:28031557-60b2-4f4f-ad50-aa00b6244ed5,key:/leases/gtm/cell.gtm.io,type:*coordination.Lease (15-Nov-2024 08:08:25.595) (total time: 6416ms):

2024-11-15T16:08:32.012513501+08:00 stderr F Trace[891349870]: ---"Txn call finished" err:rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed out 9758ms (08:08:32.012)
2024-11-15T16:08:32.012513501+08:00 stderr F Trace[891349870]: [9.75949209s] [9.75949209s] END
2024-11-15T16:08:32.012513501+08:00 stderr F E1115 08:08:32.012404      20 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc0bc1a7f98)}: rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed out

2024-11-15T16:08:32.012736479+08:00 stderr F I1115 08:08:32.012514      20 trace.go:205] Trace[859168237]: "GuaranteedUpdate etcd3" audit-id:71717ed4-0c00-458a-8897-0d9df2388bfb,key:/leases/machine-config-operator/machine-config,type:*coordination.Lease (15-Nov-2024 08:08:23.523) (total time: 8489ms):
2024-11-15T16:08:32.012736479+08:00 stderr F Trace[859168237]: ---"Txn call finished" err:rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed ou
t 8488ms (08:08:32.012)
2024-11-15T16:08:32.012736479+08:00 stderr F Trace[859168237]: [8.48927931s] [8.48927931s] END

0bd0e00/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed out"}
2024-11-15T16:08:32.013502285+08:00 stderr F I1115 08:08:32.012231      20 trace.go:205] Trace[2108509803]: "GuaranteedUpdate etcd3" audit-id:3635e940-38e3-41ca-987a-926cc584384a,key:/leases/envoy-gateway-system/5b9825d2.gateway.envoyproxy.io,type:*coordination.Lease (15-Nov-2024 08:08:30.258) (total time: 1753ms):
2024-11-15T16:08:32.013502285+08:00 stderr F Trace[2108509803]: ---"Txn call finished" err:rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:45274->10.255.69.16:2379: read: connection timed out 1752ms (08:08:32.012)
2024-11-15T16:08:32.013502285+08:00 stderr F Trace[2108509803]: [1.753630806s] [1.753630806s] END

2024-11-15T16:08:34.699497756+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:34.699Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00f16d880/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:50918->10.255.69.16:2379: read: connection timed out"}
2024-11-15T16:08:34.699497756+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:34.699Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc007d40000/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = error reading from server: read tcp 10.255.69.15:47940->10.255.69.16:2379: read: connection timed out"}

2024-11-15T16:08:35.070903277+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:35.070Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0a2a90700/10.255.69.14:2379","attempt":0,"error":"rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout"}
2024-11-15T16:08:35.082401449+08:00 stderr F W1115 08:08:35.082223      20 logging.go:59] [core] [Channel #254 SubChannel #257] grpc: addrConn.createTransport failed to connect to {
2024-11-15T16:08:35.082401449+08:00 stderr F   "Addr": "10.255.69.16:2379",
2024-11-15T16:08:35.082401449+08:00 stderr F   "ServerName": "10.255.69.16",
2024-11-15T16:08:35.082401449+08:00 stderr F   "Attributes": null,
2024-11-15T16:08:35.082401449+08:00 stderr F   "BalancerAttributes": null,
2024-11-15T16:08:35.082401449+08:00 stderr F   "Type": 0,
2024-11-15T16:08:35.082401449+08:00 stderr F   "Metadata": null
2024-11-15T16:08:35.082401449+08:00 stderr F }. Err: connection error: desc = "transport: Error while dialing dial tcp 10.255.69.16:2379: connect: connection timed out"
2024-11-15T16:08:35.082401449+08:00 stderr F W1115 08:08:35.082242      20 logging.go:59] [core] [Channel #977 SubChannel #980] grpc: addrConn.createTransport failed to connect to {
2024-11-15T16:08:35.082401449+08:00 stderr F   "Addr": "10.255.69.16:2379",
2024-11-15T16:08:35.082401449+08:00 stderr F   "ServerName": "10.255.69.16",
2024-11-15T16:08:35.082401449+08:00 stderr F   "Attributes": null,
2024-11-15T16:08:35.082401449+08:00 stderr F   "BalancerAttributes": null,
2024-11-15T16:08:35.082401449+08:00 stderr F   "Type": 0,
2024-11-15T16:08:35.082401449+08:00 stderr F   "Metadata": null
2024-11-15T16:08:35.082401449+08:00 stderr F }. Err: connection error: desc = "transport: Error while dialing dial tcp 10.255.69.16:2379: connect: connection timed out"

```

- The etcd logs show as follows:
```
etcd 

2024-11-15T16:00:03.089125997+08:00 stderr F {"level":"info","ts":"2024-11-15T08:00:03.088971Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/etcd/member/wal/00000000000014a6-0000000005fdfcae.wal"}
2024-11-15T16:01:04.261555789+08:00 stderr F {"level":"info","ts":"2024-11-15T08:01:04.261443Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":91749871}
2024-11-15T16:01:05.254012681+08:00 stderr F {"level":"info","ts":"2024-11-15T08:01:05.253911Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":91749871,"took":"966.045196ms","hash":576483144}
2024-11-15T16:01:05.254012681+08:00 stderr F {"level":"info","ts":"2024-11-15T08:01:05.253986Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":576483144,"revision":91749871,"compact-revision":91733961}
2024-11-15T16:04:09.827516612+08:00 stderr F {"level":"info","ts":"2024-11-15T08:04:09.827399Z","caller":"wal/wal.go:785","msg":"created a new WAL segment","path":"/var/lib/etcd/member/wal/00000000000014ac-0000000005ff8643.wal"
}
2024-11-15T16:04:33.101219829+08:00 stderr F {"level":"info","ts":"2024-11-15T08:04:33.101107Z","caller":"fileutil/purge.go:85","msg":"purged","path":"/var/lib/etcd/member/wal/00000000000014a7-0000000005fe3968.wal"}
2024-11-15T16:06:04.278242322+08:00 stderr F {"level":"info","ts":"2024-11-15T08:06:04.278114Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":91765934}
2024-11-15T16:06:05.109317576+08:00 stderr F {"level":"info","ts":"2024-11-15T08:06:05.109193Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":91765934,"took":"803.717455ms","h
ash":163059895}
2024-11-15T16:06:05.109317576+08:00 stderr F {"level":"info","ts":"2024-11-15T08:06:05.109259Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":163059895,"revision":91765934,"compact-revision":91749871}
2024-11-15T16:08:15.432491578+08:00 stderr F 2024/11/15 08:08:15 WARNING: [core] [Server #9] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2024-11-15T16:08:28.472483552+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:28.472346Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f7585d3c6edd2214","rtt":"8.629741ms","error":"dial tcp 10.255.69.16:2380: i/o timeout"}
2024-11-15T16:08:33.472610748+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:33.472502Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f7585d3c6edd2214","rtt":"8.629741ms","error":"dial tcp 10.255.69.16:2380: i/o timeout"}
2024-11-15T16:08:38.473043376+08:00 stderr F {"level":"warn","ts":"2024-11-15T08:08:38.472799Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f7585d3c6edd2214","rtt":"8.629741ms","error":"dial tcp 10.255.69.16:2380: i/o timeout"}
```

```
1. as we can see ,the 10.255.69.16 is unhealthy status at 15T08:08:28.472346Z

2. but the apiserver still try to connect to this unhealthy member  at 2024-11-15T16:08:32

```

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```
1.25.8
</details>


### Cloud provider

<details>
no
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
5.15.131
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>