Repository: kubernetes
Issue: How to reduce the time consumption of apiserver.latency.k8s.io/etcd
Labels: kind/support,sig/api-machinery,triage/accepted

### What happened?

In a three node environment, powering off one of the nodes triggers a controller manager master-slave switch. When starting the controller manager, one controller starts for 10 seconds。Found a get request exceeding 10 seconds through audit log localization： mainly taking time at "apiserver. latency. k8s. io/etcd"
```
{
	"kind": "Event",
	"apiVersion": "audit.k8s.io/v1",
	"level": "Metadata",
	"auditID": "bc5c73d9-f6a5-4182-8980-2557fe668491",
	"stage": "ResponseComplete",
	"requestURI": "/api/v1/namespaces/kube-system/serviceaccounts/disruption-controller",
	"verb": "get",
	"user": {
		"username": "system:kube-controller-manager",
		"groups": [
			"system:kube-controller-managers",
			"system:authenticated"
		]
	},
	"sourceIPs": [
		"192.168.139.26"
	],
	"userAgent": "kube-controller-manager1/v1.28.1 (linux/amd64) kubernetes/d5c1d7d/kube-controller-manager",
	"objectRef": {
		"resource": "serviceaccounts",
		"namespace": "kube-system",
		"name": "disruption-controller",
		"apiVersion": "v1"
	},
	"responseStatus": {
		"metadata": {},
		"code": 200
	},
	"requestReceivedTimestamp": "2024-07-03T12:10:30.755816Z",
	"stageTimestamp": "2024-07-03T12:10:40.821090Z",
	"annotations": {
		"apiserver.latency.k8s.io/etcd": "10.061571043s",
		"apiserver.latency.k8s.io/response-write": "1.93µs",
		"apiserver.latency.k8s.io/serialize-response-object": "31.099µs",
		"apiserver.latency.k8s.io/total": "10.065274195s",
		"apiserver.latency.k8s.io/transform-response-object": "1.774µs",
		"authorization.k8s.io/decision": "allow",
		"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""
	}
}
```
apiserver log:

```
I0703 12:10:40.820718      11 trace.go:236] Trace[245231940]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:bc5c73d9-f6a5-4182-8980-2557fe668491,client:192.168.139.26,protocol:HTTP/2.0,resource:serviceaccounts,scope:resource,url:/api/v1/namespaces/kube-system/serviceaccounts/disruption-controller,user-agent:kube-controller-manager1/v1.28.1 (linux/amd64) kubernetes/d5c1d7d/kube-controller-manager,verb:GET (03-Jul-2024 12:10:30.758) (total time: 10061ms):
Trace[245231940]: ---"About to Get from storage" 0ms (12:10:30.758)
Trace[245231940]: ---"About to write a response" 10061ms (12:10:40.820)
Trace[245231940]: ---"Writing http response done" 0ms (12:10:40.820)
Trace[245231940]: [10.061771961s] [10.061771961s] END
I0703 12:10:40.820770      11 handler.go:153] kube-aggregator: GET "/apis/coordination.k8s.io/v1/namespaces/manager/leases/appcontroller.leader.lease" satisfied by nonGoRestful
```



### What did you expect to happen?

Controller get requests respond quickly, so that kube-controller-manager switches quickly

### How can we reproduce it (as minimally and precisely as possible)?

On a three node environment, the node where the down powered main kube controller manager is located.  There is a certain probability that this problem will occur

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1


</details>


### Cloud provider

<details>
none
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

NAME="EulerOS"
VERSION="2.0 (SP12x86_64)"
ID="euleros"
VERSION_ID="2.0"
PRETTY_NAME="EulerOS 2.0 (SP12x86_64)"
ANSI_COLOR="0;31"

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>