Repository: kubernetes
Issue: apiserver timeouts and random shutdown
Labels: kind/bug,sig/scalability,sig/api-machinery,triage/accepted

### What happened?

I'm running a k8s cluster using `kind`. Currently, there are ~3k agents connected. But in my tests, I frequently see connection timeouts, or connection reset messages from the `apiserver`. Example:
```
❯ kubectl get pods
Get "https://REDACTED:6443/api/v1/namespaces/default/pods?limit=500": net/http: TLS handshake timeout - error from a previous attempt: read tcp 10.219.21.215:58434->10.219.21.215:6443: read: connection reset by peer
```

Another issue is that the apiserver seems to shutdown without any obvious error message. Here are the logs from apiserver captured when I saw the 'connection reset' error, also indicates apiserver shutdown:
```
E1119 13:37:45.788062       1 writers.go:122] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1119 13:37:45.788243       1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 156.135µs, panicked: false, err: context canceled, panic-reason: <nil>" logger="UnhandledError"
E1119 13:37:45.789372       1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1119 13:37:45.791612       1 writers.go:135] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1119 13:37:45.792698       1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.6593ms" method="POST" path="/api/v1/namespaces/default/events" result=null
E1119 13:37:49.739514       1 writers.go:122] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1119 13:37:49.739573       1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 24.511µs, panicked: false, err: context canceled, panic-reason: <nil>" logger="UnhandledError"
E1119 13:37:49.739805       1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
E1119 13:37:49.740573       1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1119 13:37:49.740921       1 writers.go:122] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1119 13:37:49.741688       1 writers.go:135] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1119 13:37:49.741964       1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1119 13:37:49.742969       1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.77214ms" method="PATCH" path="/api/v1/namespaces/default/events/58a19fee-2f57-4f96-a4f6-0e70fba87637.1809622ae123cfd8" result=null
E1119 13:37:49.743003       1 writers.go:135] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1119 13:37:49.744130       1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.933539ms" method="GET" path="/api/v1/nodes/58a19fee-2f57-4f96-a4f6-0e70fba87637" result=null
E1119 13:39:54.379155       1 compact.go:124] etcd: endpoint ([https://127.0.0.1:2379]) compact failed: etcdserver: mvcc: required revision has been compacted
I1119 13:40:03.367380       1 controller.go:128] Shutting down kubernetes service endpoint reconciler
W1119 13:40:03.391092       1 lease.go:265] Resetting endpoints for master service "kubernetes" to []
I1119 13:40:03.411543       1 controller.go:86] Shutting down OpenAPI V3 AggregationController
I1119 13:40:03.411720       1 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller
I1119 13:40:03.411735       1 storage_flowcontrol.go:187] APF bootstrap ensurer is exiting
I1119 13:40:03.412052       1 available_controller.go:440] Shutting down AvailableConditionController
I1119 13:40:03.412063       1 controller.go:132] Ending legacy_token_tracking_controller
I1119 13:40:03.412067       1 controller.go:133] Shutting down legacy_token_tracking_controller
I1119 13:40:03.412075       1 system_namespaces_controller.go:76] Shutting down system namespaces controller
I1119 13:40:03.412082       1 autoregister_controller.go:168] Shutting down autoregister controller
I1119 13:40:03.412094       1 gc_controller.go:91] Shutting down apiserver lease garbage collector
I1119 13:40:03.412101       1 apf_controller.go:389] Shutting down API Priority and Fairness config worker
```

### What did you expect to happen?

apiserver shouldn't shutdown, and should work without timeout or connection-reset messages.

### How can we reproduce it (as minimally and precisely as possible)?

I don't have a minimal reproducer right now. I can describe my set-up in detail, if required.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
❯ kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.31.0
```

</details>


### Cloud provider

<details>
Not using Cloud
</details>


### OS version

<details>

```console
# On Linux:
❯ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.10 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.10 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.10
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.10"

❯ uname -a
Linux born22.toa.des.co 4.18.0-553.22.1.el8_10.x86_64 #1 SMP Wed Sep 11 18:02:00 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>