Repository: kubernetes
Issue: List request timeout not respected during response write stage
Labels: kind/bug,sig/scalability,sig/api-machinery,triage/accepted

### What happened?

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is serializing the response and writing it back to the client (whereas it is enforced on other request stages such as when listing from storage). Specifically, slow clients with insufficient CPU or network resources, can introduce arbitrarily high list latencies by being slow to receive the response. These requests will occupy APF seats for the entire duration 1+ minute duration they are executing which can starve other clients using the same FlowSchema. 

### What did you expect to happen?

We expected the list requests to be subject to the duration specified by the request-timeout flag.

### How can we reproduce it (as minimally and precisely as possible)?

1. Create 50k secrets.
2. List all the secrets in this namespace and force the HTTP client to be slow in receiving the response by leveraging the limit-rate flag.
```
curl --cert ./client.pem --key ./client-key.pem --cacert ./ca.pem $server/api/v1/namespaces/default/secrets --limit-rate 200k | wc -c
```
3. APIServer logs show the request took 5+ minutes to complete:
```
I0714 19:37:10.706663 11 trace.go:205] Trace[449374747]: "List" url:/api/v1/namespaces/default/secrets,user-agent:curl/8.0.1,audit-id:a09c55c2-0f84-4d6e-8fe6-fcde2e226b31,client:205.251.233.233,accept:*/*,protocol:HTTP/2.0 (14-Jul-2023 19:31:34.658) (total time: 336048ms):
Trace[449374747]: ---"Listing from storage done" 13749ms (19:31:48.408)
Trace[449374747]: ---"Writing http response done" count:40803 322298ms (19:37:10.706)
Trace[449374747]: [5m36.04813648s] [5m36.04813648s] END
```
4. The audit log entry gives the latency breakdown:
```
  "annotations": {
    "apiserver.latency.k8s.io/etcd": "13.522549999s",
    "apiserver.latency.k8s.io/response-write": "5m21.445820827s",
    "apiserver.latency.k8s.io/serialize-response-object": "5m22.298206407s",
    "apiserver.latency.k8s.io/total": "5m36.049590964s",
    "apiserver.latency.k8s.io/transform-response-object": "1.724µs",
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"eks:k8s-metrics\" of ClusterRole \"eks:k8s-metrics\" to User \"eks:k8s-metrics\""
  }
```
5. Checking APF concurrency in use shows the global-default bucket is using 10 seats for the full 5 minute request:
```
 kubectl get --raw /metrics --kubeconfig kubeconfig | grep apiserver_flowcontrol_request_concurrency_in_use
...
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="global-default",priority_level="global-default"} 10
```


### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.13-eks-0a21954", GitCommit:"6305d65c340554ad8b4d7a5f21391c9fa34932cb", GitTreeState:"clean", BuildDate:"2023-04-15T00:37:31Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.13-eks-0a21954", GitCommit:"6305d65c340554ad8b4d7a5f21391c9fa34932cb", GitTreeState:"clean", BuildDate:"2023-04-15T00:33:45Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"
```

</details>


### Cloud provider

<details>
AWS
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>