Repository: kubernetes
Issue: client-go does not propagate updated timeout in request query parameter on retries
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

1. When client-go has a request timeout specified, it creates a `context.WithTimeout` context for the HTTP requests: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/rest/request.go#L969.

2. client-go propagates this timeout to the server by adding a `timeout` query parameter: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/rest/request.go#L518.

3. client-go retries GET requests if the server returns a retriable error (like 429).  All retries are passed the context created in 1.  The `timeout` query parameter value is always the same on every retry.  This causes the server and the client to have mismatched timeouts for requests.

For example, say I run `kubectl get configmaps --request-timeout 5s -v10` against a server that is being heavily throttled.  I see logs like:

```
round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.26.3 (linux/amd64) kubernetes/9e64410" -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" 'https://127.0.0.1:49179/api/v1/namespaces/default/configmaps?timeout=5s'
round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:49179 succeed
round_trippers.go:553] GET https://127.0.0.1:49179/api/v1/namespaces/default/configmaps?timeout=5s 429 Too Many Requests in 6 milliseconds
with_retry.go:234] Got a Retry-After 1s response for attempt 1 to https://127.0.0.1:49179/api/v1/namespaces/default/configmaps/big-666?timeout=5s
round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.26.3 (linux/amd64) kubernetes/9e64410" 'https://127.0.0.1:49179/api/v1/namespaces/default/configmaps?timeout=5s'
round_trippers.go:553] GET https://127.0.0.1:49179/api/v1/namespaces/default/configmaps?timeout=5s 429 Too Many Requests in 9 milliseconds
with_retry.go:234] Got a Retry-After 1s response for attempt 2 to https://127.0.0.1:49179/api/v1/namespaces/default/configmaps?timeout=5s
...
error: client rate limiter Wait returned an error: context deadline exceeded
```

### What did you expect to happen?

Retried requests should send an updated `timeout` query parameter that more closely matches how much time on the parent context is left.

### How can we reproduce it (as minimally and precisely as possible)?

Create a minikube cluster with `--extra-config apiserver.max-requests-inflight=1`. Apply the following manifests:

```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: throttled
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: throttled
subjects:
  - kind: User
    name: throttled
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: throttled
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: PriorityLevelConfiguration
metadata:
  name: throttled
spec:
  limited:
    lendablePercent: 0
    borrowingLimitPercent: 1
    limitResponse:
      type: Reject
    nominalConcurrencyShares: 1
  type: Limited
---
apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: FlowSchema
metadata:
  name: throttled
spec:
  distinguisherMethod:
    type: ByUser
  matchingPrecedence: 3
  priorityLevelConfiguration:
    name: throttled
  rules:
  - resourceRules:
    - apiGroups: ["*"]
      namespaces: ["*"]
      resources: ["*"]
      verbs: ["*"]
    subjects:
    - kind: User
      user:
        name: throttled
```

Generate a x509 certificate with '/CN=throttled' and create a new kubectl user and context using the certificate (see https://medium.com/@HoussemDellai/rbac-with-kubernetes-in-minikube-4deed658ea7b).  Change the kubectl context to the created context.  Kubectl traffic should now be mapped to the 'throttled' priority level (default is 'exempt' because the minikube user is in system:masters group).

Create a bunch of big configmaps and run some parallel `kubectl get configmaps --request-timeout=5s`.  Some of the requests will be rejected with 429 and the retries will have an incorrect `timeout=5s` query parameter.  

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
None
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>