Repository: kubernetes
Issue: flowcontrol: maxCL is unreachable
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

With the current implementation of `maxCL`, it is possible that the value reported by `apiserver_flowcontrol_upper_limit_seats` exceeds the total concurrency limit. Since the concurrency limit of a given priority level is bound by the total concurrency limit, this effectively means the value reported by the metric is too loose.

Moreover, in practice a priority level of type `Limited` cannot borrow more than the total number of seats that other priority levels are configured to lend. This means that the total concurrency limit is generally speaking a loose upper bound that can be restricted even further.

Here below are the values of `apiserver_flowcontrol_upper_limit_seats` for an apiserver running with the default total concurrency limit of 600.

```bash
$ kubectl get --raw /metrics | grep apiserver_flowcontrol_upper_limit_seats
# HELP apiserver_flowcontrol_upper_limit_seats [ALPHA] Configured upper bound on number of execution seats available to each priority level
# TYPE apiserver_flowcontrol_upper_limit_seats gauge
apiserver_flowcontrol_upper_limit_seats{priority_level="catch-all"} 613
apiserver_flowcontrol_upper_limit_seats{priority_level="exempt"} 600
apiserver_flowcontrol_upper_limit_seats{priority_level="global-default"} 649
apiserver_flowcontrol_upper_limit_seats{priority_level="leader-election"} 625
apiserver_flowcontrol_upper_limit_seats{priority_level="node-high"} 698
apiserver_flowcontrol_upper_limit_seats{priority_level="system"} 674
apiserver_flowcontrol_upper_limit_seats{priority_level="workload-high"} 698
apiserver_flowcontrol_upper_limit_seats{priority_level="workload-low"} 845
```

### What did you expect to happen?

With the default priority levels and default values, that is total concurrency limit of 600 and 343 borrowable seats in total, we expect the following upper limits

```console
$ kubectl get --raw /metrics | grep apiserver_flowcontrol_upper_limit_seats
# HELP apiserver_flowcontrol_upper_limit_seats [ALPHA] Configured upper bound on number of execution seats available to each priority level
# TYPE apiserver_flowcontrol_upper_limit_seats gauge
apiserver_flowcontrol_upper_limit_seats{priority_level="catch-all"} 356        # = lower limit ( 13 ) + borrowable ( 343 )
apiserver_flowcontrol_upper_limit_seats{priority_level="exempt"} 600
apiserver_flowcontrol_upper_limit_seats{priority_level="global-default"} 367   # = lower limit ( 24 ) + borrowable ( 343 )
apiserver_flowcontrol_upper_limit_seats{priority_level="leader-election"} 368  # = lower limit ( 25 ) + borrowable ( 343 )
apiserver_flowcontrol_upper_limit_seats{priority_level="node-high"} 416        # = lower limit ( 73 ) + borrowable ( 343 )
apiserver_flowcontrol_upper_limit_seats{priority_level="system"} 393           # = lower limit ( 50 ) + borrowable ( 343 )
apiserver_flowcontrol_upper_limit_seats{priority_level="workload-high"} 392    # = lower limit ( 49 ) + borrowable ( 343 )
apiserver_flowcontrol_upper_limit_seats{priority_level="workload-low"} 367     # = lower limit ( 24 ) + borrowable ( 343 )
```

### How can we reproduce it (as minimally and precisely as possible)?

This behavior can be reproduced with a kind cluster using the default configuration
The total number of lendable seats sums up to 343, this is the maximum number of seats any limited priority level can borrow

```console
$ kind create cluster
$ kubectl get --raw /metrics | grep apiserver_flowcontrol_nominal_limit_seats
# HELP apiserver_flowcontrol_nominal_limit_seats [BETA] Nominal number of execution seats configured for each priority level
# TYPE apiserver_flowcontrol_nominal_limit_seats gauge
apiserver_flowcontrol_nominal_limit_seats{priority_level="catch-all"} 13        #  0% lendable =   0 lendable seats
apiserver_flowcontrol_nominal_limit_seats{priority_level="exempt"} 0
apiserver_flowcontrol_nominal_limit_seats{priority_level="global-default"} 49   # 50% lendable =  25 lendable seats
apiserver_flowcontrol_nominal_limit_seats{priority_level="leader-election"} 25  #  0% lendable =   0 lendable seats
apiserver_flowcontrol_nominal_limit_seats{priority_level="node-high"} 98        # 25% lendable =  25 lendable seats
apiserver_flowcontrol_nominal_limit_seats{priority_level="system"} 74           # 33% lendable =  24 lendable seats
apiserver_flowcontrol_nominal_limit_seats{priority_level="workload-high"} 98    # 50% lendable =  49 lendable seats
apiserver_flowcontrol_nominal_limit_seats{priority_level="workload-low"} 245    # 90% lendable = 220 lendable seats
```

### Anything else we need to know?

An accurate `apiserver_flowcontrol_upper_limit_seats` is useful to compare against `apiserver_flowcontrol_current_limit_seats`. If they are equal, operators can infer that the priority level cannot borrow any more seats, helping debug faster throttling issues.

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.31.3
Kustomize Version: v5.4.2
Server Version: v1.31.2
```

</details>


### Cloud provider

<details>
N/A
</details>


### OS version

<details>

```console
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux kind-control-plane 6.10.14-linuxkit #1 SMP Thu Oct 24 19:28:55 UTC 2024 aarch64 GNU/Linux
```

</details>


### Install tools

<details>
https://kind.sigs.k8s.io/#installation-and-usage
</details>


### Container runtime (CRI) and version (if applicable)

<details>
N/A
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
N/A
</details>