Repository: kubernetes
Issue: Regression in Scheduler Performance in Large Scale Clusters
Labels: kind/bug,sig/scheduling,kind/regression,needs-triage

### What happened?

Scheduler throughput and Performance has regressed in 1.31 when compared to 1.30

### What did you expect to happen?

Scheduler throughput and Performance should at least stay same as 1.30 on 1.31 or improve. 


### How can we reproduce it (as minimally and precisely as possible)?

I'm leveraging the [test](https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/testing/scheduler-throughput/config.yaml) that I have written to measure scheduler throughput and performance by directly creating pods to APIServer without KCM controllers in the picture.

Settings:

- You can run this test with [this](https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/testing/scheduler-throughput/config.yaml#L19) QPS set to `1000` and total pods [here](https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/testing/scheduler-throughput/config.yaml#L1C4-L1C32 ) set to 50k 


##### Test results: You would get roughly following latency and throughput numbers for `1.30v` 

Latency: 

```
    {
      "data": {
        "Perc50": 4853.606009,
        "Perc90": 6501.635529,
        "Perc99": 7152.44798
      },
      "unit": "ms",
      "labels": {
        "Metric": "pod_startup"
      }
    },
    {
      "data": {
        "Perc50": 3750.943,
        "Perc90": 5345.859,
        "Perc99": 5861.101
      },
      "unit": "ms",
      "labels": {
        "Metric": "create_to_schedule"
      }
    }
``` 

Throughput:



```
{
  "perc50": 803,
  "perc90": 856,
  "perc99": 936,
  "max": 936
}
```




##### Test results: You would get roughly following latency and throughput numbers for `1.31v` 

Latency:

```
    {
      "data": {
        "Perc50": 10675.556409,
        "Perc90": 17805.60988,
        "Perc99": 19445.20954
      },
      "unit": "ms",
      "labels": {
        "Metric": "pod_startup"
      }
    }
    
        {
      "data": {
        "Perc50": 9579.25,
        "Perc90": 16696.174,
        "Perc99": 18275.346
      },
      "unit": "ms",
      "labels": {
        "Metric": "create_to_schedule"
      }
    },
  ```
  
  Throughput:
  
  ```
  {
  "perc50": 638,
  "perc90": 683,
  "perc99": 713,
  "max": 713
}

```
  
  
  
  




### Anything else we need to know?



You can see that on 1.31v, the latency for `create_to_schedule` phase  increased 3X or more ( I have posted the one that has lowest latency and highest throughput among other tests that I have run) and Throughput has reduced significantly  from ~936 to ~704 at peak/p99.



When I looked at the  pprof of the runs on 1.30v and 1.31v, major differences showed up  as following:

-  Prometheus.(*gauge).Add [k8s.io/client-go/util/workqueue.ParallelizeUntil.func1]  (main contributing factor)
- k8s.io/kubernetes/pkg/scheduler/framework/parallelize.Parallelizer.Until.func1
k8s.io/kubernetes/pkg/scheduler/framework/parallelize/parallelism.go ( overall this is slightly higher on 1.31v i.e; ~57% vs ~46% on 1.30v) 

##### 1.31v pprof 

<img width="1575" alt="Screenshot 2024-10-07 at 5 19 48 PM" src="https://github.com/user-attachments/assets/e6bfd4c8-4e6b-4dc0-8868-1c63d8435913">


<img width="1562" alt="Screenshot 2024-10-07 at 5 20 19 PM" src="https://github.com/user-attachments/assets/f82c4ca7-c881-435e-91b1-818856968323">



##### 1.30v pprof

<img width="1576" alt="Screenshot 2024-10-07 at 5 07 22 PM" src="https://github.com/user-attachments/assets/9d287529-fcfc-433a-84f5-89172c42d483">
<img width="1590" alt="Screenshot 2024-10-07 at 5 21 18 PM" src="https://github.com/user-attachments/assets/a73647cf-cf45-4fe0-a5e9-e549eab053c8">



You can see that % of cpu cycles/time spent is more than doubled for Prometheus operations for the same amount of work i.e 50k pods with 1K QPS  ^^^


I can post flame graphs as well 


#### Food for thought: 

Generally we should `batch prometheus gauge operations` for performance improvement given the CPU cycles consumption in Scheduler_one go routine (as we schedule pods serially in one go routine given the nature of Scheduler) for better performance. Also, I don't think consumers would need precision at the current level that we are doing today. Generally users scrape prometheus metrics at the very least at 10sec, 30sec, 1min or 5mins interval. Would like to know what community thinks about this ? At least we should have a gating feature to configure the precision of omitting Prometheus metrics ?

### Kubernetes version

<details>

```console
$ kubectl version

sh-4.2$ kubectl version
Client Version: v1.30.4-eks-a737599
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.5-eks-ce1d5eb
```

```
sh-4.2$ kubectl version
Client Version: v1.31.0-eks-a737599
Kustomize Version: v5.4.2
Server Version: v1.31.0-eks-a737599

````

</details>


### Cloud provider

<details>
AWS
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
```
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
```
$ uname -a
```
Linux ip-172-16-60-69.us-west-2.compute.internal 5.10.224-212.876.amzn2.x86_64 #1 SMP Thu Aug 22 16:55:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
```

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>