Repository: kubernetes
Issue: Analyzing and Addressing Unforeseen Performance Issues in a Large-Scale Kubernetes Cluster
Labels: kind/bug,sig/scalability,sig/api-machinery,triage/accepted,sig/etcd

### What happened?

Hi everyone,

I hope this message finds you well. I'm reaching out to share and seek advice on a performance issue we've encountered in our production Kubernetes (k8s) cluster. Our setup includes 300 nodes and supports 8,000 Pods, and we've recently started experiencing some concerns.

The issue began with a Pod we deployed, designed to periodically scrape Pod information by making an API call to `/api/v1/pods?fieldSelector=spec.nodeName%3D<node-name>` on the k8s API server. This Pod updates Pod information from its storage while periodically initiating these API calls.

As illustrated in Figure 1 below, there's been a noticeable increase in the request rate from an average of 2.75 to 4 requests per second. During this period, we observed significant delays in the k8s API servers when processing LIST verb API calls, as highlighted in Figure 2. Concurrently, there was a dramatic increase in the memory usage of the API servers, depicted in Figure 3. Additionally, the gRPC traffic from etcd to the k8s API servers also saw an increase, as shown in Figure 4.

However, what puzzles me is that the workload generated by this Pod appears to be too small to cause such a significant impact. It only contributes to 1.5 requests per second per API server, amounting to a total of 4.5 requests per second.

I'm eager to hear your thoughts, insights, or any advice you might have on addressing this issue. 

**[Fig. 1: requests per sec from k8s API servers]**
![k8s api 의 list verb의 total request](https://github.com/kubernetes/kubernetes/assets/4262433/d7eb114e-8823-4f36-8a56-c0092fad6291)

**[Fig. 2: 90%-tile Latency of Each Verb from k8s API server]**
![k8s api 의 verb 별 tail latency](https://github.com/kubernetes/kubernetes/assets/4262433/15810101-6817-45fc-b15a-f0e23e1a8b2e)

**[Fig. 3: k8s API servers' memory usage]**
![k8s apiserver 메모리 footprint](https://github.com/kubernetes/kubernetes/assets/4262433/7f53ad44-6e7b-41e6-8569-bd98f6dfe44f)


**[Fig. 4: outboud gRPC traffic from etcd]**
![etcd Client Traffic Out (해당 기간에 증가했음)](https://github.com/kubernetes/kubernetes/assets/4262433/b0027b3e-4d4a-4137-87d2-be99440a86bc)


### What did you expect to happen?

I expected that the workload would not impact the performance of the k8s API servers.

### How can we reproduce it (as minimally and precisely as possible)?

I think you should be able to replicate this issue by setting up a cluster with over 8,000 Pods and periodically fetching the Pods' information through API calls.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:10:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6", GitCommit:"d921bc6d1810da51177fbd0ed61dc811c5228097", GitTreeState:"clean", BuildDate:"2021-10-27T17:44:26Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
We have set up our cluster on-premises.

```console
$ cat /proc/cpuinfo
...
processor	: 111
vendor_id	: GenuineIntel
cpu family	: 6
model		: 106
model name	: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz
stepping	: 6
microcode	: 0xd0003a5
cpu MHz		: 3400.000

$ cat /proc/meminfo 
MemTotal:       528038752 kB
MemFree:        284106660 kB
MemAvailable:   464637832 kB
Buffers:         5636388 kB
Cached:         168626216 kB
SwapCached:            0 kB
Active:         66852988 kB
Inactive:       144133368 kB
Active(anon):   38236244 kB
Inactive(anon):   101448 kB
Active(file):   28616744 kB
Inactive(file): 144031920 kB
Unevictable:     9536672 kB
Mlocked:         9536672 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:            116556 kB
Writeback:             0 kB
AnonPages:      39823220 kB
Mapped:          4326780 kB
Shmem:           1242292 kB
KReclaimable:   10923516 kB
Slab:           13771744 kB
SReclaimable:   10923516 kB
SUnreclaim:      2848228 kB
KernelStack:       77232 kB
PageTables:       242660 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    261131696 kB
Committed_AS:   153182256 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      779924 kB
VmallocChunk:          0 kB
Percpu:           456704 kB
HardwareCorrupted:     0 kB
AnonHugePages:   4622336 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:    2820
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         5775360 kB
DirectMap4k:     7577012 kB
DirectMap2M:    355958784 kB
DirectMap1G:    175112192 kB
```

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ uname -a
Linux n6-main-master01 5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

</details>


### Install tools

<details>
kubespray v2.19
</details>


### Container runtime (CRI) and version (if applicable)

<details>
containerd v1.5.8
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
CNI=calico
CSI=weka-fs-plugin
</details>