Repository: kubernetes
Issue: Kubernetes appears to use a lot of memory for its own components (≅80GiB)
Labels: kind/bug,sig/scalability,needs-triage

### What happened?

I recently installed Kubernetes on an Ubuntu 22.04 system environment. I set up the Kubernetes environment as follows: as you can see, I created one control node and two worker nodes.

```
-----------+---------------------------+--------------------------+------------
           |                           |                          |
       eth0|10.0.0.10              eth0|10.0.0.11             eth0|10.0.0.22
+----------+-----------+   +-----------+-----------+   +-----------+-----------+
|  [ ctrl.myk8s     ]  |   |  [ node01.myk8s    ]  |   |  [ node02.myk8s    ]  |
|     Control Node     |   |       Worker Node     |   |      Worker Node      |
+----------------------+   +-----------------------+   +-----------------------+
```

I started the Kubernetes service daemons and ran some pods. After a few hours, every time I ran the `free -h` command, I noticed that 80GB of memory was being used. Most of this memory usage is attributed to Kubernetes processes. 

I am curious why Kubernetes uses so much physical memory, even when running just one or two pods. I welcome any comments, hints, or information.

### Memory usage: 

```bash
$ free -h 
    total          used        free       shared   buff/cache   available
Mem:           124Gi       85.6Gi       38Gi        21Mi       452Mi          38Gi
Swap:             0B              0B          0B

$  ps -eo uid,euser,wchan,comm,rss,vsize,%cpu,nlwp | awk '$5 != 0' | sort -k5,5rn

uid   euser    wchan  comm            rss   vsize   %cpu nlwp 
----------------------------------------------------------------------
    0 root     futex_ kubelet         117576 6850960 1.1   76
    0 root     futex_ containerd      100948 7420620 0.6   82
    0 root     futex_ calico-node     93024  5417080 0.4   56
    0 root     futex_ calico-node     83600  3494160 0.0   30
    0 root     futex_ calico-node     83192  3715612 0.0   33
    0 root     futex_ calico-node     80348  3641880 0.0   32
    0 root     futex_ calico-node     79912  3494416 0.0   30
    0 root     futex_ calico-node     74932  3493392 0.0   30
    0 root     futex_ kube-proxy      72944  1295544 0.0   32
    0 root     do_pol python3         31636   42076  0.0    1
    0 root     futex_ node-driver-reg 29436  1248032 0.0   32
    0 root     futex_ tuned           26628  257916  0.0    4
  997 polkitd  do_pol polkitd         22804  2983876 0.0   12
    0 root     futex_ csi-driver      21824  1237992 0.0   24
    0 root     do_pol NetworkManager  21480  259196  0.0    3
    0 root     futex_ containerd-shim 17492  1238184 0.0   14
    0 root     futex_ containerd-shim 16600  1238184 0.0   13
    0 root     ep_pol systemd         15192  175176  0.0    1
    0 root     futex_ containerd-shim 13688  1238184 0.0   13
    0 root     do_sel snmpd           13056   25748  0.0    1
    0 root     ep_pol systemd-journal 13056   27820  0.0    1
 1000 codegrok ep_pol systemd         13056   23928  0.0    1
    0 root     futex_ containerd-shim 12704  1238184 0.0   13

```


![k8s-85gb-mem-hog](https://github.com/user-attachments/assets/c4f4919f-db8f-467e-9f0d-73d81460d0dc)



### What did you expect to happen?

I expected Kubernetes to use less than 8GB of memory when running just one or two pods.

### How can we reproduce it (as minimally and precisely as possible)?

### [1] 	Install Containerd and apply some requirements on all Nodes.
```
root@ctrl:~# apt -y install containerd
root@ctrl:~# cat > /etc/sysctl.d/99-k8s-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
EOF
root@ctrl:~# sysctl --system
root@ctrl:~# modprobe overlay; modprobe br_netfilter

root@ctrl:~# echo -e overlay\\nbr_netfilter > /etc/modules-load.d/k8s.conf
# needs [iptables-legacy] for iptables backend
# if nftables is enabled, change to [iptables-legacy]

root@ctrl:~# update-alternatives --config iptables

There are 2 choices for the alternative iptables (providing /usr/sbin/iptables).

  Selection    Path                       Priority   Status
------------------------------------------------------------
* 0            /usr/sbin/iptables-nft      20        auto mode
  1            /usr/sbin/iptables-legacy   10        manual mode
  2            /usr/sbin/iptables-nft      20        manual mode

Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in manual mode

# disable swap

root@ctrl:~# swapoff -a

root@ctrl:~# vi /etc/fstab

# comment out
#/swap.img      none    swap    sw      0       0

# switch to Cgroup v1 (default is v2)

root@ctrl:~# vi /etc/default/grub
# line 11 : add

GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0
"
root@ctrl:~# update-grub
```



### [2] 	Install Kubeadm, Kubelet, Kubectl on all Nodes.
```
root@ctrl:~# curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg -o /etc/apt/keyrings/kubernetes-keyring.gpg

root@ctrl:~# echo "deb [signed-by=/etc/apt/keyrings/kubernetes-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list

root@ctrl:~# apt update

root@ctrl:~# apt -y install kubeadm kubelet kubectl
root@ctrl:~# vi /etc/default/kubelet
# create new

KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock
root@ctrl:~# systemctl edit containerd.service
# add follows

[Service]
KillMode=
KillMode=mixed
root@ctrl:~# systemctl restart containerd.service 
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
# kubectl version
Client Version: v1.31.3
Kustomize Version: v5.4.2
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -a
Linux mymate 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>