Repository: kubernetes
Issue: kubelet system-reserved unexpectedly sets memory.max in system-reserved-cgroup under systemd
Labels: kind/bug,priority/backlog,sig/node,triage/accepted

### What happened?

When starting kubelet with config `system-reserved=memory=1.5Gi` as part of [reserved
compute resources](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#system-reserved), running under `systemd`, with a `system-reserved-cgroup=systemreserved` (as per [Recommended Cgroups Setup](system-reserved-cgroup=systemreserved), that cgroup (`systemreserved`) has it's memory.max set to the value of `system-reserved=memory`, away from the default which is `memory.max=max`:

*Without* `system-reserved=memory=1.5Gi`:

```
[root@k8s-node1 systemreserved.slice]# pwd
/sys/fs/cgroup/systemreserved.slice
[root@k8s-node1 systemreserved.slice]# cat memory.max 
max
```


*With* `system-reserved=memory=1.5Gi`:
```
[root@k8s-node1 systemreserved.slice]# pwd
/sys/fs/cgroup/systemreserved.slice
[root@k8s-node1 systemreserved.slice]# cat memory.max 
1610612736 # 1.5Gi
```

### What did you expect to happen?

When I write `system-reserved=memory=1.5Gi` to the kubelet config, I don't expect that to result in `systemreserved` services
(service units within `systemreserved`, such as `sshd`, and any other system service within that cgroup to be collectivly *limmited* with a `memory.max` which sets a [*hard limit*](https://docs.kernel.org/admin-guide/cgroup-v2.html#:~:text=Memory%20usage%20hard%20limit) on these systemreserved services when they previously were not. This feels the *opposite* of what is intended- it should be the inverse (non systemreserved should 'back off' in the face of memory pressure to allow for 1.5G for the `systemreserved` cgroup.
 
Having `memory.max` set that may lead to these `systemreserved` services being *more likely* to be OOM killed since they get altered away from their default `memory.max=max`.


### How can we reproduce it (as minimally and precisely as possible)?

Start kubelet with --config (or args) with the following:

> Please note you have to restart the entire node to reset (kubelet does not clean up reset `system-reserved=memory=x` back to `max` after removing /restarting kubelet).

Apply the config to your kubelet start behaviour:

```
--system-reserved=memory=1.5Gi
--enforce-node-allocatable=pods,system-reserved
--system-reserved-cgroup=systemreserved
```

Ensure you have created the cgroup `systemreserved`, for example, by adding `Slice=systemreserved` to sshd:

File: /usr/lib/systemd/system/sshd.service
```
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.target
Wants=sshd-keygen.target

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
Slice=systemreserved.slice
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target
```
And reload your systemd daemon/restart sshds etc. (`systemctl daemon-reload && systemctl restart sshd`)

Observe memory.max being set:

```
[root@k8s-node1 systemreserved.slice]# cd /sys/fs/cgroup/systemreserved.slice
[root@k8s-node1 systemreserved.slice]# cat memory.max 
1610612736 # 1.5Gi
```

### Anything else we need to know?

I suspect Kubernetes needs to account for the Allocatable memory somewhere, which might by why `memory.max` is being altered?


Chris Down's suggestions may be relvant here, if kubelet *needs* to edit the cgroup, perhaps `memory.low` is more appropriate, and/or perhaps even
better allowing more fine grained control over which memory max/min/low get set rather than `--memory=` which doesn't appear to allow that level of expression.

> Our ultimate goal here is to keep the workload running ... so
what if could could encode that rather than sprinking memory.max everywhere? ... memory.low is a funamental different way about how we've controled memory for the last ~50 years. Instead we should just say how much memory the applications which we want to protect need, and let the system sort it out. ... memory.low hooks into the kernel's reclaiming structure in order to protect some memory for a cgroup. src: https://youtu.be/kPMZYoRxtmg?feature=shared&t=1242

### Kubernetes version

<details>

```console
$ kubectl version
Server Version: v1.29.3+rke2r1
```

</details>


### Cloud provider

<details>
Bare metal / not applicable
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.3 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.3 (Blue Onyx)"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

$ uname -a
Linux 5.14.0-362.13.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Dec 13 14:07:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>