Repository: kubernetes
Issue: Pod Memory Request=Limit Triggers Pagecache Reclamation and Service Jitter on Node
Labels: kind/bug,priority/backlog,sig/node,triage/accepted

### What happened?

#### Description:
When deploying pods on a Kubernetes node with memory request equal to memory limit, despite no memory overcommitment on the host, scheduling a new pod triggers memory reclamation, leading to service jitter for some pods.
<img width="1285" alt="image" src="https://github.com/kubernetes/kubernetes/assets/58540165/dfdfa8ed-3d09-4060-b929-250688c8d786">


#### Observations:
1. Monitoring indicates that pagecache reclamation is being triggered.
2. Before the issue occurs, the node has sufficient memory requests available for scheduling the pod, and the host memory is not overcommitted.
3. Comparison of RSS (via `ps/top` on the host or `docker stats`) and `container_memory_usage_bytes` (from cAdvisor metrics, which are derived from `/sys/fs/cgroup/<pod_uid>/<container_uid>/memory/memory.usage_in_bytes`) shows a significant difference:
   - RSS: 14.2GB
   - `container_memory_usage_bytes`: 32GB (equal to the limit)

   This discrepancy can be attributed to RSS not including pagecache, while `memory.usage_in_bytes` does.

#### Questions and Analysis:
- If the host memory is not overcommitted and there are sufficient memory requests available for scheduling, why does scheduling a new pod trigger pagecache reclamation?
- Given that the limit already constrains the total memory usage (including pagecache), why does this issue arise?

#### Detailed Scenario:
- Our pods (Kafka, Elasticsearch) make extensive use of pagecache.
- Based on documentation, RSS metrics are from the `/proc` filesystem, while `memory.usage_in_bytes` is from the cgroup filesystem. The former excludes pagecache, and the latter includes it.
- This makes sense because the pods are designed to use significant pagecache. However, in a non-overcommitted memory scenario with sufficient remaining requests, new pod scheduling shouldn't cause pagecache reclamation issues.

#### Additional Information:
- Logs and monitoring metrics can be provided upon request.
- Detailed setup and configuration files can be shared if needed.

Thank you for investigating this issue.

### What did you expect to happen?

The node should handle new pod scheduling without triggering pagecache reclamation if the total memory usage is within limits and memory requests are available.

### How can we reproduce it (as minimally and precisely as possible)?

1. Deploy pods on a Kubernetes node with memory requests equal to memory limits.
2. Ensure the node has available memory requests and is not overcommitted.
3. Schedule a new pod on the node.
4. Observe the triggered pagecache reclamation and service jitter in existing pods.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.18.8-aliyun.1
WARNING: version difference between client (1.29) and server (1.18) exceeds the supported minor version skew of +/-1
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Alibaba Cloud Linux"
VERSION="3 (Soaring Falcon)"
ID="alinux"
ID_LIKE="rhel fedora centos anolis"
VERSION_ID="3"
PLATFORM_ID="platform:al8"
PRETTY_NAME="Alibaba Cloud Linux 3 (Soaring Falcon)"
ANSI_COLOR="0;31"
HOME_URL="https://www.aliyun.com/"
$ uname -a
Linux xxx 5.10.134-12.2.al8.x86_64 #1 SMP Thu Oct 27 10:07:15 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>