Repository: kubernetes
Issue: Node memory usage on cgroupv2 reported higher than cgroupv1
Labels: kind/bug,sig/node,triage/accepted

### What happened?

Node-level memory usage increases on a cluster with otherwise identical node and pod configurations by anywhere from 250-750 Mi (dependent on real resource usage, which is not exactly 1:1).

Reproducible on empty AKS/GKE clusters on Ubuntu (or containeros on GKE) by varying Kubernetes version to get cgroupv1 or cgroupv2 (AKS uses v2 on 1.25+, GKE on 1.26+).

This is causing a lot of pain for AKS users, see:
https://github.com/Azure/AKS/issues/3715
https://github.com/Azure/AKS/issues/3443

In my eyes this is a fairly substantial behavioral regression for cgroupv2 vs v1, but it's probably been there from the start.

### What did you expect to happen?

Memory usage should be roughly comparable across same-node configuration regardless of cgroupv1 or cgroupv2 usage (give or take).

Indeed, `/proc/meminfo` side by side shows total usage _is_ similar.

Note, this only affects node level stats, not pod level stats.

The reason is difference in cadvisor/runc calculation of the _root_ cgroup stats only (as memory.current does not exist for root cgroup in v2, while memory.usage_in_bytes does exist on root cgroup for v1).

### How can we reproduce it (as minimally and precisely as possible)?

Create an empty Kubernetes cluster (any version) with nodes on cgroupv1 and your tooling of choice.

Create the same cluster with nodes on cgroupv1. 

Compare `kubectl top node` on both clusters. Note that they are quite different.

### Anything else we need to know?

I _think_ this is an issue with cadvisor collection for root cgroup statistics.

I opened the issue here first for feedback before trying to fix it upstream.

## Background

kubelet uses cadvisor for node-level statistics (whether it uses cri or cadvisor stats provider for pods): https://github.com/kubernetes/kubernetes/blob/3c380199e9895efb47c6ad9035b2438a0a8bce5e/pkg/kubelet/stats/provider.go#L42-L70

`kubectl top node` or `kubectl get --raw /api/v1/nodes/foo/proxy/stats/summary | jq -C .node.memory` show higher values than the same cluster config + pods with cgroupv1.

`kubectl top node` uses node working set bytes: https://github.com/kubernetes-sigs/metrics-server/blob/5daafd91f74725f21f543c31682ecc3e3838a126/pkg/scraper/client/resource/decode.go#L34

kubelet stats provider uses cadvisor to get the cgroup stats:
https://github.com/kubernetes/kubernetes/blob/48a6fb0c428440fc43dfb2fb4ea707fac9dd60b9/pkg/kubelet/server/stats/summary.go#L130-L133
https://github.com/kubernetes/kubernetes/blob/48a6fb0c428440fc43dfb2fb4ea707fac9dd60b9/pkg/kubelet/stats/helper.go#L290-L295

Skipping some caching indirection in cadvisor, we eventually get a cgroup manager which is different based on cgroup v1/v2
https://github.com/google/cadvisor/blob/8164b38067246b36c773204f154604e2a1c962dc/container/libcontainer/helpers.go#L164-L169

these implementations differ in their calculation of memory usage for the root cgroup

v1 uses memory usage from `memory.usage_in_bytes`
https://github.com/opencontainers/runc/blob/92c71e725fc6421b6375ff128936a23c340e2d16/libcontainer/cgroups/fs/memory.go#L204-L224

v2 uses `/proc/meminfo` and calculates usage as `total - free` **(!!! this is key)**:
https://github.com/opencontainers/runc/blob/92c71e725fc6421b6375ff128936a23c340e2d16/libcontainer/cgroups/fs2/memory.go#L217

usage_in_bytes is roughly RSS + Cache. working set is usage - inactive file.

back in cadvisor, we drop inactive_file for working set: https://github.com/google/cadvisor/blob/8164b38067246b36c773204f154604e2a1c962dc/container/libcontainer/handler.go#L835-L844

## Hypothesis

my hypothesis of the problem: using `total - free` counts `inactive_anon` as part of usage, which the kernel will not count: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/memcontrol.c#n3720

This seems to only count NR_ANON_MAPPED. However, `total - free - inactive_file` still includes `inactive_anon`, which `mem_cgroup_usage` (`memory.usage_in_bytes` for v1) does not appear to count. Or at the very least, I believe NR_ANON_MAPPED != NR_INACTIVE_ANON + NR_ACTIVE_ANON, and can be less (there are anon pages which are not counted against NR_ANON_MAPPED (?)).

In my test nodes, `inactive_anon` almost exactly accounts for the diff I see.

---

forgive some minor discrepancies in precise numbers, these measurements were taken a few seconds apart, but should highlight the issue.

---

## cadvisor cgroupv1

```bash
~ # kubectl --context ace-v1 top node
NAME                                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
aks-nodepool1-57822035-vmss000000   98m          2%     1512Mi          12%
aks-nodepool1-57822035-vmss000001   99m          2%     1454Mi          11%
aks-nodepool1-57822035-vmss000002   94m          2%     1448Mi          11%
```

```bash
~ # cat /sys/fs/cgroup/memory/memory.usage_in_bytes
6236864512
~ # cat /sys/fs/cgroup/memory/memory.stat
cache 44662784
rss 3260416
rss_huge 2097152
shmem 65536
mapped_file 11083776
dirty 135168
writeback 0
pgpgin 114774
pgpgout 103506
pgfault 165891
pgmajfault 99
inactive_anon 135168
active_anon 3645440
inactive_file 5406720
active_file 39333888
unevictable 0
hierarchical_memory_limit 9223372036854771712
total_cache 5471584256
total_rss 767148032
total_rss_huge 559939584
total_shmem 1921024
total_mapped_file 605687808
total_dirty 270336
total_writeback 0
total_pgpgin 51679194
total_pgpgout 50291069
total_pgfault 97383769
total_pgmajfault 5610
total_inactive_anon 1081344
total_active_anon 772235264
total_inactive_file 4648124416
total_active_file 820551680
total_unevictable 0
```

```
~ # cat /proc/meminfo
MemTotal:       16393244 kB
MemFree:         9744148 kB
MemAvailable:   15020900 kB
Buffers:          132344 kB
Cached:          5207356 kB
SwapCached:            0 kB
Active:          1557252 kB
Inactive:        4526668 kB
Active(anon):     745916 kB
Inactive(anon):      792 kB
Active(file):     811336 kB
Inactive(file):  4525876 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               636 kB
Writeback:             0 kB
AnonPages:        618992 kB
Mapped:           624384 kB
Shmem:              2496 kB
KReclaimable:     285824 kB
Slab:             423600 kB
SReclaimable:     285824 kB
SUnreclaim:       137776 kB
KernelStack:        8400 kB
PageTables:         9060 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8196620 kB
Committed_AS:    2800016 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       40992 kB
VmallocChunk:          0 kB
Percpu:             4432 kB
HardwareCorrupted:     0 kB
AnonHugePages:    270336 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      302344 kB
DirectMap2M:     3891200 kB
DirectMap1G:    14680064 kB
```

### current calculation

memory.current - memory.stat.total_inactive_file = 6236864512 - 4648124416 = 1515 Mi  -> reported by kubelet

### "what if" cadvisor used v2 logic on this node

usage = total - free = 16393244 - 9744148 = 6649096 Ki -> fairly close to above, 10s of Mi off.
working_set = total - free - inactive_file = 16393244 - 9744148 - 4525876 = 2207 Mi

proposed_usage = total - free - inactive_anon = 16393244 - 9744148 - 792 = 6648304 Ki
proposed_working_set = total - free - inactive_file - inactive_anon = 16393244 - 9744148 - 4525876 - 792 = 2072 Mi -> matches v2 usage roughly

### cadvisor cgroupv2

```bash
~ # kubectl --context ace-v2 top node
NAME                                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
aks-nodepool1-38234323-vmss000000   113m         2%     2196Mi          17%
aks-nodepool1-38234323-vmss000001   112m         2%     2171Mi          17%
aks-nodepool1-38234323-vmss000002   113m         2%     2180Mi          17%
```

```
MemTotal:       16374584 kB
MemFree:         9505980 kB
MemAvailable:   14912544 kB
Buffers:          155164 kB
Cached:          5335576 kB
SwapCached:            0 kB
Active:           872420 kB
Inactive:        5399340 kB
Active(anon):       2568 kB
Inactive(anon):   791340 kB
Active(file):     869852 kB
Inactive(file):  4608000 kB
Unevictable:       30740 kB
Mlocked:           27668 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               148 kB
Writeback:             0 kB
AnonPages:        716552 kB
Mapped:           608424 kB
Shmem:              6320 kB
KReclaimable:     274360 kB
Slab:             355976 kB
SReclaimable:     274360 kB
SUnreclaim:        81616 kB
KernelStack:        8064 kB
PageTables:         7692 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8187292 kB
Committed_AS:    2605012 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       48092 kB
VmallocChunk:          0 kB
Percpu:             3472 kB
HardwareCorrupted:     0 kB
AnonHugePages:    409600 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      271624 kB
DirectMap2M:     8116224 kB
DirectMap1G:    10485760 kB
```

usage = total - free = 16374584 - 9505980
working_set = total - free - inactive_file = 16374584 - 9505980 - 4608000 = 2207 Mi -> reported by kubelet

### proposed correction

proposed_usage = total - free - inactive_anon = 16374584 - 9505980 - 791340 = 6077264 Ki
proposed_working_set = total - free - inactive_file - inactive_anon = 16374584 - 9505980 - 4608000 - 791340 = 1434 Mi  -> matches v1 usage roughly

---

this is also consistent with _pod_ level metrics working, while _node_ level data is wrong (it only affects the root cgroup due to v2 behavior).

---
## potential fixes

Either one could work. The runc fix is likely "more correct", since cgroupv2 usage is overestimated. And less code :)

### runc
```diff
diff --git a/libcontainer/cgroups/fs2/memory.go b/libcontainer/cgroups/fs2/memory.go
index e3b857dc..2f3c9fec 100644
--- a/libcontainer/cgroups/fs2/memory.go
+++ b/libcontainer/cgroups/fs2/memory.go
@@ -111,6 +111,7 @@ func statMemory(dirPath string, stats *cgroups.Stats) error {
                }
                return err
        }
+       memoryUsage.Usage = memoryUsage.Usage - stats.MemoryStats.Stats["inactive_anon"]
        stats.MemoryStats.Usage = memoryUsage
        swapUsage, err := getMemoryDataV2(dirPath, "swap")
        if err != nil {
```

### cadvisor

```diff
diff --git a/container/libcontainer/handler.go b/container/libcontainer/handler.go
index 2c4709e2..1dfcea82 100644
--- a/container/libcontainer/handler.go
+++ b/container/libcontainer/handler.go
@@ -832,6 +832,14 @@ func setMemoryStats(s *cgroups.Stats, ret *info.ContainerStats) {
                inactiveFileKeyName = "inactive_file"
        }

+       // correct usage to exclude inactive_anon.
+       // this would otherwise be counted for v2 but not v1.
+       // v1 only counts anon_mapped for usage
+       // alternatively, we could correct only working set.
+       if cgroups.IsCgroup2UnifiedMode() {
+               ret.Memory.Usage = ret.Memory.Usage - s.MemoryStats.Stats["inactive_anon"]
+       }
+
        workingSet := ret.Memory.Usage
        if v, ok := s.MemoryStats.Stats[inactiveFileKeyName]; ok {
                if workingSet < v {
```


### Kubernetes version

<details>

```console
$ kubectl version
1.27 client
1.26.3 and 1.24.10 servers
```

</details>


### Cloud provider

<details>
AKS v1.24.10, v1.26.3
GKE v1.24.14, v1.26.3
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>
containerd 1.7.1 or 1.6.18 (same issue)
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>