Repository: kubernetes
Issue: Container Limit cgroup causing OOMkilled 
Labels: kind/bug,sig/node,needs-triage

### What happened?

We recently applied a 500MiB limit to a deployment with one pod and one container in that pod. 

Upon doing so, the container started to be OOMkilled, however, the Prometheus metrics and K9S both reported that Memory usage never reached the limits, instead hovering around 300MiB. 

![image](https://user-images.githubusercontent.com/8082564/204023367-e77c38f7-484c-434b-8ec2-3063d8dfed2f.png)
<img width="1782" alt="image" src="https://user-images.githubusercontent.com/8082564/204023488-a39016d5-6735-4f4b-b2eb-9869a6989b27.png">

So I decided to look at the cgroup for the pod. First I grabbed the UUID with 
`➜  ~ kubectl get pods -n rentals germany-rental-order-vehicle-deployment-84675b696d-qz5g8 -o jsonpath='{.metadata.uid}'
22697648-cc63-47d6-ba97-fe420988dfc1%`

SSH'ed into the node and this is what I found:
```sh-4.2$ pwd
/sys/fs/cgroup/memory/kubepods/burstable/pod22697648-cc63-47d6-ba97-fe420988dfc1
sh-4.2$ cat memory.usage_in_bytes
501346304
sh-4.2$ cat memory.max_usage_in_bytes
524292096
```
```
sh-4.2$ pwd
/sys/fs/cgroup/memory/kubepods/burstable/pod22697648-cc63-47d6-ba97-fe420988dfc1
sh-4.2$ cat memory.limit_in_bytes
524288000
```
This doesn't match up to the data that K9S and Prometheus is displaying for the container, but it does match the green line here?

<img width="1791" alt="image" src="https://user-images.githubusercontent.com/8082564/204029050-446d6aae-0f90-42a7-a5c8-2ff55d444a31.png">


I wrote a quick one liner to output the memory usage and max memory usage for the 'Pod' folder and for the subfolders created under it, I assume those folders are separate containers, although I've not been able to confirm in docs, it would seem to make sense considering the values returned?

```/sys/fs/cgroup/memory/kubepods/burstable/pod22697648-cc63-47d6-ba97-fe420988dfc1
Fri Nov 25 16:33:13 UTC 2022
--------------
Pod Folder Memory Usage:
509992960
Pod Folder Max Memory Usage:
524292096
Sub Folders Memory Usage:
318349312
1097728
Sub Folders Max Memory Usage:
320966656
4866048
```

### What did you expect to happen?

I would expect the container not to be OOMkilled, as it would seem the container never actually reaches the limits set on this container of 500Mi. 

### How can we reproduce it (as minimally and precisely as possible)?

Hopefully the explanation above helps? 

On a minimal level, provision a deployment with one pod/one container that increases consuming memory with limits of 500Mi. 
Being a deployment, the RestartPolicy will be set to `Always`, and the container should restart in the pod. 

I can reproduce it in our environment, happy to share our manifest in a private conversation if needed. 

### Anything else we need to know?

```
[1488832.144451] kmem: usage 18592kB, limit 9007199254740988kB, failcnt 0
[1488832.148523] Memory cgroup stats for /kubepods/burstable/pod22697648-cc63-47d6-ba97-fe420988dfc1:
[1488832.148531] anon 322469888
                 file 181776384
                 kernel_stack 184320
                 slab 8921088
                 sock 0
                 shmem 0
                 file_mapped 0
                 file_dirty 109080576
                 file_writeback 72855552
                 anon_thp 0
                 inactive_anon 0
                 active_anon 322781184
                 inactive_file 90910720
                 active_file 91201536
                 unevictable 0
                 slab_reclaimable 6549504
                 slab_unreclaimable 2371584
                 pgfault 2490323
                 pgmajfault 323
                 workingset_refault 7524
                 workingset_activate 0
                 workingset_nodereclaim 25872
                 pgrefill 40672586
                 pgscan 43235632
                 pgsteal 1712550
                 pgactivate 41512034
                 pgdeactivate 40672586
                 pglazyfree 0
                 pglazyfreed 0
                 thp_fault_alloc 0
                 thp_collapse_alloc 0
[1488832.230699] Tasks state (memory values in pages):
[1488832.234160] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[1488832.240545] [  12924] 65535 12924      242        1    24576        0          -998 pause
[1488832.246805] [   1284]     0  1284   301156    88004  9060352        0           968 node
[1488832.253001] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=a716b36b230fce7613d219d41de103419a69e248b51e6f5ffeaef2913573f801,mems_allowed=0,oom_memcg=/kubepods/burstable/pod22697648-cc63-47d6-ba97-fe420988dfc1,task_memcg=/kubepods/burstable/pod22697648-cc63-47d6-ba97-fe420988dfc1/a716b36b230fce7613d219d41de103419a69e248b51e6f5ffeaef2913573f801,task=node,pid=1284,uid=0
[1488832.273119] Memory cgroup out of memory: Killed process 1284 (node) total-vm:1204624kB, anon-rss:315652kB, file-rss:36364kB, shmem-rss:0kB, UID:0 pgtables:8848kB oom_score_adj:968
```

### Kubernetes version

<details>
Kubernetes version
1.23

```
➜  ~ kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:25:45Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.13-eks-fb459a0", GitCommit:"55bd5d5cb7d32bc35e4e050f536181196fb8c6f7", GitTreeState:"clean", BuildDate:"2022-10-24T20:35:40Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

```

Kubelet version
v1.23.13-eks-fb459a0

</details>


### Cloud provider

<details>
AWS EKS 
</details>


### OS version

```
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
```

AMI Version
1.23.9-20220926

Kernel version
5.4.219-126.411.amzn2.x86_64

### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>
docker://20.10.17
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>