Repository: kubernetes
Issue: User can easily crash an entire cluster by creating pod with memory-backed EmptyDir
Labels: kind/bug,sig/storage,sig/node,triage/accepted

### What happened?

**The following phenomena were observed:**
* Pods that mounts a memory-backed EmptyDir with a default size (allocatable) was created on the node, and the app ran that quickly consumes this EmptyDir.
* Since it is not counted as memory usage unless EmptyDir is actually consumed, multiple such Pods can be created on the same node.
* Naturally, multiple tmpfs with approximately the same capacity as the node's memory size are also created.
  + This causes memory overcommited situation.
* Multiple EmptyDirs were consumed fast enough, then K8s tried to delete Pods and EmptyDirs due to excessive memory usage.
  + Pod status becomes OOMKilled
  + Other attempts to deploy pods (nginx) are not scheduled
* The deletion of tmpfs could not do in time, so the moment of memory exhaustion came and the node crashed.
  + If the number of replicas in a Deployment is set to be sufficiently higher, all schedulable nodes crashed.
  + Pod status becomes ContainerStatusUnknown
  + Pod status is Terminating but not deleted
  + The memory backed tmpfs on node is no longer deleted
  + The node becomes NotReady and you can't even log in with SSH
  + _**This could be a security issue...**_

**Operations and logs**
* kubelet log ([deployment-memvol-replica-6.log](https://github.com/kubernetes/kubernetes/files/12250107/deployment-memvol-replica-6.log))
  + 10:08 Worker VMs were restarted
  + Deploy `memvol` deployment with 6 replicas.
  + Each worker has two pods deployed
  + The Pods kept the Running state for about 5 minutes
, although they consumed most of the memory and CPU.
  + The node seems to be NotReady, so I deleted the deployment.
  + Stopped the worker VMs because the Pod remained in Terminating state.
  + Start worker VMs again.

### What did you expect to happen?

* Memory usage by EmptyDir should be strictry managed by k8s
  + Memory overcommit should be allowed by cluster admin, not by default.
  + Or enable to be controlled the total size of memory-backed EmptyDir per worker node.
  + Add taint to node for that memory-overcommit-allowed or memory-backed-emptydir-allowed.
* The default size of memory-backed EmptiDir should be smaller size that is less likely to cause problems.
  + e.g. 50Mi.

I hope that the cluster admins can prevent such situations where this kind of phenomenon can easily occur, whether due to a mistake or malicious intent, then the cluster admins can safely provide RAM disk to user.

### How can we reproduce it (as minimally and precisely as possible)?

Deploy following deployment with that `replicas` has double number of nodes:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: memvol
  name: memvol
spec:
  replicas: 6
  selector:
    matchLabels:
      app: memvol
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: memvol
    spec:
      containers:
      - image: busybox
        name: busybox
        command: ["sh"]
        args:
        - "-c"
        - "sleep 5 && dd if=/dev/zero of=/memvol/dump bs=1M count=4000"
        volumeMounts:
        - name: memvol
          mountPath: /memvol
      volumes:
      - name: memvol
        emptyDir:
          medium: Memory
```

### Anything else we need to know?

**The point would be how k8s manages memory usage especially in such case of overcommitted memory .
In such cases, the kube-reserved and system-reserved settings are not so effective.**

**Current workaround and its problems**
* Limiting user namespace memory usage using ResourceQuota seems to meet the requirements of a cluster admin. However, I think it would be difficult to get users to accept this, as it would require them to add memory usage information to all manifests.
* I understand that this can be achieved by adding an admission webhook on your own, but I think it would be better if K8s supported it.

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:49Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
```

</details>

### Cloud provider

<details>

```console
$ libvirtd --version
libvirtd (libvirt) 8.0.0
```

control plane * 1: 2CPU, 4GB memory, 25GB disk
worker * 3: 2CPU, 4GB memory, 25GB disk
* Results of `kubectl describe no`: https://github.com/kubernetes/kubernetes/files/13214680/nodes.txt

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux master01 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

</details>


### Install tools

<details>

```console
# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:19:40Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Container runtime (CRI) and version (if applicable)

<details>

```
root@master01:~# crictl --version
crictl version 1.26.0
root@master01:~# containerd --version
containerd github.com/containerd/containerd v1.7.2 0cae528dd6cb557f7201036e9f43420650207b58
```

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>