Repository: kubernetes
Issue: Eviction Manager should check the disk usage of dead containers
Labels: kind/bug,sig/node,triage/accepted

### What happened?

Under the condition of disk pressure, Eviction-manager checks disk usage of only living containers, which causes that it evicts wrong pods.

For example:
Pod A has 1 living container whose disk usage is 5 GB
Pod B has 1 living container whose disk usage is 0 GB and 1 dead container whose disk usage is 10 GB

When disk pressure occurs, Eviction manager attempts to evict Pod A prior to Pod B

### What did you expect to happen?

The Eviction Manager should also check the disk usage of dead containers and therefore in above example, it should evict Pod B prior than Pod A.

### How can we reproduce it (as minimally and precisely as possible)?

1. Deploying two pods having yaml

```
apiVersion: v1
kind: Pod
metadata:
  name: gc-test-a
spec:
  containers:
  - name: gc-test-a
    image: busybox
    command: ["/bin/sh", "-c"]
    args:
    - |
          trap "trap - TERM; kill 0; exit" TERM
          tail -f /dev/null & wait
  terminationGracePeriodSeconds: 30
  tolerations:
  - key: "conttips"
    operator: "Equal"
    value: "occupying"
    effect: "NoSchedule"
```
```
[vagrant@localhost ~]$ kubectl get pod 
NAME        READY   STATUS    RESTARTS   AGE
gc-test-a   1/1     Running   0          18s
gc-test-b   1/1     Running   0          15s
```
2.  Writing a 5 GB file to local disk on Pod A
```
[vagrant@localhost ~]$ kubectl exec -it gc-test-a -- /bin/sh
/ # dd if=/dev/zero of=/tmp/data-5G bs=1G count=5
5+0 records in
5+0 records out
5368709120 bytes (5.0GB) copied, 3.400959 seconds, 1.5GB/s
/ # exit
```

3.  Writing 10 GB file to local disk on Pod B, then kill Pod's process once for creating a dead container
```
[vagrant@localhost ~]$ kubectl exec -it gc-test-b -- /bin/sh
/ # dd if=/dev/zero of=/tmp/data-10G bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (10.0GB) copied, 7.148451 seconds, 1.4GB/s
/ # kill 1
/ # command terminated with exit code 137
```

Checking the status of both pods
```
[vagrant@localhost ~]$ kubectl get pod
NAME        READY   STATUS    RESTARTS      AGE
gc-test-a   1/1     Running   0             4m59s
gc-test-b   1/1     Running   1 (41s ago)   4m56s
```

Now Pod A has 1 living container using around 5GB of disk space and Pod B has 2 containers, 1 living container using very low disk space and 1 dead container having 10GB disk space

4. Checking the disk space of Pod A
```
[root@localhost vagrant]# crictl ps -a | grep gc-test-a
2df737ddcce30       docker.io/library/busybox@sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c   15 minutes ago      Running             gc-test-a                 0                   521de689e152a
[root@localhost vagrant]# crictl inspect 2df737ddcce30 | grep overlay | grep path
        "path": "/var/lib/containers/storage/overlay/e48aa1e8e6a8d3613830852fbe1608b6378e01ed827b37374dbba499519eb57c/merged"
[root@localhost vagrant]# ll /var/lib/containers/storage/overlay/e48aa1e8e6a8d3613830852fbe1608b6378e01ed827b37374dbba499519eb57c/merged/tmp
total 5242880
-rw-r--r--. 1 root root 5368709120 Jan 19 09:26 data-5G
```

5. Checking the disk space of Pod B
```
[root@localhost vagrant]# crictl ps -a | grep gc-test-b
68d80de2190e5       docker.io/library/busybox@sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c   13 minutes ago      Running             gc-test-b                 1                   66d602386868f
d257642c2f4c7       docker.io/library/busybox@sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c   17 minutes ago      Exited              gc-test-b                 0                   66d602386868f
[root@localhost vagrant]# crictl inspect 68d80de2190e5 | grep overlay | grep path
        "path": "/var/lib/containers/storage/overlay/3f106a027bf998c9158106f49207b43d75a3c007bba739358707efe34e5013a6/merged"
[root@localhost vagrant]# ll /var/lib/containers/storage/overlay/3f106a027bf998c9158106f49207b43d75a3c007bba739358707efe34e5013a6/merged/tmp
total 0
[root@localhost vagrant]# crictl inspect d257642c2f4c7 | grep overlay | grep path
        "path": "/var/lib/containers/storage/overlay/de79c84aa90961bd093edb88b454218518b6e127d53d45cca6c078bf9818e833/merged"
[root@localhost vagrant]# ll /var/lib/containers/storage/overlay/de79c84aa90961bd093edb88b454218518b6e127d53d45cca6c078bf9818e833/merged/tmp
total 10485760
-rw-r--r--. 1 root root 10737418240 Jan 19 09:27 data-10G
```

6. Creating a disk pressure on node
```
[root@localhost tmp]# dd if=/dev/zero of=data-10G-17 bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 6.90692 s, 1.6 GB/s
```

7. Checking the pod status
```
NAME        READY   STATUS                   RESTARTS	   AGE
gc-test-a   0/1     Evicted                  1             27m
gc-test-b   1/1     Running                  1 (22m ago)   27m
```
Pod A was Evicted first inspite of having less disk space than Pod B
Also, checking the logs, PodA's eviction rank was higher than PodB's

```
[root@host-192-168-167-146 tmp]# journalctl -u kubelet --since=today 
.....
Jan 19 09:50:14 localhost.localdomain kubelet[1329]: I0119 09:50:14.620676    1329 eviction_manager.go:367] "Eviction manager: pods ranked for eviction" pods=[default/gc-test-a kube-system/etcd-localhost.localdomain kube-system/kube-scheduler-localhost.localdomain kube-system/kube-controller-manager-localhost.localdomain tigera-operator/tigera-operator-946fd59c5-9b9mp kube-system/kube-apiserver-localhost.localdomain kube-system/metrics-server-7cf8b65d65-8v667 kube-system/calico-kube-controllers-84564464dc-zgss2 kube-system/coredns-64897985d-8fnbh kube-system/coredns-64897985d-8ln4n kube-system/calico-node-lzjw8 kube-system/kube-proxy-nq9mj default/gc-test-b]
.....
```


### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
k8s v1.21
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>
crio
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>