Repository: kubernetes
Issue: eviction manager fails to prioritize Hostpath Volume Pods for Eviction Under DiskPressure
Labels: priority/backlog,sig/node,kind/feature,triage/accepted

### What happened?

I have a GKE Cluster with a node that is running a bunch of innocent pods and a hostpath volume consuming pod. The pod eats up most of the disk space and puts Node under DiskPressure. Due to the DiskPressure NodeCondition, the eviction manager kicks in and ranks pods for eviction. However, the innocent pods are evicted before the culprit pod.

I have the repro script below which contains 15 innocent pods and 1 disk-eater pod. Here's the kubelet log when eviction manager ranks the pod for eviction:

```
I0923 00:46:49.313933    1787 eviction_manager.go:366] "Eviction manager: attempting to reclaim" resourceName="ephemeral-storage"
I0923 00:46:59.970765    1787 eviction_manager.go:377] "Eviction manager: must evict pod(s) to reclaim" resourceName="ephemeral-storage"
I0923 00:46:59.976721    1787 eviction_manager.go:395] "Eviction manager: pods ranked for eviction" pods=["default/hostnames-665b954fb4-v75xd","default/hostnames-665b954fb4-7gqt5","default/hostnames-665b954fb4-2xxdn","default/hostnames-665b954fb4-84gbj","default/hostnames-665b954fb4-v9442","default/hostnames-665b954fb4-nldw2","default/hostnames-665b954fb4-jqz7x","default/hostnames-665b954fb4-gwwrb","default/hostnames-665b954fb4-qmksn","default/hostnames-665b954fb4-p58gl","default/hostnames-665b954fb4-gxhlp","default/hostnames-665b954fb4-zds24","default/hostnames-665b954fb4-kkqpp","default/hostnames-665b954fb4-2tccd","default/hostnames-665b954fb4-pf2t4","kube-system/event-exporter-gke-78fb679b7b-dmb7t","default/disk-eater","gmp-system/collector-d2pgl","gmp-system/gmp-operator-76d464d74c-7p6bb","kube-system/konnectivity-agent-6584f57d94-gk4tn","kube-system/kube-dns-57cbb7b587-27mbb","gke-managed-cim/kube-state-metrics-0","kube-system/metrics-server-v1.30.3-7887b8869c-r6nqv","kube-system/konnectivity-agent-autoscaler-677598db7-6dkkl","kube-system/kube-dns-autoscaler-6f896b6968-x4bz2","kube-system/l7-default-backend-6697bb6dfd-fw88z","kube-system/gke-metrics-agent-f6vkb","kube-system/fluentbit-gke-4k7zj","kube-system/pdcsi-node-7dvb7","kube-system/kube-proxy-gke-cluster-2-default-pool-5e2b65c0-nvg6"]
```

It ranks 15 innocent pods for eviction first, then one system pod and then the disk eater pod.

### What did you expect to happen?

1. I think the safest thing to do in this case is kill the pod that is consuming hostpath volume. Since the lifecycle of hostpath volume is not tied up to the pod, kubelet cannot free up the space but it can at least kill the pod to stop the node becoming out of disk space.

2. Another issue is around telemetry signals. It is difficult to figure out that the disk pressure is due to hostPath volume consumption. Per-container metrics show low disk usage but node metrics show high disk usage. Unless you have the knowledge of how disk metrics are captured in k8s, it is difficult to come to conclusion that the disk pressure is due to hostPath. It would be good to have a signal that the DiskPressure is due to a hostPath volume usage.

### How can we reproduce it (as minimally and precisely as possible)?

1. Create a k8s cluster with a node that has disk space of 100Gi.
2. Deploy a bunch of innocent pods :

```
kubectl create deployment hostnames --image=nginx --replicas=15
```

3. Deploy a hostpath volume eater pod. 

```
apiVersion: v1
kind: Pod
metadata:
  name: disk-eater
spec:
  containers:
  - name: disk-eater
    image: busybox
    command: ["sh", "-c", "dd if=/dev/zero of=/data/file-$i bs=1G count=1024;"]
    volumeMounts:
    - name: host-volume
      mountPath: /data
  volumes:
  - name: host-volume
    hostPath:
      path: /home/kubernetes/data
      type: DirectoryOrCreate
```

### Anything else we need to know?

/sig node

### Kubernetes version

<details>

```console
1.30
```

</details>


### Cloud provider

<details>
GKE
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>