Repository: kubernetes
Issue: Ephemeral storage limits error ContainerStatusUnknown 
Labels: kind/bug,sig/storage,sig/node,priority/important-longterm,needs-priority,triage/accepted

### What happened?

Pods are not being cleaned up once Ephemeral storage limits are hit

```
k describe pods -n ephemeral-metrics grow-test-7484cd87cb-wvpgt 
Events:
  Type     Reason               Age    From               Message
  ----     ------               ----   ----               -------
  Normal   Scheduled            4m52s  default-scheduler  Successfully assigned ephemeral-metrics/grow-test-7484cd87cb-q9ngs to ephemeral-metrics-cluster-worker
  Normal   Pulled               4m53s  kubelet            Container image "local.io/local/grow-test:latest" already present on machine
  Normal   Created              4m53s  kubelet            Created container grow-test
  Normal   Started              4m53s  kubelet            Started container grow-test
  Warning  Evicted              4m51s  kubelet            Pod ephemeral local storage usage exceeds the total limit of containers 5Ki.
  Normal   Killing              4m51s  kubelet            Stopping container grow-test
  Warning  ExceededGracePeriod  4m41s  kubelet            Container runtime did not kill the pod within specified grace period.
```

```
$ kubectl get pods 
ephemeral-metrics    grow-test-7484cd87cb-7lxp4                                        0/1     ContainerStatusUnknown   1              2m31s
ephemeral-metrics    grow-test-7484cd87cb-clqxj                                        0/1     ContainerStatusUnknown   1              3m3s
ephemeral-metrics    grow-test-7484cd87cb-gpxfk                                        0/1     ContainerStatusUnknown   1              119s
ephemeral-metrics    grow-test-7484cd87cb-gxqqr                                        0/1     ContainerStatusUnknown   1              3m35s
ephemeral-metrics    grow-test-7484cd87cb-j8fb7                                        1/1     Running                  0              23s
ephemeral-metrics    grow-test-7484cd87cb-psjpf                                        0/1     ContainerStatusUnknown   1              55s
ephemeral-metrics    grow-test-7484cd87cb-wvpgt                                        0/1     Error                    0              4m13s
ephemeral-metrics    grow-test-7484cd87cb-xqnnx                                        0/1     ContainerStatusUnknown   1              87s
```

### What did you expect to happen?

Pods to be cleaned up when an error occurs and a status that reflects the error better.

### How can we reproduce it (as minimally and precisely as possible)?

Add this to a pod manifest.

```
          resources:
            requests:
              ephemeral-storage: "1Ki"
            limits:
              ephemeral-storage: "5Ki"
``` 

I used this containerfile. I can publish if needed.
https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics/blob/master/DockerfileTestGrow

### Anything else we need to know?

_No response_

### Kubernetes version

Kind Cluster

```console
Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.0
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>