Repository: kubernetes
Issue: stdout logs should not be accounted in ephemeral storage usage
Labels: sig/node,kind/feature,needs-triage

### What happened?

This question has originally been reported under https://discuss.kubernetes.io/t/why-stdout-logs-are-accounted-in-ephemeral-storage-usage/27815.
However, the more I think about it, the more I'm convinced that this is an actual issue.
When Kubelet calculates ephemeral storage usage, it incorporates the CRI managed logs' usage in the calculation:
Code snippet from pkg/kubelet/eviction/eviction_manager.go:
```
func (m *managerImpl) podEphemeralStorageLimitEviction(podStats statsapi.PodStats, pod *v1.Pod) bool {
	podLimits := resourcehelper.PodLimits(pod, resourcehelper.PodResourcesOptions{})
	_, found := podLimits[v1.ResourceEphemeralStorage]
	if !found {
		return false
	}

	// pod stats api summarizes ephemeral storage usage (container, emptyDir, host[etc-hosts, logs])
	podEphemeralStorageTotalUsage := &resource.Quantity{}
	if podStats.EphemeralStorage != nil && podStats.EphemeralStorage.UsedBytes != nil {
		podEphemeralStorageTotalUsage = resource.NewQuantity(int64(*podStats.EphemeralStorage.UsedBytes), resource.BinarySI)
	}
	podEphemeralStorageLimit := podLimits[v1.ResourceEphemeralStorage]
	if podEphemeralStorageTotalUsage.Cmp(podEphemeralStorageLimit) > 0 {
		// the total usage of pod exceeds the total size limit of containers, evict the pod
		message := fmt.Sprintf(podEphemeralStorageMessageFmt, podEphemeralStorageLimit.String())
		if m.evictPod(pod, 0, message, nil, nil) {
			metrics.Evictions.WithLabelValues(signalEphemeralPodFsLimit).Inc()
			return true
		}
		return false
	}
	return false
}
```
However those log messages are completely out of control of the container itself, as it has no influence of their retention / lifecycle after leaving the container premises: Instead it's Kubelet's responsibility to define how long it wants to retain those logs, based on the --container-log-max-files , --container-log-max-size settings.

Because of this, if the container-log-max-size * container-log-max-files is more than the container's  ‘ephemeral-storage’ limit, the container could be evicted (without writing a single byte to any of it's mounted volumes) over it's lifetime.

### What did you expect to happen?

POD / container should not be evicted without reaching the container defined limit related to the ephemeral volumes under the container's control.

### How can we reproduce it (as minimally and precisely as possible)?

PoC example: 
```
---
apiVersion: v1
kind: Pod
metadata:
  name: log-test
spec:
  containers:
  - image: k8s.gcr.io/busybox:latest
    name: test
    command: ["/bin/sh"]
    args:
    - -c
    - |-
      yes $(printf 'Hello world!!!!\n%.0s' `seq 1 64`) | dd bs=1024 count=204800
      while sleep 3600; do
        true
      done
    resources:
      limits:
        ephemeral-storage: "10Mi"
```

### Anything else we need to know?

_No response_

### Kubernetes version

```
$ kubectl version --output=json
{
  "clientVersion": {
    "major": "1",
    "minor": "27",
    "gitVersion": "v1.27.6",
    "gitCommit": "741c8db18a52787d734cbe4795f0b4ad860906d6",
    "gitTreeState": "clean",
    "buildDate": "2023-09-13T09:21:34Z",
    "goVersion": "go1.20.8",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "kustomizeVersion": "v5.0.1",
  "serverVersion": {
    "major": "1",
    "minor": "28",
    "gitVersion": "v1.28.4",
    "gitCommit": "bae2c62678db2b5053817bc97181fcc2e8388103",
    "gitTreeState": "clean",
    "buildDate": "2023-11-15T16:48:54Z",
    "goVersion": "go1.20.11",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}
```

### Cloud provider

N/A

### OS version

_No response_

### Install tools

_No response_

### Container runtime (CRI) and version (if applicable)

_No response_

### Related plugins (CNI, CSI, ...) and versions (if applicable)

_No response_