Repository: kubernetes
Issue: NotReady node with ContainerGCFailed warning
Labels: kind/documentation,sig/node,kind/feature,needs-kind,triage/accepted

### What happened?

**Kubernetes version** : 1.23 \
**Container runtime**: Docker

`kubectl describe node node1`

```
  Type     Reason             Age                   From     Message
  ----     ------             ----                  ----     -------
  Warning  ContainerGCFailed  2m35s (x507 over 9h)  kubelet  rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (16777539 vs. 16777216)
```

**Worker node details** \
`df -h`

```
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         32G     0   32G   0% /dev
tmpfs            32G     0   32G   0% /dev/shm
tmpfs            32G  1.1M   32G   1% /run
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/nvme0n1p1  200G   36G  165G  18% /
tmpfs           6.3G     0  6.3G   0% /run/user/1000
```

`docker system df`
```
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          27        26        18.23GB   1.433GB (7%)
Containers      24577     11        1.333GB   1.333GB (100%)
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B
```

Have enough space on worker nodes and no issues with memory/CPUs, issue might be because of lot of dead containers on worker nodes(24577). In earlier versions of `kubelet` configuration, there is an option to delete dead containers `--maximum-dead-containers` but it is deprecated now and it is replaced with `--eviction-hard` or `--eviction-soft` settings, with available eviction settings I can only configure `memory` `nodefs` `pid` eviction signals

How to specify `--maximum-dead-containers` or equivalent signal using updated `--eviction-hard` or `--eviction-soft` settings ?

`kubeletConfiguration` : https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/



### What did you expect to happen?

specify `--maximum-dead-containers` or equivalent signal using updated `--eviction-hard` or `--eviction-soft` settings 

### How can we reproduce it (as minimally and precisely as possible)?

Not quite sure if this can be reproducible as we have to create 1000s of containers to see this error

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-12T01:09:16Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.16-eks-48e63af", GitCommit:"e6332a8a3feb9e0fe3db851878f88cb73d49dd7a", GitTreeState:"clean", BuildDate:"2023-01-24T19:18:15Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
AWS EKS with AMI v1.23.17-eks-a59e1f0

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>