Repository: kubernetes
Issue: failed to delete cgroup paths
Labels: kind/bug,sig/node,triage/accepted

### What happened?

Pods stuck in terminating state

lots of log entries like this one
```
journalctl -u kubelet --since -1m -f | grep "failed to delete cgroup paths"
Feb 28 05:57:26 worker kubelet[592400]: E0228 05:57:26.352096  592400 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable podc5659845-8967-408a-aa34-1223384f0ade] : unable to destroy cgroup paths for cgroup [kubepods burstable podc5659845-8967-408a-aa34-1223384f0ade] : Failed to remove paths: map[:/sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice blkio:/sys/fs/cgroup/blkio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice cpu:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice cpuacct:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice cpuset:/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice devices:/sys/fs/cgroup/devices/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice freezer:/sys/fs/cgroup/freezer/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice hugetlb:/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice memory:/sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice misc:/sys/fs/cgroup/misc/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice name=systemd:/sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice net_cls:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice net_prio:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice perf_event:/sys/fs/cgroup/perf_event/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice pids:/sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice rdma:/sys/fs/cgroup/rdma/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice]" pod="nams-prod-de/attachment-upload-service-cb986ddbc-zz85t" podUID="c5659845-8967-408a-aa34-1223384f0ade"
```

### What did you expect to happen?

Pods terminating and cgroups paths deleted

### How can we reproduce it (as minimally and precisely as possible)?

Not sure. In one cluster I have seen the issue multiple times. Other clusters with the same configuration do not show the same symptoms. 

### Anything else we need to know?

Unable to delete cgroup scope
```
rmdir /sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice/cri-containerd-1495ffbe17ae4d061ef354162d59a10fa7536e6512f8d280ceb1a21fa31fe684.scope
rmdir: failed to remove '/sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5659845_8967_408a_aa34_1223384f0ade.slice/cri-containerd-1495ffbe17ae4d061ef354162d59a10fa7536e6512f8d280ceb1a21fa31fe684.scope': Device or resource busy
```

Workaround
```
apt install cgroup-tools
lscgroup | grep <part-of-pod-id>
for cgroup in $(lscgroup | grep <part-of-pod-id>); do cgdelete -r $cgroup; done
```

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.10", GitCommit:"b8609d4dd75c5d6fba4a5eaa63a5507cb39a6e99", GitTreeState:"clean", BuildDate:"2023-10-18T11:44:31Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.5", GitCommit:"506050d61cf291218dfbd41ac93913945c9aa0da", GitTreeState:"clean", BuildDate:"2023-12-19T13:32:53Z", GoVersion:"go1.20.12", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
none, kubeadm based cluster on VMware VMs
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ uname -a
Linux gtloclvu14373 5.15.0-94-generic #104~20.04.1-Ubuntu SMP Tue Jan 16 13:34:09 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>
Kubermatic Kubernetes Platform v2.24
</details>


### Container runtime (CRI) and version (if applicable)

<details>
containerd containerd.io 1.6.28 ae07eda36dd25f8a1b98dfbf587313b99c0190bb

runc version 1.1.12
commit: v1.1.12-0-g51d5e94
spec: 1.0.2-dev
go: go1.20.13
libseccomp: 2.5.1
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
Canal v3.26.1
</details>