Repository: kubernetes
Issue: orphaned pod <uid> found, but failed to rmdir() volume at path ...
Labels: kind/bug,sig/storage,lifecycle/rotten,needs-triage

### What happened?

After a k8s 1.28.9 node was rebooted the following errors started to be repeated in jounalctl:

`kubelet[2135]: E0911 16:27:10.436330 2135 kubelet_volumes.go:263] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"0a002fc3-b8f5-4a36-a8d2-e7a9c7bbe8ac\" found, but failed to rmdir() volume at path /var/lib/kubelet/pods/0a002fc3-b8f5-4a36-a8d2-e7a9c7bbe8ac/volumes/kubernetes.io~local-volume/local-pv-c20f968b: device or resource busy" numErrs=2`

The pod was running under a new uid and so both the old and new mounts were listed when I ran `mount | grep local-pv-c20f968b`:

```
/dev/mapper/apicSecureDisk on /var/lib/kubelet/pods/0a002fc3-b8f5-4a36-a8d2-e7a9c7bbe8ac/volumes/kubernetes.io~local-volume/local-pv-c20f968b type ext4 (rw,relatime)
/dev/mapper/apicSecureDisk on /data/secure/var/lib/kubelet/pods/0a002fc3-b8f5-4a36-a8d2-e7a9c7bbe8ac/volumes/kubernetes.io~local-volume/local-pv-c20f968b type ext4 (rw,relatime)
/dev/mapper/apicSecureDisk on /var/lib/kubelet/pods/3e64e4d8-f0b8-4031-a2c0-fa9cdcc86dd8/volumes/kubernetes.io~local-volume/local-pv-c20f968b type ext4 (rw,relatime)
/dev/mapper/apicSecureDisk on /data/secure/var/lib/kubelet/pods/3e64e4d8-f0b8-4031-a2c0-fa9cdcc86dd8/volumes/kubernetes.io~local-volume/local-pv-c20f968b type ext4 (rw,relatime)
```

In new pod any new directories that were created inside the double mounted volume got deleted within a few seconds of creation, so the new pod was throwing errors related to those directories vanishing. To fix the error and the pod I ran: `umount /var/lib/kubelet/pods/0a002fc3-b8f5-4a36-a8d2-e7a9c7bbe8ac/volumes/kubernetes.io~local-volume/local-pv-c20f968b` and then the issue was resolved.

### What did you expect to happen?

kubelet to unmount the orphaned pod directory without manual intervention

### How can we reproduce it (as minimally and precisely as possible)?

It does not happen everytime, but it does seem to only happen on node reboot. Perhaps it only happens with local storage as well.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.28.9
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -a
Linux apicdev4010 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>