Repository: kubernetes
Issue: MountVolume.SetUp failed during restart after forced pod deletion
Labels: kind/bug,sig/storage,needs-triage

### What happened?

After a `scale replicas=0` followed by forced pod deletion (to skip the `terminationGracePeriodSeconds`) and than a `scale replicas=3` to "restart" the pods in a statefulset the used worker nodes run into volume mount problems.

```shell=bash
Error: MountVolume.SetUp failed for volume "pv-d062770-481c6f7874024488a2ab2197e5bb1931-a75d6422-ef6d-4e26-84b0-2988c2dba225" (UniqueName: "kubernetes.io/csi/cinder.csi.openstack.org^ae2e8567-1e7a-46bb-aaf1-ef9828e20175") pod "mariadb-g-2" (UID: "3606937f-b284-40a4-9ecf-a36cb5250753") : applyFSGroup failed for vol ae2e8567-1e7a-46bb-aaf1-ef9828e20175: readdirent /var/lib/kubelet/pods/3606937f-b284-40a4-9ecf-a36cb5250753/volumes/kubernetes.io~csi/pv-d062770-481c6f7874024488a2ab2197e5bb1931-a75d6422-ef6d-4e26-84b0-2988c2dba225/mount: input/output error

[Fri Sep  9 08:06:17 2022] sd 0:0:2:0: [sde] Write Protect is off
[Fri Sep  9 08:06:17 2022] sd 0:0:2:0: [sde] Mode Sense: 61 00 00 00
[Fri Sep  9 08:06:17 2022] sd 0:0:2:0: [sde] Cache data unavailable
[Fri Sep  9 08:06:17 2022] sd 0:0:2:0: [sde] Assuming drive cache: write through
[Fri Sep  9 08:06:17 2022] sd 0:0:2:0: [sde] Attached SCSI disk
[Fri Sep  9 08:06:20 2022] Aborting journal on device sdb-8.
[Fri Sep  9 08:06:20 2022] Buffer I/O error on dev sdb, logical block 1081344, lost sync page write
[Fri Sep  9 08:06:20 2022] JBD2: Error -5 detected when updating journal superblock for sdb-8.
[Fri Sep  9 08:06:46 2022] EXT4-fs warning (device sdc): htree_dirblock_to_tree:1044: inode #2: lblock 0: comm kubelet: error -5 reading directory block
```

The cinder block devices before mounted as `/dev/sdb` and `/dev/sdc` have been mounted as `/dev/sdd` and `/dev/sde` or vice versa. The OS already reports the new device paths

`lsblk`
```shell=bash
NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda       8:0    0   64G  0 disk
|-sda1    8:1    0  128M  0 part  /boot
|-sda2    8:2    0    2M  0 part
|-sda3    8:3    0    1G  0 part
...
sdd       8:48   0   10G  0 disk
sde       8:64   0    1G  0 disk
```

and the `kubelet` still wants to access the old paths
```shell=bash
kernel: EXT4-fs warning (device sdb): htree_dirblock_to_tree:1044: inode #2: lblock 0: comm kubelet: error -5 reading directory block
```

The mount info is also not updated.
`mount | grep csi/pv/pv`
```shell=bash
/dev/sdb on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-d062770-481c6f7874024488a2ab2197e5bb1931-c1f4f2ca-3983-4c10-b1df-89995b0fdede/globalmount type ext4 (rw,relatime,seclabel)
/dev/sdb on /var/lib/rkt/pods/run/c641a730-d05e-4952-9fe7-f506ab3606c2/stage1/rootfs/opt/stage2/kubelet/rootfs/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-d062770-481c6f7874024488a2ab2197e5bb1931-c1f4f2ca-3983-4c10-b1df-89995b0fdede/globalmount type ext4 (rw,relatime,seclabel)
/dev/sdb on /var/lib/rkt/pods/run/c054d122-fbe6-46b8-9d5a-8a9a6f5d6555/stage1/rootfs/opt/stage2/wormhole/rootfs/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-d062770-481c6f7874024488a2ab2197e5bb1931-c1f4f2ca-3983-4c10-b1df-89995b0fdede/globalmount type ext4 (rw,relatime,seclabel)
/dev/sdb on /var/lib/rkt/pods/run/c8dce3a3-1019-43e5-af8a-e5bd01ab50e3/stage1/rootfs/opt/stage2/flannel/rootfs/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-d062770-481c6f7874024488a2ab2197e5bb1931-c1f4f2ca-3983-4c10-b1df-89995b0fdede/globalmount type ext4 (rw,relatime,seclabel)
/dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-d062770-481c6f7874024488a2ab2197e5bb1931-a75d6422-ef6d-4e26-84b0-2988c2dba225/globalmount type ext4 (rw,relatime,seclabel)
/dev/sdc on /var/lib/rkt/pods/run/c641a730-d05e-4952-9fe7-f506ab3606c2/stage1/rootfs/opt/stage2/kubelet/rootfs/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-d062770-481c6f7874024488a2ab2197e5bb1931-a75d6422-ef6d-4e26-84b0-2988c2dba225/globalmount type ext4 (rw,relatime,seclabel)
/dev/sdc on /var/lib/rkt/pods/run/c054d122-fbe6-46b8-9d5a-8a9a6f5d6555/stage1/rootfs/opt/stage2/wormhole/rootfs/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-d062770-481c6f7874024488a2ab2197e5bb1931-a75d6422-ef6d-4e26-84b0-2988c2dba225/globalmount type ext4 (rw,relatime,seclabel)
/dev/sdc on /var/lib/rkt/pods/run/c8dce3a3-1019-43e5-af8a-e5bd01ab50e3/stage1/rootfs/opt/stage2/flannel/rootfs/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-d062770-481c6f7874024488a2ab2197e5bb1931-a75d6422-ef6d-4e26-84b0-2988c2dba225/globalmount type ext4 (rw,relatime,seclabel)
```

The worker nodes `only` run into that situation if `scale replicas=3` during the unmount/detach phase triggered the by the reconciliation of the forced delete before. It does `NOT` happen if the volumes are still mounted or if they have been already unmounted.

The only known fix for that problem is to delete the worker nodes and let the pods start on new nodes.

### What did you expect to happen?

The kubelet on the worker node should be able to detect and handle this race condition and avoid having to manually fix that problem.

### How can we reproduce it (as minimally and precisely as possible)?

- statefulset with persistent volumes (CSI Cinder block storage)
- prestop hook and terminationGracePeriodSeconds that avoid a fast shutdown of the pod
- `kubectl scale replicas=0 sts/xyz` to request the shutdown
- `kubectl delete pod/xyz-0 pod/xyz-1 pod/xyz-2 --grace-period=0 --force=true` to remove the pods in the terminating phase
- wait until the cloud provider (openstack) reports the volumes are currently unmounted from the worker node VM (~ 5 minutes)
- `kubectl scale replicas=3 sts/xyz` to start the pods "again"

### Anything else we need to know?

It works properly if

- persistent volumes from [NFS](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner) are used (no CSI)
- with Openstack and Cinder CSI if the force deletion is avoided
- GKE with gce-pd block storage is used

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.12", GitCommit:"b058e1760c79f46a834ba59bd7a3486ecf28237d", GitTreeState:"clean", BuildDate:"2022-07-13T14:52:41Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.14", GitCommit:"bccf857df03c5a99a35e34020b3b63055f0c12ec", GitTreeState:"clean", BuildDate:"2022-09-14T22:36:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
Openstack on VMware
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3227.2.1
VERSION_ID=3227.2.1
BUILD_ID=2022-08-03-2004
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3227.2.1 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3227.2.1:*:*:*:*:*:*:*"
$ uname -a
Linux ... 5.15.58-flatcar #1 SMP Wed Aug 3 19:35:04 -00 2022 x86_64 ...
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
cinder-csi-plugin: v1.22.0
</details>