Repository: kubernetes
Issue: The PV may be in a Terminating state and cannot be deleted when the pod are created and then the pod and pvc are quickly force deleted .
Labels: kind/bug,sig/storage,triage/accepted

### What happened?

In my testing environment, I discovered a strange phenomenon: when I create a pod that uses a PVC and then forcefully delete the pod and PVC before the pod creation is complete, the PV bound to the PVC **may** enter a Terminating state and cannot be deleted.

The log is as follows
```
I0614 10:04:56.153700   77153 kubelet.go:2389] "SyncLoop ADD" source="api" pods=[default/hp-volume-dbwfm]

I0614 10:04:58.233778   77153 reconciler_common.go:231] "operationExecutor.MountVolume started for volume \"pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820\" (UniqueName: \"kubernetes.io/csi/XXXX^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820\") pod \"hp-volume-dbwfm\" (UID: \"42077f53-c1df-4b6f-8647-f06100647095\") " pod="default/hp-volume-dbwfm"
I0614 10:04:58.234344   77153 operation_generator.go:1101] "MapVolume.WaitForAttach entering for volume \"pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820\" (UniqueName: \"kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820\") pod \"hp-volume-dbwfm\" (UID: \"42077f53-c1df-4b6f-8647-f06100647095\") DevicePath \"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820/eac1a6c0-bdc4-4b75-804f-7f89df07694b\"" pod="default/hp-volume-dbwfm"
I0614 10:04:58.308685   77153 operation_generator.go:1111] "MapVolume.WaitForAttach succeeded for volume \"pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820\" (UniqueName: \"kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820\") pod \"hp-volume-dbwfm\" (UID: \"42077f53-c1df-4b6f-8647-f06100647095\") DevicePath \"csi-75dac338e0aff53508bd6995ba2aa58d65d6eb42310c3d9d7111f2720a1d2823\"" pod="default/hp-volume-dbwfm"

I0614 10:05:02.444088   77153 kubelet.go:2405] "SyncLoop DELETE" source="api" pods=[default/hp-volume-dbwfm]
I0614 10:05:02.520860   77153 kubelet.go:2399] "SyncLoop REMOVE" source="api" pods=[default/hp-volume-dbwfm]

I0614 10:05:17.729528   77153 operation_generator.go:1225] "MapVolume.MapPodDevice succeeded for volume \"pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820\" (UniqueName: \"kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820\") pod \"hp-volume-dbwfm\" (UID: \"42077f53-c1df-4b6f-8647-f06100647095\") volumeMapPath \"/var/lib/kubelet/pods/42077f53-c1df-4b6f-8647-f06100647095/volumeDevices/kubernetes.io~csi\"" pod="default/hp-volume-dbwfm"
I0614 10:05:17.734612   77153 operation_generator.go:1233] MapVolume.NodeExpandVolume failed for volume "pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820" (UniqueName: "kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820") pod "hp-volume-dbwfm" (UID: "42077f53-c1df-4b6f-8647-f06100647095") : mountVolume.NodeExpandVolume get PVC failed : persistentvolumeclaims "pvc-volume-251e86b0-cfe7-475b-88f4-cced2416d3ff" is forbidden: User "system:node:worker3" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "default": no relationship found between node 'worker3' and this object
I0614 10:05:17.734835   77153 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820 podName: nodeName:}" failed. No retries permitted until 2024-06-14 10:05:18.734806235 +0800 CST m=+62756.584763139 (durationBeforeRetry 1s). Error: MapVolume.MarkVolumeAsMounted failed while expanding volume for volume "pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820" (UniqueName: "kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820") pod "hp-volume-dbwfm" (UID: "42077f53-c1df-4b6f-8647-f06100647095") : mountVolume.NodeExpandVolume get PVC failed : persistentvolumeclaims "pvc-volume-251e86b0-cfe7-475b-88f4-cced2416d3ff" is forbidden: User "system:node:worker3" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "default": no relationship found between node 'worker3' and this object
I0614 10:05:18.795992   77153 operation_generator.go:1101] "MapVolume.WaitForAttach entering for volume \"pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820\" (UniqueName: \"kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820\") pod \"hp-volume-r5sp7\" (UID: \"eac1a6c0-bdc4-4b75-804f-7f89df07694b\") DevicePath \"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820/42077f53-c1df-4b6f-8647-f06100647095\"" pod="default/hp-volume-r5sp7"
I0614 10:06:03.681110   77153 reconciler_common.go:231] "operationExecutor.MountVolume started for volume \"pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820\" (UniqueName: \"kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820\") pod \"hp-volume-dbwfm\" (UID: \"42077f53-c1df-4b6f-8647-f06100647095\") " pod="default/hp-volume-dbwfm"
I0614 10:06:03.681366   77153 operation_generator.go:1101] "MapVolume.WaitForAttach entering for volume \"pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820\" (UniqueName: \"kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820\") pod \"hp-volume-dbwfm\" (UID: \"42077f53-c1df-4b6f-8647-f06100647095\") DevicePath \"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820/7fb78f1b-06f4-4b95-8514-a80de1f0294d\"" pod="default/hp-volume-dbwfm"
I0614 10:06:03.726906   77153 operation_generator.go:1111] "MapVolume.WaitForAttach succeeded for volume \"pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820\" (UniqueName: \"kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820\") pod \"hp-volume-dbwfm\" (UID: \"42077f53-c1df-4b6f-8647-f06100647095\") DevicePath \"csi-75dac338e0aff53508bd6995ba2aa58d65d6eb42310c3d9d7111f2720a1d2823\"" pod="default/hp-volume-dbwfm"
I0614 10:06:15.315317   77153 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820 podName: nodeName:}" failed. No retries permitted until 2024-06-14 10:06:15.815290561 +0800 CST m=+62813.665247389 (durationBeforeRetry 500ms). Error: MapVolume.MapPodDevice failed for volume "pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820" (UniqueName: "kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820") pod "hp-volume-dbwfm" (UID: "42077f53-c1df-4b6f-8647-f06100647095") : rpc error: code = Internal desc = failed to check client lun mapping status for volume pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820
I0614 10:06:16.176459   77153 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820 podName: nodeName:}" failed. No retries permitted until 2024-06-14 10:06:16.676425647 +0800 CST m=+62814.526382436 (durationBeforeRetry 500ms). Error: UnmapDevice failed for volume "pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820" (UniqueName: "kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820") on node "worker3" : the device "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820/dev" is still referenced from other Pods [/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820/dev/42077f53-c1df-4b6f-8647-f06100647095]
I0614 10:06:16.696898   77153 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820 podName: nodeName:}" failed. No retries permitted until 2024-06-14 10:06:17.696865394 +0800 CST m=+62815.546822186 (durationBeforeRetry 1s). Error: UnmapDevice failed for volume "pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820" (UniqueName: "kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820") on node "worker3" : the device "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820/dev" is still referenced from other Pods [/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820/dev/42077f53-c1df-4b6f-8647-f06100647095]
I0614 10:06:17.742271   77153 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820 podName: nodeName:}" failed. No retries permitted until 2024-06-14 10:06:19.742240671 +0800 CST m=+62817.592197458 (durationBeforeRetry 2s). Error: UnmapDevice failed for volume "pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820" (UniqueName: "kubernetes.io/csi/iscsi.csi.xsky.com^csi-iscsi-pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820") on node "worker3" : the device "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820/dev" is still referenced from other Pods [/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820/dev/42077f53-c1df-4b6f-8647-f06100647095]

```

kubectl get pv pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820
```
pv-dc3407bc-6c3f-4123-bcf8-8f28b7575820   2Gi       RWX            Delete           Terminating   default/pvc-dc3407bc-6c3f-4123-bcf8-8f28b7575820  csi-XXX            1h
```

 I drawn up a time-series chart for the log contents .

![image](https://github.com/kubernetes/kubernetes/assets/8870947/7f97ebcb-b963-4c02-9cce-bcc5b040a558)


### What did you expect to happen?

the PV can be delete successfully

### How can we reproduce it (as minimally and precisely as possible)?

1. create a pvc , the bound pv's persistentVolumeReclaimPolicy is **Delete**;
2. create a pod use the pvc;
3. force delete the pod and the pvc（kubectl delete po xx --force && kubectl delete pvc yy --force ）
4. It is possible that the PV may be in a Terminating state and cannot be deleted

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```

</details>
1.28

### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>