Repository: kubernetes
Issue: One pv with NFS can be mounted by two pods even with ReadWriteOncePod enabled
Labels: kind/bug,sig/storage,triage/accepted

### What happened?

Cloud provider: openstack
Csi driver: NFS(Manila)

- First, I got one replicaSet with 1 replica and RWOP enabled so pod A comes up in node A. 
- Then, edit the nodeSelector to node B and forcibly delete the pod A which results in the pod A in terminate state and pod B is running in node B. 
- During this short period, pod A and pod B can both write data to the same pv even with RWOP enabled.
- After pod A is eliminated, pod B fully controls the volume.

### What did you expect to happen?

The pv should always read-write by a single pod but not access by two pods when the old one is in grace period.

### How can we reproduce it (as minimally and precisely as possible)?

1. Apply a replicaSet with 1 replica and RWOP enabled like:
```
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: writer
        image: busybox
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: my-volume
          mountPath: "/mnt"
        command: ["/bin/sh"]
        args:
        - -c
        - |
          while true; do
            echo $MY_POD_NAME $(date) >> /mnt/myfile.txt;
            sleep 1;
          done
      volumes:
      - name: my-volume
        persistentVolumeClaim:
          claimName: pvc-rwop
     nodeSelector:
       xxx
```
3. After pod is running, change the nodeSelector in replicaSet to another node
4. Forcibly delete the pod with `kubectl delete pod test-5nffh --force `
5. A new node `test-g6vnt `should come up in a different node and the old pod should be terminated
6. At this time, pv is writable for these two pods and the file /mnt/myfile.txt is overlapped like this:
```
# cat /mnt/myfile.txt
test-5nffh Tue Apr 9 07:29:03 UTC 2024
test-5nffh Tue Apr 9 07:29:04 UTC 2024
test-5nffh Tue Apr 9 07:29:05 UTC 2024
test-g6vnt Tue Apr 9 07:29:06 UTC 2024
test-5nffh Tue Apr 9 07:29:06 UTC 2024
test-g6vnt Tue Apr 9 07:29:07 UTC 2024
test-5nffh Tue Apr 9 07:29:07 UTC 2024
test-g6vnt Tue Apr 9 07:29:08 UTC 2024
test-5nffh Tue Apr 9 07:29:08 UTC 2024
```
7. That means during the shutdown period, pv can be accessible for two pods simultaneously 



### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# 
Server Version: v1.28.6
```

</details>


### Cloud provider

<details>
openstack
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
$ uname -a
Debian 6.1.77-0gardenlinux1 (2024-02-12) x86_64 x86_64 x86_64 GNU/Linux

```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
csidriver is: nfs.manila.csi.openstack.org
</details>