Repository: kubernetes
Issue: Termination signals and pod/data safety impact
Labels: kind/bug,sig/storage,triage/accepted

### What happened?

Continuing discussion from slack and https://github.com/kubernetes/kubernetes/pull/115331#discussion_r1135663832

Background: Today, pod volume unmount completion is signaled by Pod deletion. We're discussing the various pod termination signals today, what controllers depend on those signals, and how they could impact pod + data safety.

> I realized that it may be better for us to make the kubelet not propagate the terminal phase until after volumes are both unmounted AND removed from the node, instead of just until they are unmounted today (deletion of the pod is blocked by removed).  i am trying to figure out a) what the real latency of removing volume mounts is on a node vs just unmounting them (even a gut check) and b) whether this is a safety issue that we glossed over before (currently terminal phase in 1.24 guarantees that the volume is not in active use, but not that the CSI plugin has had the chance to remove it) - i have no idea whether this is the case.

To answer the safety question, it is important to know that there is more cleanup after the pod volume unmount.

1. There is a global node-level mount that can be shared by multiple pods. The global mount is useful for serializing fs operations and avoiding mount limits from the node or storage backend. Cleanup only starts after all pods’ volume mounts are torn down, and this is typically where we flush data to disk (that the app has not done already). Kubelet signals completion of this by removing the volume from `Node.Status.VolumesInUse`
2. After the global mount has been cleaned up, then we can detach the volume from the node. The Attach-Detach controller waits for [pod termination signals](https://github.com/kubernetes/kubernetes/blob/18b5efceda2643b7175fbf712e8978d382077f9b/pkg/volume/util/util.go#L358) + the [global mount to complete teardown](https://github.com/kubernetes/kubernetes/blob/18b5efceda2643b7175fbf712e8978d382077f9b/pkg/controller/volume/attachdetach/reconciler/reconciler.go#L214). Signaling detach completion is only possible with CSI volumes + VolumeAttachment.

How this impacts pod + data safety:
- Workload controllers can create new pods before volumes have fully completed unmount + detach. Normally this is ok because new pods will block on detach from the old node. It does result in confusing “attach failed because it’s still attached to a different node” events. 
    - If a workload uses a non-attachable volume (like NFS), this potentially could result in data inconsistency if the app is relying on the unmount to flush data to disk. But this seems unlikely for typical DBs, and in general apps using multi-writer volumes already have to take care of concurrency. 
    - There's also a misconception about RWO enforcing single-writer. It allows multiple pods scheduled to the same node to share the volume, which has caused confusion in the past especially if a workload uses something like a Deployment instead of a StatefulSet. We have [ReadWriteOncePod access mode](https://github.com/kubernetes/enhancements/issues/2485) to address this.
- Node drain/eviction. Graceful node shutdown [doesn’t wait for volume teardon](https://github.com/kubernetes/kubernetes/issues/115148) even if there is time remaining on the shutdown timer. This can result in data loss/corruption since we don’t unmount cleanly. It could also cause pods to take longer to reschedule because instead of following the graceful volume detach flow, we trigger force detach logic after a 6 min timeout. A similar problem can happen during a node upgrade and using pod termination to signal when it’s safe to delete the VM. Actually, after https://github.com/kubernetes/kubernetes/pull/110721, I’m not sure if the 6 min force detach will get triggered; the path for handling Node deletion could be different. This needs further experimentation/investigation.
- PVC protection controller. This feature tries to prevent the volume from being deleted while still in use by a Pod. However, it only uses Pod deletion as a signal so it could result in stale mounts left on the node if the global mount was not cleaned up yet. https://github.com/kubernetes/kubernetes/issues/94108
- Volume snapshot/cloning. We have seen disk + fs inconsistencies if you try to snapshot or clone a volume while it is in the middle of detaching. We may need to look into ways to coordinate snapshot/clone operations with detach to address this.


### What did you expect to happen?

n/a

### How can we reproduce it (as minimally and precisely as possible)?

n/a

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>