Repository: kubernetes
Issue: kubelet restart during intermittent network issues unmounts volumes of running pods
Labels: kind/bug,sig/storage,triage/accepted

### What happened?

During kubelet startup, it checks all volumes against the desired state (`desired_state_of_world_populator.go`). One of these steps is a `Get` request [on the PVC](https://github.com/kubernetes/kubernetes/blob/v1.28.0/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go#L487C21-L487C21). If that API call fails due to an intermittent network issue between kubelet and the API server(s), the pod is not considered to be in the list of pods to mount, leading to the [`operationExecutor` to call `UnmountVolume`](https://github.com/kubernetes/kubernetes/blob/v1.28.0/pkg/kubelet/volumemanager/reconciler/reconciler_common.go#L166) in case of CSI-attached volumes. Although the volume will be mounted again as soon as the network issues resolves, the unmount/mount action leaves the bind mount in a broken and unrecoverable state depending on the CSI implementation (e.g. iscsi device name changed). To the user, nothing appears to be wrong as the pod is still in `Running` state. Only a manual deletion of the pod will fix it again.

We have observed this in production especially during maintenance where control-plane nodes are replaced and kubelet might not always be able to connect to the API server.

### What did you expect to happen?

I think kubelet should never call `UnmountVolume` on a running pod as it will leave the mounts in an unrecoverable state that is also not really visible to the user. The `desired_state_of_world_populator` should probably retry if it can't populate the desired state due to API server issues as it seems to make assumptions about the actual state which might be different.

### How can we reproduce it (as minimally and precisely as possible)?

This is quite the edge-case and requires a perfect storm of a somewhat, but not completely broken connection, between kubelet and the API server plus a badly-timed restart of kubelet. If the connection to the API server is fully gone, kubelet won't get into the `desired_state_of_world_populator` routine during startup and all is fine. So naturally, it's hard to reproduce the issue. I was able to reprocuce it with a lot of luck and inserting some well timed iptables rules just after restarting kubelet but this is far from consistent. So instead I have created a small proxy with which it can be consistently reproduced and put it into [this reproducer repo](https://github.com/ctrox/kubelet-unmount-reproducer) with all the required files and steps.

1. Clone the reproducer
    ```bash
    git clone https://github.com/ctrox/kubelet-unmount-reproducer.git
    cd kubelet-unmount-reproducer
    ```
1. Create a kind cluster (`v0.20.0` with the node image `1.27.3` was used in my case)

    ```bash
    kind create cluster
    ```

1. Install a CSI driver like [`csi-driver-host-path`](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/docs/deploy-1.17-and-later.md)

    ```bash
    kubectl apply -f csi/
    ```

1. Create a PVC and a Pod that mounts it

    ```bash
    kubectl apply -f sts.yaml
    ```

1. Build, copy and run the `pvcfailproxy`

    ```bash
    GOOS=linux go build -o pvcfailproxy
    docker cp pvcfailproxy kind-control-plane:/usr/bin/
    docker exec -ti kind-control-plane pvcfailproxy
    ```

1. Modify `/etc/kubernetes/kubelet.conf` to use the `pvcfailproxy` and restart kubelet

    ```bash
    # in another terminal
    docker exec -ti kind-control-plane sed -i 's/kind-control-plane:6443/kind-control-plane:8443/g' /etc/kubernetes/kubelet.conf
    docker exec -ti kind-control-plane systemctl restart kubelet
    ```

1. Observe the kubelet log if it contains the expected error and unmount operation

    ```bash
    docker exec -ti kind-control-plane journalctl -fu kubelet
    ```

    ```bash
    desired_state_of_world_populator.go:295] "Error processing volume" err="error processing PVC default/data-web-0: failed to fetch PV pvc-8308a5fc-c8b2-42c2-8f1c-c0183ea2bce5 from API server ...
    reconciler_common.go:172] "operationExecutor.UnmountVolume started for volume \"pvc-8308a5fc-c8b2-42c2-8f1c-c0183ea2bce5\"
    ```

### Anything else we need to know?

_No response_

### Kubernetes version

The issue was reproduced on a number of kubelet versions up to `1.28.0`. The API server version does not seem to be relevant.

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:48Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:38:14Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/arm64"}
```

</details>


### Cloud provider

<details>
should not be relevant
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='38.20230819.2.0'
$ uname -a
Linux localhost.localdomain 6.4.11-200.fc38.aarch64 #1 SMP PREEMPT_DYNAMIC Wed Aug 16 18:01:59 UTC 2023 aarch64 GNU/Linux
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>