Repository: kubernetes
Issue: The status of asw in attachDetachController is inconsistent with the actual node status
Labels: kind/bug,sig/storage,needs-triage

### What happened?

We are operating a Kubernetes (k8s) cluster that utilizes a scheduler extender to schedule pods from a StatefulSet onto fixed nodes respectively. And then, these pods would mount some volumes. We routinely delete and restart pods, performing this operation thousands of times weekly. However, we encounter an issue every few weeks where, after deleting a pod and scheduling it back to its original node, the associated volume fails to mount. As a result, this causes the pod to enter a perpetual ContainerCreating state.
```shell
# kubectl describe pod x
  Warning  FailedMount  105s (x30 over 67m)  kubelet, h34b07425.na61  Unable to mount volumes for pod "*": timeout expired waiting for volumes to attach or mount for pod "*". list of unmounted volumes=[*]. list of unattached volumes=[*]
```

We have traced the issue to the attach/detach (ad) controller. It appears that the node volume does not successfully attach, yet the actual state of the world (asw) cache within the ad controller mistakenly believes that it has. This discrepancy prevents the pod from proceeding with the mount operation.

Upon reviewing the kube-controller-manager logs, we noticed the following:
```shell
I1211 00:42:07.780851       1 node_status_updater.go:106] Updating status "{\"status\":{\"volumesAttached\":[{\"devicePath\":\"\",\"name\":\"kubernetes.io/csi/csi-hostpath^6583bc90-0677-11ee-a5d3-506b4b2b7da6\"}]}}" for node "h34b07425.na61" succeeded. VolumesAttached: [{kubernetes.io/csi/csi-hostpath^6583bc90-0677-11ee-a5d3-506b4b2b7da6 }]
...
I1211 00:42:08.261836       1 node_status_updater.go:106] Updating status "{}" for node "h34b07425.na61" succeeded. VolumesAttached: [{kubernetes.io/csi/csi-hostpath^6583bc90-0677-11ee-a5d3-506b4b2b7da6 } {kubernetes.io/csi/csi-hostpath^3d7384a6-068e-11ee-a5d3-506b4b2b7da6 }]
```
The issue seems to stem from a delay in the informer syncing the node information from the API server after a DetachVolume has been invoked, resulting in the informer not accurately reflecting the node's most recent status. It incorrectly assumes that the volume is still attached to the node. 

```go
// pkg/controller/volume/attachdetach/reconciler/reconciler.go
func (rc *reconciler) reconcile(ctx context.Context) {
		err = rc.nodeStatusUpdater.UpdateNodeStatusForNode(logger, attachedVolume.NodeName)
		...
		err = rc.attacherDetacher.DetachVolume(logger, attachedVolume.AttachedVolume, verifySafeToDetach, rc.actualStateOfWorld)
	...
	rc.attachDesiredVolumes(logger)

	// Update Node Status
	err := rc.nodeStatusUpdater.UpdateNodeStatuses(logger)
	...
}
```
After the pod schedules back to the original nodes and ad-controller finishes the AttachVolume operation, the nodeStatusUpdater.updateNodeStatus() function returns nil without retrying, regardless of whether there were any updates. 
```go
// pkg/controller/volume/attachdetach/statusupdater/node_status_updater.go
func (nsu *nodeStatusUpdater) processNodeVolumes(logger klog.Logger, nodeName types.NodeName, attachedVolumes []v1.AttachedVolume) error {
	nodeObj, err := nsu.nodeLister.Get(string(nodeName))
	...
	err = nsu.updateNodeStatus(logger, nodeName, nodeObj, attachedVolumes)
	...
}

func (nsu *nodeStatusUpdater) updateNodeStatus(logger klog.Logger, nodeName types.NodeName, nodeObj *v1.Node, attachedVolumes []v1.AttachedVolume) error {
	node := nodeObj.DeepCopy()
	node.Status.VolumesAttached = attachedVolumes
	_, patchBytes, err := nodeutil.PatchNodeStatus(nsu.kubeClient.CoreV1(), nodeName, nodeObj, node)
	if err != nil {
		return err
	}
	logger.V(4).Info("Updating status for node succeeded", "node", klog.KObj(node), "patchBytes", patchBytes, "attachedVolumes", attachedVolumes)
	return nil
}
```
As a consequence, the asw is misled by the informer to believe that all volumes that should be attached to the node have been successfully attached. However, by inspecting the output of kubectl, it is evident that status.volumesAttached does not include the said volume. 

**TL; DR**

```shell
Below is the steps where the problem occurred, with the timeline proceeding from top to bottom.
1. kubectl delete pod
2. ad-controller watch pod deleted, call dsw.Deletepod()
3. ad-controller reconciler() every 100ms period: 
		3.1 asw.RemoveVolumeFromReportAsAttached()
		3.2 nodeStatusUpdater.UpdateNodeStatusForNode()  // remove volume from node status.volumesAttached
		3.3 attacherDetacher.DetachVolume()
4. kube-controller-manager recreate the pod, and the pod is scheduled back to the original node
5. ad-controller watch pod created, call dsw.AddPod()
		3.4 attacherDetacher.AttachVolume()
		3.5 nodeStatusUpdater.UpdateNodeStatuses() // nsu.nodeLister.get() get a cached node, with the volume has not been removed by step-3.2. Since the pod is scheduled to the original node, the volumesAttached of the node is the same as before(step-3.2), thus the update operation will do nothing.
```


### What did you expect to happen?

If nodeStatusUpdater.updateNodeStatus() patches nothing by comparing with the informer's nodes, there should be a retry. 

### How can we reproduce it (as minimally and precisely as possible)?

it's hard to reproduce. But you may make it by these steps:
1. Use scheduler-extender to schedule pods onto fixed nodes
2. Increase apiserver load manually(let the informer sync delay)
3. Delete large mounts of statefulset pods with volumes for several times


### Anything else we need to know?

Our Kubernetes version is v1.13.3, but The problem still exists in the latest version

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"archive", BuildDate:"2021-05-10T07:53:22Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
N/A
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>