Repository: kubernetes
Issue: Pod Informer Update with updated Pod UID can lead to attachdetach controller desired_state_of_world cache inaccuracy
Labels: kind/bug,sig/storage,sig/node,sig/api-machinery,triage/accepted

### What happened?

Volume force detached due to incorrect attachdetach controller desired_state_of_world 

The attach-detach controller reconciles between two caches, actual_state_of_world and desired_state_of_world [Design](https://github.com/kubernetes/kubernetes/issues/20262). The desired state of the world is updated by a [pod informer](https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/attach_detach_controller.go#L196).

As discovered in #83810, a pod delete and create event with the same pod name can be delivered as an update event by the pod informer. The attachdetach controller treats pod add/update events the same, but only updates the desired_state_of_world if a [pod does not exist](https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/cache/desired_state_of_world.go#L257). This means that if the event is an add/delete event (delivered as an update event with a updated pod UID), the desired_state_of_world will not be updated with the latest UID. If this happens, the state of the desired_state_of_world is inaccurate, as the pod UID is stale.

Periodically, the desired_state_of_world_populator reconciles all pods in its cache. If it detects a pod UID mismatch with the current pod UID in the informer, the pod is [removed from desired_state_of_world](https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/populator/desired_state_of_world_populator.go#L154). Once removed, the pod can be force detached by the reconciler, if it has not existed in the desired_state_of_world for [the timeout period, 6 minutes](https://github.com/kubernetes/kubernetes/blob/86571236e32f5e5b94c4336407878beb7d41a87a/pkg/controller/volume/attachdetach/reconciler/reconciler.go#L265). Note that prior to #110721, this was done for all nodes, not just unhealthy nodes.

The attachdetach controller has a safeguard to reconcile this type of inaccuracy. In desired_state_of_world_populator, there are two loops
1. The [findAndRemoveDeletedPods()](https://github.com/kubernetes/kubernetes/blob/release-1.22/pkg/controller/volume/attachdetach/populator/desired_state_of_world_populator.go#L115) loop. This runs at a period of every 1 minute
2. The [findAndAddActivePods()](https://github.com/kubernetes/kubernetes/blob/release-1.22/pkg/controller/volume/attachdetach/populator/desired_state_of_world_populator.go#L175) loop. This runs at a period of every 3 minutes.

In this scenario, if findAndRemoveDeletedPods() removes a pod, and findAndAddActivePods() runs directly after, desired_state_of_world only stays in an inaccurate state for a brief period of time. However, if the desired_state_of_world_populator loop has some latency in processing (eg: large number of pods, CPU throttling), it's possible for more than 6 minutes to pass between when the pod is removed from desired_state_of_world and when the pod is added back in the findAndAddActivePods() loop.

### What did you expect to happen?

Expect a volume to not be force detached, if the pod that consumes the volume is quickly deleted/re-created.

### How can we reproduce it (as minimally and precisely as possible)?

Consistently reproducing is tricky, as it requires a short enough time delta to allow an delete/add to be merged into an update event and constraining the kube-controller-manager. However the general flow is:
1. Create a pod that uses a volume
2. Quickly delete/add the pod. This may be merged into an update event by the Pod Informer
3. If the DSW gets corrupted in findAndRemoveDeletedPods(), and findAndAddActivePods() doesn't run before the 6 minute window, attachdetach reconciler loop should force detach the volume. The kube-controller-manager may need to be artificially constrained (limit CPU request) for this to be reproduced.

### Anything else we need to know?

This is very similar to the issue fixed in kube-scheduler here: https://github.com/kubernetes/kubernetes/pull/91126

### Kubernetes version

1.22

### Cloud provider

<details>
GKE
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>