Repository: kubernetes
Issue: TopologySpreadConstraints - Scaling Pods during rolling updates can cause pods to be stuck in Pending
Labels: kind/bug,sig/scheduling,lifecycle/stale,needs-triage

### What happened?

We have a stateful application that requires pods to be evenly spread across zones. Each pod required a persistent volume (AWS EBS volume) to store data. To deploy this application we use a StatefulSet with the following Pod Topology Spread Constraint:

```
topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            name: my-app
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
```

During rolling updates, we update multiple pods from the same zone. For that, we use a custom controller ([zone-aware-controllers-for-k8s](https://github.com/aws/zone-aware-controllers-for-k8s)), but it would be possible to archive something similar using the [StatefulSets’ Max Unavailable](https://kubernetes.io/blog/2022/05/27/maxunavailable-for-statefulset/) alpha feature available on Kubernetes 1.24+.

When the application happens to be scaled (e.g manually or by the HPA) during a rolling updated, we have noticed old Pods getting stuck on Pending because of topology spread constraints + volume node affinity conflict: 

```
 Warning  FailedScheduling   40s   default-scheduler   0/34 nodes are available: 26 node(s) didn't match Pod's node affinity/selector, 4 node(s) didn't match pod topology spread constraints, 4 node(s) had volume node affinity conflict.
```

Example: 

Initial state (zone : pods).
- ZoneA : [PA1, PA2, PA3, PA4]
- ZoneB : [PB1, PB2, PB3, PB4]
- ZoneC : [PC1, PC2, PC3, PC4]

Assuming that we update 2 pods in Zone A:
- ZoneA : [PA1, PA2, **PA3(Terminating)**, **PA4(Terminating)**]
- ZoneB : [PB1, PB2, PB3, PB4]
- ZoneC : [PC1, PC2, PC3, PC4]

Now, scale the number of pods by the same number of pods getting updated. Noticed that new replicas are placed on Zone A first (scheduler seems to ignore the pods that are getting terminated/updated).
- ZoneA : [PA1, PA2, PA3(Terminating), PA4(Terminating), **PA5**, **PA6**]
- ZoneB : [PB1, PB2, PB3, PB4]
- ZoneC : [PC1, PC2, PC3, PC4]

In the end, some pods that were getting updated will be stuck in Pending as they can't be rescheduled into another zone because of the volume node affinity conflict (those pods can only be scheduled on ZoneA which is the zone in which their EBS volumes exists). 
- ZoneA : [PA1, PA2, PA3, **PA4(Pending)**, PA5, PA6]
- ZoneB : [PB1, PB2, PB3, PB4]
- ZoneC : [PC1, PC2, PC3, PC4]

Note that in this specific example PA3 will be scheduled on ZoneA as it satisfy `maxSkew = 1`, but PA4 will get stuck.

### What did you expect to happen?

Statefulset Pods getting restarted/updated are considered when scheduling new Pods.

### How can we reproduce it (as minimally and precisely as possible)?

The easier way to reproduce this is probably by deleting multiple pods from the same zone and, while new pods are getting recreated, try to scale the number of replicas. For example: 

- Create a Statefulset with PersistentVolume and the above TopologySpreadConstraint. 
- Scale the Statefulset so you have multiple pods per zone (e.g 12 pods distributed in 3 zones)
- Delete multiple pods from a single zone
- While pods are getting recreated, try to scale the number of replicas
- Notice that all (or majority) of new replicas are placed on the zone in which pods were deleted
- Notice that old pods are stuck in Pending

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:17:57Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.16-eks-ffeb93d", GitCommit:"52e500d139bdef42fbc4540c357f0565c7867a81", GitTreeState:"clean", BuildDate:"2022-11-29T18:41:42Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
AWS EKS
</details>


### OS version

<details>

```console
N/A
```

</details>


### Install tools

<details>
N/A
</details>


### Container runtime (CRI) and version (if applicable)

<details>
N/A
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
[Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)
</details>