Repository: kubernetes
Issue: Conflicting topologySpreadConstraints, podManagementPolicy: OrderedReady  and PVCs can lead to unschedulable pods in StatefulSets
Labels: kind/bug,sig/scheduling,needs-triage

### What happened?

I honestly don't know if this is a real bug or not, since it seems like the configuration conflicts with itself. I thought I'd file a bug and see what sig-scheduling says, since the behaviour isn't obvious to the user until the situation occurs.

With a StatefulSet it's possible to end up in a situation (after an outage) where some of these Pods are unable to schedule, if the StatefulSet has the following rules:
1. 5 replicas
2. A volumeClaimTemplates
3. Across 3 zones
4. `podManagementPolicy: OrderedReady`
5. A topologySpreadConstraints matching `topologyKey: topology.kubernetes.io/zone` with `maxSkew: 1`

Example StatefulSet below.

When deploying a StatefulSet with this configuration, if 2 Pods are lost, one in the single-pod-zone and another Pod with a lower number, then the StatefulSet is stuck can't replace those Pods due to a conflict of scheduling rules.

Let me try explain with some diagrams.

---

After applying the example StatefulSet, the Pods land in a configuration as such:
![image](https://github.com/user-attachments/assets/d039a682-1982-4272-a0b3-dc201ef4f365)
(Note, each Pod also has a related PVC with it)

This makes sense with all the rules configured and all Pods are healthy.

When `pod-1` and `pod-0` are deleted (due to node failures) we end up with this situation:
![image](https://github.com/user-attachments/assets/3c3e74e2-5404-4b58-bf98-a02246e0d7dd)

The StatefulSet Controller will then create `pod-0` (since `OrderedReady` is configured), but it will try be placed into the `Zone 2` zone (Since we have a maxSkew 1 topologySpreadConstraints zone policy).

However, the PVC for `pod-0` is in Zone 1, meaning that the Pod can never be scheduled. This stops the StatefulSet controller from continuing to `pod-1`.

It seems that the only way to fix this is to temporarily set `maxSkew` to 2 for the topologySpreadConstraints zone policy, or to delete the PVC in Zone 1

### What did you expect to happen?

It's difficult to way what I expect to happen, because the only way to schedule a Pod is to override the rule that I set as the operator.

Either schedule `pod-0` in the correct zone (bypassing the topologySpreadConstraints zone policy) 
Or schedule `pod-1` first (bypassing `OrderedReady`.

May be this situation is an exception for `OrderedReady`, since the StatefulSet isn't a new one, and is only recovering from a failure.

### How can we reproduce it (as minimally and precisely as possible)?

```
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: testing
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 5
  revisionHistoryLimit: 4
  selector:
    matchLabels:
      app.kubernetes.io/name: testing
  serviceName: ""
  template:
    metadata:
      labels:
        app.kubernetes.io/name: testing
    spec:
      automountServiceAccountToken: true
      containers:
      - image: nginx
        imagePullPolicy: IfNotPresent
        name: nginx
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /mount
          name: testing-pvc
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: testing
        maxSkew: 1
        nodeTaintsPolicy: Honor
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: testing
        maxSkew: 1
        nodeTaintsPolicy: Honor
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: testing-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 3Gi
      volumeMode: Filesystem
    status:
      phase: Pending
```

### Anything else we need to know?

Some context of the situation we're in:
1. We're using Ordered Ready, since the [vault-operator](https://github.com/bank-vaults/vault-operator/blob/v1.22.3/pkg/controller/vault/vault_controller.go#L1344-L1347) we use has that hardcoded
2. We want 5 replicas of Vault, due to the [recommendation from Hashicorp](https://developer.hashicorp.com/vault/docs/internals/integrated-storage#quorum-size-and-failure-tolerance).

I imagine if this isn't a Kubernetes bug, I should ask the vault-operator project to allow configurable `podManagementPolicy`, to avoid this situation from happening, however, I assume other users have similar constraints, and somehow fixing it in Kubernetes may be useful.

Some changes we considered:
1. Increasing replicas to 6. This should solve the problem, but conflicts with Hashicorp's suggestion of [running 5 replicas](https://developer.hashicorp.com/vault/docs/internals/integrated-storage#quorum-size-and-failure-tolerance).
2. Change maxSkew to 2. This actually helps un-stuck the pods, but on initial deployment it won't guarantee even spread across all AZs.

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.30.2```

</details>


### Cloud provider

<details>
AWS via kOps
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>

/sig scheduling