Repository: kubernetes
Issue: Scheduler topologySpreadConstraint to account for device plugin requests
Labels: sig/scheduling,kind/feature,needs-triage

### What happened?

In my cluster, I have two separate nodegroups:
- "regular" set of worker nodes that span 3 Availability Zones (`topology.kubernetes.io/zone`)
- gpu nodes with nvidia device plugin (Allocatable: `nvidia.com/gpu`) that span 2 Availability Zones

I have a Deployment with the following PodSpec:
```yaml
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: main

        resources:
          limits:
            cpu: 3500m
            memory: 16G
            nvidia.com/gpu: "1"
          requests:
            cpu: 200m
            memory: 5G
            nvidia.com/gpu: "1"

      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: foo
            app.kubernetes.io/instance: bar
        matchLabelKeys:
        - pod-template-hash
        maxSkew: 1
        nodeTaintsPolicy: Honor
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
```

Given that I have **not** defined `nodeSelector` or `affinity.nodeAffinity`, scheduler takes all available nodes into account when calculating the min topology domains. This leads to a failed scheduling of one of the pods.

```
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2025-01-22T13:56:55Z"
    lastUpdateTime: "2025-01-22T14:22:01Z"
    message: ReplicaSet "REDACATED-dfc668c69"
      has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2025-01-22T15:17:02Z"
    lastUpdateTime: "2025-01-22T15:17:02Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  observedGeneration: 55
  readyReplicas: 2
  replicas: 3
  unavailableReplicas: 1
  updatedReplicas: 3
```

This behavior matches the [topologySpreadConstraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) documentation

>nodeAffinityPolicy indicates how we will treat Pod's nodeAffinity/nodeSelector when calculating pod topology spread skew. Options are:
Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations.

### What did you expect to happen?

Given that the workload is requesting  `nvidia.com/gpu: "1"`, scheduler should only consider nodes that have Allocatable ` nvidia.com/gpu` when calculating topology domain.

### How can we reproduce it (as minimally and precisely as possible)?

Deployment

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo-bar
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: foo
      app.kubernetes.io/instance: bar
  template:
    metadata:
      labels:
        app.kubernetes.io/name: foo
        app.kubernetes.io/instance: bar
    spec:
      containers:
      - name: main
        image: nginx:1.14.2
        resources:
          limits:
            cpu: 1
            memory: 2G
            nvidia.com/gpu: "1"
          requests:
            cpu: 100m
            memory: 1G
            nvidia.com/gpu: "1"
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: foo
            app.kubernetes.io/instance: bar
        matchLabelKeys:
        - pod-template-hash
        maxSkew: 1
        nodeTaintsPolicy: Honor
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
Client Version: v1.30.5
Server Version: v1.30.6
```

</details>


### Cloud provider

<details>
aws
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy


$ uname -a
Linux ip-10-72-XXX-XXX.REDACTED 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>
CONTAINERD_VERSION=1.7.24
CRITOOLS_VERSION=1.30.1
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
aws vpc cni v1.19.0

CNI_VERSION=1.6.2
</details>