Repository: kubernetes
Issue: uneven Pods spreading across domains
Labels: sig/scheduling,kind/feature,needs-triage

## What 

Implement an uneven Pod spreading constraints between domains. Like domainA:domainB = x:y.

This requires us to introduce a new field to `topologySpreadConstraints` (name it `domains` for now). 
When `domains` field is non-empty, `maxSkew` should be nil. (vise-versa)

```yaml
- topologySpreadConstraints:
    # ...
    domains:
      - topologyLabel: "domainA"
        weight: 7
      - topologyLabel: "domainB"
        weight: 3
```

## Usecases

### Usecase1: spread Pods between zones

```yaml
- topologySpreadConstraints:
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
       matchLabels:
         app: foo
    topologyKey: "zone"
    domains:
      - topologyLabel: "zonea"
        weight: 7
      - topologyLabel: "zoneb"
        weight: 3
```

We have a DB in zoneA, so we want to schedule the Pods mostly to zoneA for better latency.
But, want to schedule smaller part of them to zone-b in case of zone failure.

### Usecases2: spread Pods for on-demand/spot

```yaml
- topologySpreadConstraints:
    labelSelector:
       matchLabels:
         app: foo
    topologyKey: "node-availability"
    domains:
      - topologyLabel: "on-demand"
        weight: 7
      - topologyLabel: "spot"
        weight: 3
```

So, we want to keep a large part of Pods running in on-demand Nodes because the spot instance might be deleted suddenly.
But, we're greedy, we want to keep some Pods in the spot Nodes for sure for the cost saving.

### Usecase3: gradual infrastructure migration

```yaml
- topologySpreadConstraints:
    labelSelector:
       matchLabels:
         app: foo
    topologyKey: "nodepool"
    domains:
      - topologyLabel: "old-nodepool"
        weight: 9 # will be reduced gradually
      - topologyLabel: "new-nodepool"
        weight: 1 # will be increased gradually
```

When we migrate workloads from one domain to another, we want to do it gradually.

## Why
 
Our company has one huge cluster to accommodate tons of Pods.

Currently, in order to achieve such uneven spreading, we need to choose either:
- create multiple deployments (+ HPA etc), one per domain. And somehow keep them to a specific ratio via some custom autoscaling solution.
  - It's hard to prepare multiple deployments in the first place when doing that for many services.(the usecase 2 / 3)
- use one deployment, but create the mutating webhook to add the required nodeaffinity to each Pod based on the current number of Pods in each domain. Also, descheduling should be done somehow.
- EDIT: Or follow [a similar way of Karpenter](https://github.com/kubernetes/kubernetes/issues/121211#issuecomment-1761840082). 

All are not simple solutions, which require engineering effort a lot and could cause serious problems in case of bugs.

Also, again we're greedy, we want to have a fallback mechanism like https://github.com/kubernetes/enhancements/issues/3990 when it comes to usecase 2. So, there could be a stockout problem in spot instance↓, and have to fallback to on-demand if necessary. That is something very hard to achieve with a custom solution, but can be _hopefully_ done with this proposal + the enhancement mentioned earlier.

> Spot VMs have no availability guarantees. Design your systems under the assumption that GKE might reclaim any or all your Spot VMs at any time, with no guarantee of when new instances become available.
https://cloud.google.com/kubernetes-engine/docs/concepts/spot-vms

/kind feature
/sig scheduling
/assign @sanposhiho