Repository: kubernetes
Issue: Zone-aware down scaling behavior
Labels: sig/scheduling,kind/feature,needs-triage

### What happened?

During scale-in we have an interesting zone unbalance behavior for hpa workloads.

We use topology spread `maxSkew: 1` in our deployment, but during the night
when scale-in happens we have sometimes only 1 pod for a workload spanning 3 zones
 with minreplica 9.

relevant deployment snippet:
```yaml
       topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              # This is kind of stupid, but would work for now. Ideally we should just stop filtering out the pods in
              # kube-system in our admitters, since we've never really had any issues with them.
              parent-resource-hash: 71556441059f2d033fb06b1e73df03598c7ecaa6
```

Impact: one pod has high CPU usage during low number of pods time:
![image](https://github.com/kubernetes/kubernetes/assets/50872/9ee227cb-7ff7-4618-ba45-1a6a7491edec)
The HPA data during the same period of time:
![image](https://github.com/kubernetes/kubernetes/assets/50872/e1f7d7e1-5d05-413b-a349-2504c32b0370)

We could track it back that the pod with high CPU usage is the only one in the affected zone and the rest of the pods are split into the other 2 zones.


We are now collecting more data to have better visibility. You can already see the "miss behavior" in the picture, which shows 2 zones have 7 pods and 1 zone has 5 pods:
![image](https://github.com/kubernetes/kubernetes/assets/50872/bb1536d5-4f17-435d-b8f7-36aa51b957b5)


### What did you expect to happen?

I expect that scale-in respects somehow topology spread `maxSkew: 1` behavior, so in the pods by zone split picture I would expect at most `+/- 1` pod by zone, but I see 2.

### How can we reproduce it (as minimally and precisely as possible)?

As far as I know scale-in happens by subresource API call to replicaset, so I guess you can reproduce it by manipulating a test that has a deployment with a similar config like shown above. You would need to have maybe 9 pods with an even split by zone. The pods in one zone should have a modified creation(?) timestamp, such that these are the newest that will be terminated first at scale-in. After applied modification you can scale to 7 and you should observe the zone split from 3,3,3 going down to 3,3,1

### Anything else we need to know?

Deployment for reference:
https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/manifests/skipper/deployment.yaml#L73-L81

There might be some related issues, but none which were precise enough to show the impact.
https://github.com/kubernetes/kubernetes/issues/107598
https://github.com/kubernetes/kubernetes/issues/115911
https://github.com/kubernetes/kubernetes/issues/123541
https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/2255-pod-cost

### Kubernetes version

<details>

```console
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.11", GitCommit:"b9e2ad67ad146db566be5a6db140d47e52c8adb2", GitTreeState:"clean", BuildDate:"2024-02-14T10:40:40Z", GoVersion:"go1.21.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.12", GitCommit:"12031002905c0410706974560cbdf2dad9278919", GitTreeState:"clean", BuildDate:"2024-03-15T02:06:14Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/arm64"}
```

</details>


### Cloud provider

<details>
AWS
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -a
Linux ip-172-31-4-205 6.5.0-1014-aws #14~22.04.1-Ubuntu SMP Thu Feb 15 19:20:45 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
```

</details>


### Install tools

<details>
https://github.com/zalando-incubator/kubernetes-on-aws
https://github.com/zalando-incubator/cluster-lifecycle-manager
</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>