Repository: kubernetes
Issue: HPA scales up despite utilization being under target
Labels: kind/bug,sig/autoscaling,needs-triage

### What happened?

We use Argo Rollouts to perform canary deployments of our services. During a canary deployment, new pods are brought up (the canary pods) which are included in the [Status](https://github.com/kubernetes/kubernetes/blob/5ba2b78eae18645744b51d94d279582bdcccec23/pkg/apis/autoscaling/types.go#L51) of the Rollout's scale subresource. When HPA is configured to scale on a metric with a high utilization ratio (generally memory), this results in HPA scaling out, despite the fact that the utilization is under the target. 

This seems to be a result of the behaviour of the [replica calculator](https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/replica_calculator.go#L65) where:
- the recommended replicas is a simple function of utilizationRatio and the total number of pods returned by the selector
- missing pods with a utilization ratio of below 1 are assumed to be because of a "scale down" and are treated as if they are [consuming 100% of resources](https://github.com/kubernetes/kubernetes/blob/5ba2b78eae18645744b51d94d279582bdcccec23/pkg/controller/podautoscaler/replica_calculator.go#L108-L113) (whereas in this case we are scaling out during a rollout)

In addition, the utilization ratio is never checked prior to updating prior to HPA increasing the desired replicas leading to [incorrect/misleading log messages](https://github.com/kubernetes/kubernetes/blob/5ba2b78eae18645744b51d94d279582bdcccec23/pkg/controller/podautoscaler/horizontal.go#L857-L859) like "New size: X; reason: memory resource utilization (percentage of request) above target". 

Further details can be found in issues logged for Argo Rollouts [here](https://github.com/argoproj/argo-rollouts/issues/2857) and [here](https://github.com/argoproj/argo-rollouts/issues/3849).

### What did you expect to happen?

HPA should not scale up when utilization is below target (I.e. when `utilizationRatio < 1`)

### How can we reproduce it (as minimally and precisely as possible)?

The issue can be reproduced by simulating the behaviour of Argo Rollouts and creating a 2nd `ReplicaSet` with selectors that match an existing `Deployment`. For example:

Create base `Deployment` and `HorizontalPodAutoscaler` resources:  
``` yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: nginx
          name: nginx
          resources:
            requests:
              memory: "10Mi"
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx
  namespace: test
spec:
  maxReplicas: 50
  minReplicas: 3
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 50
```

Wait to ensure that metrics are available and HPA is stable at 3 pods.
``` bash
$ kubectl get pods,hpa
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-7fb4f6d65f-jgdzm   1/1     Running   0          30s
pod/nginx-7fb4f6d65f-mtnxv   1/1     Running   0          30s
pod/nginx-7fb4f6d65f-tc6c8   1/1     Running   0          30s

NAME                                        REFERENCE          TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/nginx   Deployment/nginx   memory: 42%/50%   3         50        3          30s
```

Create a 2nd replica set: 
``` yaml
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx-canary
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
      role: canary
  template:
    metadata:
      labels:
        app: nginx
        role: canary
    spec:
      containers:
        - image: nginx
          name: nginx
          resources:
            requests:
              memory: "10Mi"
```

Observe that HPA will scale the existing `Deployment`:
``` bash
$ kubectl get pods,hpa                                                                                     
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-7fb4f6d65f-2hh7f   1/1     Running   0          2m36s
pod/nginx-7fb4f6d65f-6lfr5   1/1     Running   0          2m6s
pod/nginx-7fb4f6d65f-9wgfn   1/1     Running   0          6s
pod/nginx-7fb4f6d65f-gpzp9   1/1     Running   0          2m6s
pod/nginx-7fb4f6d65f-hdsff   1/1     Running   0          3m6s
pod/nginx-7fb4f6d65f-j24kp   1/1     Running   0          3m6s
pod/nginx-7fb4f6d65f-jgdzm   1/1     Running   0          4m36s
pod/nginx-7fb4f6d65f-kxbvz   1/1     Running   0          66s
pod/nginx-7fb4f6d65f-l4rzr   1/1     Running   0          2m36s
pod/nginx-7fb4f6d65f-mtnxv   1/1     Running   0          4m36s
pod/nginx-7fb4f6d65f-qvpqx   1/1     Running   0          3m6s
pod/nginx-7fb4f6d65f-rq2r8   1/1     Running   0          96s
pod/nginx-7fb4f6d65f-s52hj   1/1     Running   0          66s
pod/nginx-7fb4f6d65f-tc6c8   1/1     Running   0          4m36s
pod/nginx-7fb4f6d65f-zwpfc   1/1     Running   0          36s
pod/nginx-canary-2prb6       1/1     Running   0          3m41s
pod/nginx-canary-9l5rc       1/1     Running   0          3m41s
pod/nginx-canary-g79tk       1/1     Running   0          3m41s

NAME                                        REFERENCE          TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/nginx   Deployment/nginx   memory: 42%/50%   3         50        14         4m36s
```

Check events on HPA to see details on why scaling occurred:
```
$ kubectl describe hpa nginx
Name:                                                     nginx
Namespace:                                                test
Labels:                                                   <none>
Annotations:                                              <none>
CreationTimestamp:                                        Mon, 16 Dec 2024 10:20:04 +0100
Reference:                                                Deployment/nginx
Metrics:                                                  ( current / target )
  resource memory on pods  (as a percentage of request):  42% (4503324444m) / 50%
Min replicas:                                             3
Max replicas:                                             50
Deployment pods:                                          8 current / 8 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type     Reason                   Age                    From                       Message
  ----     ------                   ----                   ----                       -------
  Warning  FailedGetResourceMetric  4m35s (x3 over 4m46s)  horizontal-pod-autoscaler  unable to get metric memory: no metrics returned from resource metrics API
  Normal   SuccessfulRescale        3m31s                  horizontal-pod-autoscaler  New size: 6; reason: memory resource utilization (percentage of request) above target
  Normal   SuccessfulRescale        3m1s                   horizontal-pod-autoscaler  New size: 8; reason: memory resource utilization (percentage of request) above target
  Normal   SuccessfulRescale        2m31s                  horizontal-pod-autoscaler  New size: 10; reason: memory resource utilization (percentage of request) above target
  Normal   SuccessfulRescale        2m1s                   horizontal-pod-autoscaler  New size: 11; reason: memory resource utilization (percentage of request) above target
  Normal   SuccessfulRescale        91s                    horizontal-pod-autoscaler  New size: 13; reason: memory resource utilization (percentage of request) above target
  Normal   SuccessfulRescale        61s                    horizontal-pod-autoscaler  New size: 14; reason: memory resource utilization (percentage of request) above target
  Normal   SuccessfulRescale        31s                    horizontal-pod-autoscaler  New size: 15; reason: memory resource utilization (percentage of request) above target
  Normal   SuccessfulRescale        1s                     horizontal-pod-autoscaler  New size: 16; reason: memory resource utilization (percentage of request) above target
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.31.3
Kustomize Version: v5.4.2
Server Version: v1.30.5-gke.1443001
```

</details>


### Cloud provider

<details>
GKE
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>