Repository: kubernetes
Issue: HPA stuck at maxReplicas even though metric under target
Labels: kind/bug,sig/autoscaling,needs-triage

### What happened?

HPA does not reduce `Deployment` replica count even though resource metric is below target. It is stuck at `maxReplicas`.

### What did you expect to happen?

Deployment replica count should be reduced.

### How can we reproduce it (as minimally and precisely as possible)?

We can see multiple examples in our clusters but not sure how to reproduce it exactly.

Here's some relevant `kubectl` output:

```console
$ kubectl top pod -l app=application-one,country=ar
NAME                                                  CPU(cores)   MEMORY(bytes)
application-one-ar-76fd9bc76d-25wnr                   35m          909Mi
application-one-ar-76fd9bc76d-4d72r                   42m          778Mi
application-one-ar-76fd9bc76d-6pt7r                   35m          1189Mi
application-one-ar-76fd9bc76d-6z2mr                   29m          793Mi
application-one-ar-76fd9bc76d-cv6r9                   29m          837Mi
application-one-ar-76fd9bc76d-hrpd9                   32m          824Mi
application-one-ar-76fd9bc76d-mrgt8                   45m          1180Mi
application-one-ar-76fd9bc76d-qwjbs                   43m          1186Mi
application-one-ar-76fd9bc76d-sqf5h                   41m          797Mi
application-one-ar-76fd9bc76d-tlr6k                   39m          920Mi
application-one-collect-metrics-ar-7df9868cbf-h4db8   7m           595Mi


$ kubectl get deployment application-one-ar
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
application-one-ar   10/10   10           10          565d

$ kubectl get hpa application-one-ar
NAME                      REFERENCE                  TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
application-one-ar   Deployment/application-one-ar   20%/70%   3         10        10         565d

$ kubectl get pod -l app=application-one,country=ar
NAME                                             READY   STATUS      RESTARTS   AGE
application-one-ar-76fd9bc76d-25wnr              1/1     Running     0          3h56m
application-one-ar-76fd9bc76d-4d72r              1/1     Running     0          147m
application-one-ar-76fd9bc76d-6pt7r              1/1     Running     0          4d
application-one-ar-76fd9bc76d-6z2mr              1/1     Running     0          147m
application-one-ar-76fd9bc76d-cv6r9              1/1     Running     0          176m
application-one-ar-76fd9bc76d-hrpd9              1/1     Running     0          147m
application-one-ar-76fd9bc76d-mrgt8              1/1     Running     0          3d4h
application-one-ar-76fd9bc76d-qwjbs              1/1     Running     0          4d
application-one-ar-76fd9bc76d-sqf5h              1/1     Running     0          146m
application-one-ar-76fd9bc76d-tlr6k              1/1     Running     0          3h56m
application-one-clean-up-ar-28257720-96544       0/1     Completed   0          2d5h
application-one-clean-up-ar-28259160-zj2f4       0/1     Completed   0          29h
application-one-clean-up-ar-28260600-687q4       0/1     Completed   0          5h19m
application-one-xxxx-ar-7df9868cbf-h4db8         1/1     Running     0          2d11h
application-one-yyyyy-ar-28240560-5vtwj          0/1     Completed   0          14d
application-one-zzzzz-ar-28250640-sr28d          0/1     Completed   0          7d3h
```

The `Deployment`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "115"
    meta.helm.sh/release-name: application-one-ar
    meta.helm.sh/release-namespace: default
  labels:
    app: application-one
    app.kubernetes.io/managed-by: Helm
    country: ar
    custom_app: application-one
    custom_country: ar
    custom_env: production
    custom_region: cloud-region-one
    environment: production
  name: application-one-ar
  namespace: default
spec:
  minReadySeconds: 10
  progressDeadlineSeconds: 300
  replicas: 10
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: application-one
      country: ar
  strategy:
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2023-06-25T14:57:33+02:00"
        prometheus.io/path: /prometheus
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: application-one
        country: ar
        custom_app: application-one
        custom_country: ar
        custom_env: production
        custom_region: cloud-region-one
        environment: production
        module: web
    spec:
      containers:
      - env: # removed
        imagePullPolicy: Always
        name: application-one-ar-production
        ports:
        - containerPort: 8080
          protocol: TCP
        readinessProbe:
          failureThreshold: 6
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "3"
            memory: 1536M
          requests:
            cpu: 170m
            memory: 1536M
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30

status:
  availableReplicas: 10
  conditions:
  - lastTransitionTime: "2022-03-09T12:49:18Z"
    lastUpdateTime: "2023-09-21T14:24:09Z"
    message: ReplicaSet "application-one-ar-76fd9bc76d" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2023-09-25T12:54:20Z"
    lastUpdateTime: "2023-09-25T12:54:20Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 8975
  readyReplicas: 10
  replicas: 10
  updatedReplicas: 10
```

The `HorizontalPodAutoscaler`:

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    meta.helm.sh/release-name: application-one-ar
    meta.helm.sh/release-namespace: default
  labels:
    app: application-one
    app.kubernetes.io/managed-by: Helm
    country: ar
    custom_app: application-one
    custom_country: ar
    custom_env: production
    custom_region: cloud-region-one
    environment: production
  name: application-one-ar
  namespace: default
spec:
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 70
        type: Utilization
    type: Resource
  minReplicas: 3
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: application-one-ar
status:
  conditions:
  - lastTransitionTime: "2022-03-09T12:49:33Z"
    message: recommended size matches current size
    reason: ReadyForNewScale
    status: "True"
    type: AbleToScale
  - lastTransitionTime: "2023-09-21T14:22:44Z"
    message: the HPA was able to successfully calculate a replica count from cpu resource
      utilization (percentage of request)
    reason: ValidMetricFound
    status: "True"
    type: ScalingActive
  - lastTransitionTime: "2023-09-25T10:52:44Z"
    message: the desired count is within the acceptable range
    reason: DesiredWithinRange
    status: "False"
    type: ScalingLimited
  currentMetrics:
  - resource:
      current:
        averageUtilization: 21
        averageValue: 33m
      name: cpu
    type: Resource
  currentReplicas: 10
  desiredReplicas: 10
  lastScaleTime: "2023-09-19T10:49:13Z"
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:21:56Z", GoVersion:"go1.18.4", Compiler:"gc", Platform:"darwin/arm64"}

Kustomize Version: v4.5.4

Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.12-eks-2d98532", GitCommit:"0aa16cf4fac4da27b9e9e9ba570b990867f6a3d8", GitTreeState:"clean", BuildDate:"2023-07-28T16:52:04Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
AWS EKS
</details>


### OS version

_No response_

### Install tools

_No response_

### Container runtime (CRI) and version (if applicable)

<details>
containerd://1.6.19
</details>

### Related plugins (CNI, CSI, ...) and versions (if applicable)

_No response_