Kubernetes: HorizontalPodAutoscaler — an overview with examples
Tags: monitoring, kubernetes, tutorial, todayilearned
Published: 2020-08-12 11:44:23

---
title: Kubernetes: HorizontalPodAutoscaler — an overview with examples
published: true
date: 2020-08-12 06:42:45 UTC
tags: monitoring,kubernetes,tutorial,todayilearned
canonical_url: https://itnext.io/kubernetes-horizontalpodautoscaler-an-overview-with-examples-51f0d520bfe6
---

### Kubernetes: HorizontalPodAutoscaler — an overview with examples

![](https://cdn-images-1.medium.com/max/130/0*_GX-UK17fVKwPoyT.png)

[Kubernetes HorizontalPodAutoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) automatically scales Kubernetes Pods under [`ReplicationController`](https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/), [`Deployment`](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), or [`ReplicaSet`](https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/) controllers basing on its CPU, memory, or other metrics.

It was shortly discussed in the [Kubernetes: running metrics-server in AWS EKS for a Kubernetes Pod AutoScaler](https://rtfm.co.ua/en/kubernetes-running-metrics-server-in-aws-eks-for-a-kubernetes-pod-autoscaler/) post, now let’s go deeper to check all options available for scaling.

For HPA you can use three API types:

- `metrics.k8s.io`: default metrics, basically provided by the [metrics-server](https://github.com/kubernetes-incubator/metrics-server)
- [`custom.metrics.k8s.io`](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md): metrics, provided by adapters from inside of a cluster, for example - [Microsoft Azure Adapter](https://github.com/Azure/azure-k8s-metrics-adapter), [Google Stackdriver](https://github.com/GoogleCloudPlatform/k8s-stackdriver), [Prometheus Adapter](https://github.com/directxman12/k8s-prometheus-adapter) (the Prometheus Adapter will be used in this post later), check the full list [here>>>](https://github.com/kubernetes/metrics/blob/master/IMPLEMENTATIONS.md#custom-metrics-api)
- [`external.metrics.k8s.io`](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/external-metrics-api.md): similar to the Custom Metrics API, but metrics are provided by an external system, such as AWS CloudWatch

Documentation: [Support for metrics APIs](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-metrics-apis), and [Custom and external metrics for autoscaling workloads](https://cloud.google.com/kubernetes-engine/docs/concepts/custom-and-external-metrics).

Besides the HorizontalPodAutoscaler (HPA) you also can use [Vertical Pod Autoscaling (VPA)](https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler) and they can be used together although with some limitations, see [Horizontal Pod Autoscaling Limitations](https://cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler#limitations).

### Content

- [Create HorizontalPodAutoscaler](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/#Create_HorizontalPodAutoscaler)
- [Load testing HorizontalPodAutoscaler scaling](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/#Load_testing_HorizontalPodAutoscaler_scaling)
- [Multi-metrics scaling](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/#Multimetrics_scaling)
- [autoscaling API group versions](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/#autoscaling_API_group_versions)
- [Custom Metrics](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/#Custom_Metrics)
- [Memory metrics scaling](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/#Memory_metrics_scaling)
- [Application-based metrics scaling](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/#Applicationbased_metrics_scaling)
- [Kubernetes ServiceMonitor](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/#Kubernetes_ServiceMonitor)
- [Prometheus Adapter ConfigMap — seriesQuery and metricsQuery](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/#Prometheus_Adapter_ConfigMap_%E2%80%93_seriesQuery_and_metricsQuery)

### Create HorizontalPodAutoscaler

Let’s start with a simple HPA which will scale pods basing on CPU usage:

```
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-example
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-example
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 10
```

Here:

- `apiVersion: autoscaling/v1` - an API group `autoscaling`, pay attention to the API version, as in the `v1` at the time of writing, scaling was available by the CPU metrics only, thus memory and custom metrics can be used only with the API `v2beta2` (still, you can use v1 with annotations), see [API Object](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#api-object).
- `spec.scaleTargetRef`: specify for НРА which controller will be scaled (`ReplicationController`, `Deployment`, `ReplicaSet`), in this case, HPA will look for the Deployment object called _deployment-example_
- `spec.minReplicas`, `spec.maxReplicas`: minimal and maximum pods to be running by this HPA
- `targetCPUUtilizationPercentage`: CPU usage % from the [requests](https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits) when HPA will add or remove pods

Create it:

```
$ kubectl apply -f hpa-example.yaml
horizontalpodautoscaler.autoscaling/hpa-example created
```

Check:

```
$ kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-example Deployment/deployment-example <unknown>/10% 1 5 0 89s
```

Currently, its `TARGETS` has the _<unknown>_ value as there are no pods created yet, but metrics are already available:

```
$ kubectl get — raw “/apis/metrics.k8s.io/” | jq{
“kind”: “APIGroup”,
“apiVersion”: “v1”,
“name”: “metrics.k8s.io”,
“versions”: [
{
“groupVersion”: “metrics.k8s.io/v1beta1”,
“version”: “v1beta1”
}
],
“preferredVersion”: {
“groupVersion”: “metrics.k8s.io/v1beta1”,
“version”: “v1beta1”}
}
```

Add the _deployment-example_ `Deployment`:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-example
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      application: deployment-example
  template:
    metadata:
      labels:
        application: deployment-example
    spec: 
      containers:
      - name: deployment-example-pod
        image: nginx
        ports:
          - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
```

Here we defined Deployment which will spin up one pod with NINGX with `requests` for 100 millicores and 100 [mebibyte](https://en.wikipedia.org/wiki/Mebibyte) memory, see [Kubernetes best practices: Resource requests and limits](https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits).

Create it:

```
$ kubectl apply -f hpa-deployment-example.yaml
deployment.apps/deployment-example created
```

Check the HPA now:

```
$ kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-example Deployment/deployment-example 0%/10% 1 5 1 14m
```

Our НРА found the deployment and started checking its pods’ metrics.

Let’s check those metrics — find a pod:

```
$ kubectl get pod | grep example | cut -d “ “ -f 1
deployment-example-86c47f5897–2mzjd
```

And run the following API request:

```
$ kubectl get — raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/deployment-example-86c47f5897–2mzjd | jq
{
“kind”: “PodMetrics”,
“apiVersion”: “metrics.k8s.io/v1beta1”,
“metadata”: {
“name”: “deployment-example-86c47f5897–2mzjd”,
“namespace”: “default”,
“selfLink”: “/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/deployment-example-86c47f5897–2mzjd”,
“creationTimestamp”: “2020–08–07T10:41:21Z”
},
“timestamp”: “2020–08–07T10:40:39Z”,
“window”: “30s”,
“containers”: [
{
“name”: “deployment-example-pod”,
“usage”: {
“cpu”: “0”,
“memory”: “2496Ki”
}
}
]
}
```

CPU usage — zero, memory — 2 megabytes, let’s confirm with the `top`:

```
$ kubectl top pod deployment-example-86c47f5897–2mzjd
NAME CPU(cores) MEMORY(bytes)
deployment-example-86c47f5897–2mzjd 0m 2Mi
```

_“Alright, these guys_!” (с)

Okay  -  we got our metrics, we’ve created the HPA and deployment -  let’s go to see how the scaling will work here.

#### Load testing HorizontalPodAutoscaler scaling

For load testing, we can use the [loadimpact/loadgentest-wrk](https://hub.docker.com/r/loadimpact/loadgentest-wrk) utility image.

Now, run ports redirect from the local workstation to the pod with NGINX, as we didn’t add any LoadBalancer (see [Kubernetes: ClusterIP vs NodePort vs LoadBalancer, Services, and Ingress — an overview with examples](https://rtfm.co.ua/en/kubernetes-clusterip-vs-nodeport-vs-loadbalancer-services-and-ingress-an-overview-with-examples/)):

```
$ kubectl port-forward deployment-example-86c47f5897–2mzjd 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
```

Check resources once again:

```
$ kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-example Deployment/deployment-example 0%/10% 1 5 1 33m
```

0% CPU is used, one pod is running (`REPLICAS` _1_).

Run the test:

```
$ docker run — rm — net=host loadimpact/loadgentest-wrk -c 100 -t 100 -d 5m [http://127.0.0.1:8080](http://127.0.0.1:8080)
Running 5m test @ [http://127.0.0.1:8080](http://127.0.0.1:8080)
```

Here:

- open 100 connections using 100 threads
- run the test for 5 minutes

Check the pod:

```
$ kubectl top pod deployment-example-86c47f5897–2mzjd
NAME CPU(cores) MEMORY(bytes)
deployment-example-86c47f5897–2mzjd 49m 2Mi
```

CPU usage now 49mi, and in the requests, we've set the 10 milicpu limit - check the HPA:

```
$ kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-example Deployment/deployment-example 49%/10% 1 5 4 42m
```

`TARGETS` 49% from the 10% limit and our HPA started new pods - `REPLICAS` _4_:

```
$ kubectl get pod | grep example
deployment-example-86c47f5897–2mzjd 1/1 Running 0 31m
deployment-example-86c47f5897–4ntd4 1/1 Running 0 24s
deployment-example-86c47f5897-p7tc7 1/1 Running 0 8s
deployment-example-86c47f5897-q49gk 1/1 Running 0 24s
deployment-example-86c47f5897-zvdvz 1/1 Running 0 24s
```

### Multi-metrics scaling

Okay, we were able to scale by the CPU usage, but what if you want to scale by both CPU and memory usage?

Add another `Resource` with memory limit set to the same 10%:

```
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-example
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-example
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 10
  - type: Resource
    resource:
      name: memory 
      targetAverageUtilization: 10
```

#### autoscaling API group versions

Let’s go back to the API versions.

In the first manifest for our HPA, we’ve used the [`autoscaling/v1`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#horizontalpodautoscaler-v1-autoscaling) API which has the only `targetCPUUtilizationPercentage` parameter.

Check the [`autoscaling/v2beta1`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#metricspec-v2beta1-autoscaling)  -  now, it has the `metrics` field added which is the [`MetricSpec`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#metricspec-v2beta1-autoscaling) array which can hold for new fields - the `external`, `object`, `pods`, `resource`.

in its turn, the `resource` has the [`ResourceMetricSource`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#resourcemetricsource-v2beta1-autoscaling), which holds two fields - `targetAverageUtilization`, and `targetAverageValue`, which are used now in the metrics instead of the `targetCPUUtilizationPercentage`.

Apply the HPA update:

```
$ kubectl apply -f hpa-example.yaml
horizontalpodautoscaler.autoscaling/hpa-example configured
```

Check it:

```
$ kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-example Deployment/deployment-example 3%/10%, 0%/10% 1 5 2 126m
```

The `TARGETS` now displaying two metrics - CPU and memory.

It’s hard to make NGINX consume a lot of memory, so let’s go to see how much it uses now with the following kubectl command:

```
$ kubectl get — raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/deployment-example-c4d6f96db-jv6nm | jq ‘.containers[].usage.memory’
“2824Ki”
```

2 megabytes.

Let’s update our PHA once again and set a new limit, but at this time we’ll use raw values instead of the percent — _1024Ki,_ 1 megabyte using targetAverageUtilization instead of the previously used targetAverageUtilization:

```
...
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 10
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 1024Ki
```

Apply and check:

```
$ kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AG
hpa-example Deployment/deployment-example 2551808/1Mi, 0%/10% 1 5 3 2m8s
```

`REPLICAS` == 3 now, pods were scaled, check the value from the `TARGETS` - convert it to kilobytes:

```
$ echo 2551808/1024 | bc
2492
```

And check real memory usage:

```
$ kubectl get — raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/deployment-example-c4d6f96db-fldl2 | jq ‘.containers[].usage.memory’
“2496Ki”
```

2492 ~= 2496Ki.

Okay, so — we are able now to scale the `Deployment` by both CPU and memory usage.

### Custom Metrics

#### Memory metrics scaling

Apart from metrics provided by the API server and cAdvisor we can use any other metrics, for example - metrics, collected by Prometheus.

It can be metrics collected by a Cloudwatch exporter, Prometheus’ node_exporter, or metrics from an application.

Documention is [here>>>](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md).

As we are using Prometheus (see [Kubernetes: monitoring with Prometheus — exporters, a Service Discovery, and its roles](https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/) and [Kubernetes: a cluster’s monitoring with the Prometheus Operator](https://rtfm.co.ua/en/kubernetes-a-clusters-monitoring-with-the-prometheus-operator/) so let’s add its adapter:

![](https://cdn-images-1.medium.com/max/415/0*-tx4MNLtz4vhHf_d.png)

If you’ll try to access external or custom API endpoints now you’ll get the error:

```
$ kubectl get — raw /apis/custom.metrics.k8s.io/
Error from server (NotFound): the server could not find the requested resource

$ kubectl get — raw /apis/external.metrics.k8s.io/
Error from server (NotFound): the server could not find the requested resource
```

Install the adapter from the Helm chart:

```
$ helm install prometheus-adapter stable/prometheus-adapter
NAME: prometheus-adapter
LAST DEPLOYED: Sat Aug 8 13:27:36 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
prometheus-adapter has been deployed.
```

In a few minutes you should be able to list metrics using the following command(s):

```
$ kubectl get — raw /apis/custom.metrics.k8s.io/v1beta
```

Wait for a minute or two and check API again:

```
$ kubectl get — raw=”/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
“kind”: “APIResourceList”,
“apiVersion”: “v1”,
“groupVersion”: “custom.metrics.k8s.io/v1beta1”,
“resources”: []
}
```

Well, but why the `"resources":[]` is empty?

Check the adapter’s pod logs:

```
$ kubectl logs -f prometheus-adapter-b8945f4d8-q5t6x
I0808 10:45:47.706771 1 adapter.go:94] successfully using in-cluster auth
E0808 10:45:47.752737 1 provider.go:209] unable to update list of all metrics: unable to fetch metrics for query “{__name__=~"^container_.*”,container!=”POD”,namespace!=””,pod!=””}”: Get [http://prometheus.default.svc:9090/api/v1/series?match%5B%5D=%7B__name__%3D~%22%5Econtainer_.%2A%22%2Ccontainer%21%3D%22POD%22%2Cnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1596882347.736:](http://prometheus.default.svc:9090/api/v1/series?match%5B%5D=%7B __name__ %3D~%22%5Econtainer_.%2A%22%2Ccontainer%21%3D%22POD%22%2Cnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1596882347.736:) dial tcp: lookup prometheus.default.svc on 172.20.0.10:53: no such host
I0808 10:45:48.032873 1 serving.go:306] Generated self-signed cert (/tmp/cert/apiserver.crt, /tmp/cert/apiserver.key)
…
```

Here is the error:

> dial tcp: lookup prometheus.default.svc on 172.20.0.10:53: no such host

Let try to access our Prometheus Operator from the pod by its Service DNS name:

```
$ kubectl exec -ti deployment-example-c4d6f96db-fldl2 curl prometheus-prometheus-oper-prometheus.monitoring.svc.cluster.local:9090/metrics | head -5
HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile=”0"} 2.0078e-05
go_gc_duration_seconds{quantile=”0.25"} 3.8669e-05
go_gc_duration_seconds{quantile=”0.5"} 6.956e-05
```

Okay, we are able to reach it using the _prometheus-prometheus-oper-prometheus.monitoring.svc.cluster.local:9090_ URL.

Edit the adapter’s Deployment:

```
$ kubectl edit deploy prometheus-adapter
```

Update its `prometheus-url`:

```
...
    spec:
      affinity: {}
      containers:
      - args:
        - /adapter
        - --secure-port=6443
        - --cert-dir=/tmp/cert
        - --logtostderr=true
        - --prometheus-url=http://prometheus-prometheus-oper-prometheus.monitoring.svc.cluster.local:9090
...
```

Apply changes and check again:

```
$ kubectl get — raw “/apis/custom.metrics.k8s.io/v1beta1” | jq . |grep “pods/” | head -5
“name”: “pods/node_load15”,
“name”: “pods/go_memstats_next_gc_bytes”,
“name”: “pods/coredns_forward_request_duration_seconds_count”,
“name”: “pods/rest_client_requests”,
“name”: “pods/node_ipvs_incoming_bytes”,
```

Nice — we’ve got our metrics and can use them now in the HPA.

Check the API server for the `memory_usage_bytes` metric:

```
$ kubectl get — raw=”/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/memory_usage_bytes” | jq .
{
“kind”: “MetricValueList”,
“apiVersion”: “custom.metrics.k8s.io/v1beta1”,
“metadata”: {
“selfLink”: “/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/memory_usage_bytes”
},
“items”: [
{
“describedObject”: {
“kind”: “Pod”,
“namespace”: “default”,
“name”: “deployment-example-c4d6f96db-8tfnw”,
“apiVersion”: “/v1”
},
“metricName”: “memory_usage_bytes”,
“timestamp”: “2020–08–08T11:18:53Z”,
“value”: “11886592”,
“selector”: null
},
…
```

Update the НРА’s manifest:

```
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-example
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-example
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: memory_usage_bytes
      targetAverageValue: 1024000
```

Check the HPA’s values now:

```
$ kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-example Deployment/deployment-example 4694016/1Mi, 0%/10% 1 5 5 69m
```

Apply the latest changes:

```
$ kubectl apply -f hpa-example.yaml
horizontalpodautoscaler.autoscaling/hpa-example configured
```

Check again:

```
$ kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-example Deployment/deployment-example 11853824/1024k 1 5 1 16s
```

Still have 1 replica, check events:

```
…
43s Normal ScalingReplicaSet Deployment Scaled up replica set deployment-example-c4d6f96db to 1
16s Normal ScalingReplicaSet Deployment Scaled up replica set deployment-example-c4d6f96db to 4
1s Normal ScalingReplicaSet Deployment Scaled up replica set deployment-example-c4d6f96db to 5
16s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 4; reason: pods metric memory_usage_bytes above target
1s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 5; reason: pods metric memory_usage_bytes above target
…
```

And HPA again:

```
$ kubectl get hpa hpa-example
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-example Deployment/deployment-example 6996787200m/1024k 1 5 5 104s
```

Great — “It works!” ©

And it’s good for metrics which are already present in the cluster like the `memory_usage_bytes` by default collected by the cAdvisor from all containers in the cluster.

Let’s try to use more custom metrics, for example  -  let’s scale a Gorush-server by using its own metrics, see [Kubernetes: running a push-server with Gorush behind an AWS LoadBalancer](https://rtfm.co.ua/en/kubernetes-running-a-push-server-with-gorush-behind-an-aws-loadbalancer/).

#### Application-based metrics scaling

So, we have the Gorush server running in our cluster which is used to send push-notifications to mobile clients.

It has the built-in `/metrics` endpoint which returns standard time-series metrics that can be used in Prometheus.

To run the testing Gorush server we can use such `Service`, `ConfigMap`, and `Deployment`:

```
apiVersion: v1
kind: Service
metadata:
  name: gorush
  labels:
    app: gorush
    tier: frontend
spec:
  selector:
    app: gorush
    tier: frontend
  type: ClusterIP
  ports:
  - name: gorush
    protocol: TCP
    port: 80
    targetPort: 8088
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: gorush-config
data:
  stat.engine: memory
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gorush
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gorush
      tier: frontend
  template:
    metadata:
      labels:
        app: gorush
        tier: frontend
    spec:
      containers:
      - image: appleboy/gorush
        name: gorush
        imagePullPolicy: Always
        ports:
        - containerPort: 8088
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8088
          initialDelaySeconds: 3
          periodSeconds: 3
        env:
        - name: GORUSH_STAT_ENGINE
          valueFrom:
            configMapKeyRef:
              name: gorush-config
              key: stat.engine
```

Create a dedicated namespace:

```
$ kubectl create ns eks-dev-1-gorush
namespace/eks-dev-1-gorush created
```

Create the application:

```
$ kubectl -n eks-dev-1-gorush apply -f my-gorush.yaml
service/gorush created
configmap/gorush-config created
deployment.apps/gorush created
```

Check its pods:

```
$ kubectl -n eks-dev-1-gorush get pod
NAME READY STATUS RESTARTS AGE
gorush-5c6775748b-6r54h 1/1 Running 0 83s
```

Gorush Service:

```
$ kubectl -n eks-dev-1-gorush get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gorush ClusterIP 172.20.186.251 <none> 80/TCP 103s
```

Run the port-forward to its pod:

```
$ kubectl -n eks-dev-1-gorush port-forward gorush-5c6775748b-6r54h 8088:8088
Forwarding from 127.0.0.1:8088 -> 8088
Forwarding from [::1]:8088 -> 8088
```

Check metrics:

```
$ curl -s localhost:8088/metrics | grep gorush | head
HELP gorush_android_fail Number of android fail count
TYPE gorush_android_fail gauge
gorush_android_fail 0
HELP gorush_android_success Number of android success count
TYPE gorush_android_success gauge
gorush_android_success 0
HELP gorush_ios_error Number of iOS fail count
TYPE gorush_ios_error gauge
gorush_ios_error 0
HELP gorush_ios_success Number of iOS success count
```

Or another way: by using its Service name, we can reach it directly.

Find the Service:

```
$ kubectl -n eks-dev-1-gorush get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gorush ClusterIP 172.20.186.251 <none> 80/TCP 26m
```

Open proxy to the API-server:

```
$ kubectl proxy — port=8080
Starting to serve on 127.0.0.1:8080
```

And connect to the Service:

```
$ curl -sL localhost:8080/api/v1/namespaces/eks-dev-1-gorush/services/gorush:gorush/proxy/metrics | head
HELP go_gc_duration_seconds A summary of the GC invocation durations.
TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile=”0"} 9.194e-06
go_gc_duration_seconds{quantile=”0.25"} 1.2092e-05
go_gc_duration_seconds{quantile=”0.5"} 2.1812e-05
go_gc_duration_seconds{quantile=”0.75"} 5.1794e-05
go_gc_duration_seconds{quantile=”1"} 0.000145631
go_gc_duration_seconds_sum 0.001080551
go_gc_duration_seconds_count 32
HELP go_goroutines Number of goroutines that currently exist.
```

#### Kubernetes ServiceMonitor

The next thing to do is to add a `ServiceMonitor` to the Kubernetes cluster for our Prometheus Operator which will collect those metrics, check the [Adding Kubernetes ServiceMonitor](https://rtfm.co.ua/en/kubernetes-a-clusters-monitoring-with-the-prometheus-operator/#Adding_Kubernetes_ServiceMonitor).

Check metrics now with port-forward:

```
$ kubectl -n monitoring port-forward prometheus-prometheus-prometheus-oper-prometheus-0 9090:9090
Forwarding from [::1]:9090 -> 9090
Forwarding from 127.0.0.1:9090 -> 9090
```

Try to access them:

```
$ curl “localhost:9090/api/v1/series?match[]=gorush_total_push_count&start=1597141864”
{“status”:”success”,”data”:[]}
```

The `"data":[]` is empty now - or Prometheus doesn't collect those metrics yet.

Define the `ServiceMonitor`:

```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    serviceapp: gorush-servicemonitor
    release: prometheus
  name: gorush-servicemonitor
  namespace: monitoring
spec:     
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 15s
    port: gorush
  namespaceSelector:
    matchNames: 
    - eks-dev-1-gorush
  selector: 
    matchLabels:
      app: gorush
```

**_Note_** _: The Prometheus resource includes a field called_ _serviceMonitorSelector, which defines a selection of ServiceMonitors to be used. By default and before the version_ _v0.19.0, ServiceMonitors must be installed in the same namespace as the Prometheus instance. With the Prometheus Operator_ _v0.19.0 and above, ServiceMonitors can be selected outside the Prometheus namespace via the_ _serviceMonitorNamespaceSelector field of the Prometheus resource._  
 _See_ [_Prometheus Operator_](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md)

Create this `ServiceMonitor`:

```
$ kubectl apply -f ../../gorush-service-monitor.yaml
servicemonitor.monitoring.coreos.com/gorush-servicemonitor created
```

Check it in the _Targets_:

![](https://cdn-images-1.medium.com/max/1024/0*y54Jyx_-wICOnn8v.png)

_UP_, good.

And in a couple of minutes check for metrics again:

```
$ curl “localhost:9090/api/v1/series?match[]=gorush_total_push_count&start=1597141864”
{“status”:”success”,”data”:[{“__name__”:”gorush_total_push_count”,”endpoint”:”gorush”,”instance”:”10.3.35.14:8088",”job”:”gorush”,”namespace”:”eks-dev-1-gorush”,”pod”:”gorush-5c6775748b-6r54h”,”service”:”gorush”}]}
```

Or in this way:

```
$ curl -s localhost:9090/api/v1/label/__name__/values | jq | grep gorush
“gorush_android_fail”,
“gorush_android_success”,
“gorush_ios_error”,
“gorush_ios_success”,
“gorush_queue_usage”,
“gorush_total_push_count”,
```

Nice, we’ve got our metrics, let’s go ahead and use them in the HorizontalPodAutoscaler of this Deployment.

Check the metrics groups available here:

```
$ kubectl get — raw “/apis/custom.metrics.k8s.io/v1beta1” | jq . | grep “gorush”
“name”: “services/gorush_android_success”,
“name”: “pods/gorush_android_fail”,
“name”: “namespaces/gorush_total_push_count”,
“name”: “namespaces/gorush_queue_usage”,
“name”: “pods/gorush_ios_success”,
“name”: “namespaces/gorush_ios_success”,
“name”: “jobs.batch/gorush_ios_error”,
“name”: “services/gorush_total_push_count”,
“name”: “jobs.batch/gorush_queue_usage”,
“name”: “pods/gorush_queue_usage”,
“name”: “jobs.batch/gorush_android_fail”,
“name”: “services/gorush_queue_usage”,
“name”: “services/gorush_ios_success”,
“name”: “jobs.batch/gorush_android_success”,
“name”: “jobs.batch/gorush_total_push_count”,
“name”: “pods/gorush_ios_error”,
“name”: “pods/gorush_total_push_count”,
“name”: “pods/gorush_android_success”,
“name”: “namespaces/gorush_android_success”,
“name”: “namespaces/gorush_android_fail”,
“name”: “namespaces/gorush_ios_error”,
“name”: “jobs.batch/gorush_ios_success”,
“name”: “services/gorush_ios_error”,
“name”: “services/gorush_android_fail”,
```

Add a new manifest with the HPA which will use the `gorush_queue_usage` from the Pods group:

```
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: gorush-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gorush
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: gorush_total_push_count
      targetAverageValue: 2
```

With such settings, the HPA has to scale pods once the `gorush_total_push_count`'s value will be over _2_.

Create it:

```
$ kubectl -n eks-dev-1-gorush apply -f my-gorush.yaml
service/gorush unchanged
configmap/gorush-config unchanged
deployment.apps/gorush unchanged
horizontalpodautoscaler.autoscaling/gorush-hpa created
```

Check its value now:

```
$ kubectl get — raw=”/apis/custom.metrics.k8s.io/v1beta1/namespaces/eks-dev-1-gorush/pods/*/gorush_total_push_count” | jq ‘.items[].value’
“0”
```

Check the НРА:

```
$ kubectl -n eks-dev-1-gorush get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
gorush-hpa Deployment/gorush 0/1 1 5 1 17s
```

`TARGETS` _0/1_, okay.

Send a push:

```
$ curl -X POST a6095d18859c849889531cf08baa6bcf-531932299.us-east-2.elb.amazonaws.com/api/push -d ‘{“notifications”:[{“tokens”:[“990004543798742”],”platform”:2,”message”:”Hello Android”}]}’
{“counts”:1,”logs”:[],”success”:”ok”}
```

Check the `gorush_total_push_count` metric again:

```
$ kubectl get — raw=”/apis/custom.metrics.k8s.io/v1beta1/namespaces/eks-dev-1-gorush/pods/*/gorush_total_push_count” | jq ‘.items[].value’
“1”
```

One push was sent.

Check the HPA one more time:

```
$ kubectl -n eks-dev-1-gorush get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
gorush-hpa Deployment/gorush 1/2 1 5 1 9m42s
```

`TARGETS` _1/2_, `REPLICAS` still has 1, send another push, and check the events:

```
$ kubectl -n eks-dev-1-gorush get events — watch
LAST SEEN TYPE REASON KIND MESSAGE
18s Normal Scheduled Pod Successfully assigned eks-dev-1-gorush/gorush-5c6775748b-x8fjs to ip-10–3–49–200.us-east-2.compute.internal
17s Normal Pulling Pod Pulling image “appleboy/gorush”
17s Normal Pulled Pod Successfully pulled image “appleboy/gorush”
17s Normal Created Pod Created container gorush
17s Normal Started Pod Started container gorush
18s Normal SuccessfulCreate ReplicaSet Created pod: gorush-5c6775748b-x8fjs
18s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 2; reason: pods metric gorush_total_push_count above target
18s Normal ScalingReplicaSet Deployment Scaled up replica set gorush-5c6775748b to 2
```

And the HPA:

```
$ kubectl -n eks-dev-1-gorush get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
gorush-hpa Deployment/gorush 3/2 1 5 2 10m
```

Great! So, scaling by the `gorush_total_push_count` is working.

But here a trap: the `gorush_total_push_count` is cumulative metric, e.g. on the Production graph it will look like the next:

![](https://cdn-images-1.medium.com/max/1024/0*iWSRALPylq6FeVjC.png)

And in such a case our HPA will scale pods till the end of time.

#### Prometheus Adapter ConfigMap - seriesQuery and metricsQuery

To mitigate this let’s add an own metric.

The Prometheus Adapter has its own `ConfigMap`:

```
$ kubectl get cm prometheus-adapter
NAME DATA AGE
prometheus-adapter 1 46h
```

Wich contains the `config.yaml`, see its example [here>>>](https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/sample-config.yaml).

Create a [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query which will return pushes count per second:

```
rate(gorush_total_push_count{instance="push.server.com:80",job="push-server"}[5m])
```

![](https://cdn-images-1.medium.com/max/1024/0*gaZQKuCcWCdNYj-C.png)

Update the `ConfigMap` and add new query there:

```
apiVersion: v1
data:
  config.yaml: |
    rules:
    - seriesQuery: '{__name__=~"gorush_total_push_count"}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        matches: ""
        as: "gorush_push_per_second"
      metricsQuery: rate(<<.Series>>{<<.LabelMatchers>>}[5m])
```

Save and exit, check it:

```
$ kubectl get — raw=”/apis/custom.metrics.k8s.io/v1beta1/namespaces/eks-dev-1-gorush/pods/*/gorush_push_per_second” | jq
Error from server (NotFound): the server could not find the metric gorush_push_per_second for pods
```

Re-create the pod so it will apply the changes (see [Kubernetes: ConfigMap and Secrets — data auto-reload in pods](https://rtfm.co.ua/en/kubernetes-configmap-and-secrets-data-auto-reload-in-pods/)):

```
$ kubectl delete pod prometheus-adapter-7c56787c5c-kllq6
pod “prometheus-adapter-7c56787c5c-kllq6” deleted
```

Check it:

```
$ kubectl get — raw=”/apis/custom.metrics.k8s.io/v1beta1/namespaces/eks-dev-1-gorush/pods/*/gorush_push_per_second” | jq
{
“kind”: “MetricValueList”,
“apiVersion”: “custom.metrics.k8s.io/v1beta1”,
“metadata”: {
“selfLink”: “/apis/custom.metrics.k8s.io/v1beta1/namespaces/eks-dev-1-gorush/pods/%2A/gorush_push_per_second”
},
“items”: [
{
“describedObject”: {
“kind”: “Pod”,
“namespace”: “eks-dev-1-gorush”,
“name”: “gorush-5c6775748b-6r54h”,
“apiVersion”: “/v1”
},
“metricName”: “gorush_push_per_second”,
“timestamp”: “2020–08–11T12:28:03Z”,
“value”: “0”,
“selector”: null
},
…
```

Update the НРА to use the `gorush_push_per_second`:

```
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: gorush-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gorush
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: gorush_push_per_second
      targetAverageValue: 1m
```

Check it:

```
$ kubectl -n eks-dev-1-gorush get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
gorush-hpa Deployment/gorush 0/1m 1 5 1 68m
```

Events:

```
…
0s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 4; reason: pods metric gorush_push_per_second above target
0s Normal ScalingReplicaSet Deployment Scaled up replica set gorush-5c6775748b to 4
…
```

And the HPA now:

```
$ kubectl -n eks-dev-1-gorush get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
gorush-hpa Deployment/gorush 11m/1m 1 5 5 70m
```

Done.

### Useful links

- [Kubernetes Autoscaling in Production: Best Practices for Cluster Autoscaler, HPA and VPA](https://www.replex.io/blog/kubernetes-in-production-best-practices-for-cluster-autoscaler-hpa-and-vpa)
- [Ultimate Kubernetes Resource Planning Guide](https://medium.com/dev-genius/ultimate-kubernetes-resource-planning-guide-449a4fddd1d6)
- [Horizontal Autoscaling in Kubernetes #2 — Custom Metrics](https://dev.to/cdennig/horizontal-autoscaling-in-kubernetes-2-custom-metrics-549k)
- [Kubernetes HPA : ExternalMetrics+Prometheus](https://blog.kloia.com/kubernetes-hpa-externalmetrics-prometheus-acb1d8a4ed50)
- [Prometheus Custom Metrics Adapter](https://rancher.com/docs/rancher/v2.x/en/cluster-admin/tools/monitoring/custom-metrics/)
- [Horizontal Pod Autoscaling (HPA) triggered by Kafka event](https://medium.com/@ranrubin/horizontal-pod-autoscaling-hpa-triggered-by-kafka-event-f30fe99f3948)
- [Custom and external metrics for autoscaling workloads](https://cloud.google.com/kubernetes-engine/docs/concepts/custom-and-external-metrics)
- [Prometheus Metrics Based Autoscaling in Kubernetes](https://www.metricfire.com/blog/prometheus-metrics-based-autoscaling-in-kubernetes/)
- [Kubernetes best practices: Resource requests and limits](https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits)
- [Знакомство с Kubernetes. Часть 19: HorizontalPodAutoscaler](https://ealebed.github.io/posts/2019/%D0%B7%D0%BD%D0%B0%D0%BA%D0%BE%D0%BC%D1%81%D1%82%D0%B2%D0%BE-%D1%81-kubernetes-%D1%87%D0%B0%D1%81%D1%82%D1%8C-19-horizontalpodautoscaler/)
- [How to Use Kubernetes for Autoscaling](https://dzone.com/articles/how-to-use-kubernetes-for-autoscaling)
- [Horizontal Pod Autoscaling by memory](https://koudingspawn.de/kubernetes-autoscaling/)
- [Autoscaling apps on Kubernetes with the Horizontal Pod Autoscaler](https://itnext.io/autoscaling-apps-on-kubernetes-with-the-horizontal-pod-autoscaler-798750ab7847)
- [Horizontally autoscale Kubernetes deployments on custom metrics](https://banzaicloud.com/blog/k8s-horizontal-pod-autoscaler/)
- [Kubernetes pod autoscaler using custom metrics](https://sysdig.com/blog/kubernetes-autoscaler/)
- [Kubernetes HPA Autoscaling with Custom and External Metrics](https://medium.com/uptime-99/kubernetes-hpa-autoscaling-with-custom-and-external-metrics-da7f41ff7846)
- [Horizontal pod auto scaling by using custom metrics](https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.0/manage_cluster/hpa.html)
- [Horizontal Pod Autoscale with Custom Prometheus Metrics](https://itnext.io/horizontal-pod-autoscale-with-custom-metrics-8cb13e9d475)
- [Kubernetes HPA using Custom Metrics](https://medium.com/@avkanumuri/kubernetes-hpa-using-custom-metrics-b14d70e50818)
- [Kubernetes HPA Autoscaling with Custom Metrics](https://icicimov.github.io/blog/kubernetes/Kubernetes_HPA_Autoscaling_with_Custom_Metrics/)
- [Building a K8s Autoscaler with Custom Metrics](https://nuvalence.io/building-a-k8s-autoscaler-with-custom-metrics/)

_Originally published at_ [_RTFM: Linux, DevOps и системное администрирование_](https://rtfm.co.ua/en/kubernetes-horizontalpodautoscaler-an-overview-with-examples/)_._

* * *