Request-based autoscaling in Kubernetes: scaling to zero
Tags: kubernetes
Published: 2022-09-12 12:56:17

_TL;DR: In this article, you will learn how to monitor the HTTP requests to your apps in Kubernetes and how to define autoscaling rules to increase and decrease replicas for your workloads._

Reducing infrastructure costs boils down to turning apps off when you don't use them.

_That's easy to do manually, but how to turn them on automatically when you need them?_

**You can do so with a scale-to-zero strategy.**

Let me show you how to implement it in Kubernetes.

Imagine you are running a reasonably resource-intensive app on the dev cluster.

Ideally, **it's off when people leave the office** and on when they start the day.

_You could use a CronJob to scale up/down, but what happens during the weekend?_

_And public holidays?_

![A development and production Kubernetes clusters](https://res.cloudinary.com/uasabi/image/upload/v1661831726/threads/scale-to-zero-2.png)

Instead of coming up with an evergrowing list of exceptions, you could scale up your workloads based on traffic:

- When there's traffic → autoscale on usage.
- When there is no traffic → turn off the app.

And that's the idea behind scaling to zero.

![Scale to zero strategy](https://res.cloudinary.com/uasabi/image/upload/v1661831727/threads/scale-to-zero-3.png)

_But what do you need to make it work?_

The first step is collecting metrics and measuring the app traffic flow.

You could intercept the traffic by injecting a proxy in front.

**The proxy could temporarily buffer requests when the app is scaled to 0.**

When the app is ready, the proxy can forward all requests.

In Kubernetes, that's [an extra proxy container in the Pod.](https://learnk8s.io/sidecar-containers-patterns#:~:text=look%20at%20the-,Sidecar%20Pattern,-%2C%20where%20you%20add)

![Intercepting all traffic that flows to the app with a proxy container](https://res.cloudinary.com/uasabi/image/upload/v1661831726/threads/scale-to-zero-4.png)

Once you have the metrics, you need two things:

1. [A metrics server](https://github.com/kubernetes-sigs/metrics-server) to store and aggregate metrics (Kubernetes doesn't come with one by default).
1. A way to export the metrics to the metrics server.

![Kubernetes metrics server](https://res.cloudinary.com/uasabi/image/upload/v1661831652/threads/scale-to-zero-5.png)

And finally, the last piece of the puzzle is the autoscaler.

**You need to configure the Horizontal Pod Autoscaler to consume the metrics from the metrics server and adjust the replicas count accordingly.**

![Collecting metrics, exposing them to the Kubernetes API and driving the Horizontal Pod Autoscaler](https://res.cloudinary.com/uasabi/image/upload/v1661831652/threads/scale-to-zero-6.png)

In Kubernetes, [you can have exports, metrics servers and proxy bundled into one with KEDA.](https://keda.sh)

KEDA is an event-driven autoscaler that implements the [Custom Metrics API.](https://github.com/kubernetes-sigs/custom-metrics-apiserver)

It's made of 3 components:

1. **A Scaler.**
1. **A Metrics Adapter.**
1. **A Controller.**

![KEDA architecture](https://res.cloudinary.com/uasabi/image/upload/v1661831652/threads/scale-to-zero-7.png)

[Scalers](https://keda.sh/docs/2.8/scalers/) are like adapters that can collect metrics from databases, message brokers, telemetry systems etc.

KEDA has a special [scaler that creates an HTTP proxy](https://github.com/kedacore/http-add-on) that measures and buffers requests before they reach the app.

![KEDA HTTP add-on](https://res.cloudinary.com/uasabi/image/upload/v1661831727/threads/scale-to-zero-8.png)

KEDA's metrics adapter is responsible for exposing the metrics collected by the HTTP scaler in a format that the Kubernetes metrics pipeline can consume.

![Autoscaling with KEDA in Kubernetes](https://res.cloudinary.com/uasabi/image/upload/v1661831652/threads/scale-to-zero-9.png)

Finally, the KEDA controller glues all the components together:

- **Collects the metrics** using the adapter and exposes them to the metrics API.
- **Registers** and manages the KEDA-specific **Custom Resource Definitions (CRDs)**.
- Creates and manages the **Horizontal Pod Autoscaler.**

![KEDA and the HTTPScaledObject](https://res.cloudinary.com/uasabi/image/upload/v1661831652/threads/scale-to-zero-10.png)

With KEDA, you don't create the Horizontal Pod Autoscaler (HPA).

Instead, you create an [HTTPScaledObject](https://github.com/kedacore/http-add-on/blob/9feb10ca31fd66dc76e58adbf24d0f5cd88e4ecf/docs/ref/v0.1.0/http_scaled_object.md), which is a wrapper around the HPA that includes:

1. How to connect to the source of the metrics.
1. How to route the traffic to the app.
1. The settings to create the Horizontal Pod Autoscaler.

![HTTPScaledObject YAML definition](https://res.cloudinary.com/uasabi/image/upload/v1661831653/threads/scale-to-zero-11.png)

_You can see the setup in action here._

- There is no active app.
- 6 requests are issued.
- Eventually, the app (octopus icon) appears and serves the six requests.

![Scale to zero in action](https://res.cloudinary.com/uasabi/image/upload/v1661832502/threads/scale-to-zero-preview.gif)

_Scaling to zero… isn't that serverless? Not really._

Once scaled to at least one replica, the app still behaves as a regular app (e.g. it can handle hundreds of connections on its own).

Instead, serverless frameworks might create more functions to handle concurrent traffic.

Also, serverless frameworks handle incoming requests. You only have to define the application logic.

**With a scale to zero, you are still handling the traffic.**

If you wish to see this in action, [Salman Iqbal](https://www.youtube.com/c/SoulmanIqbal) demos scaling to zero here https://event.on24.com/wcc/r/3917950/F23E579DEBEB05039B2E2BEA1F6FF853?partnerref=learnk8s

And finally, if you've enjoyed this short post, you might also like the [Kubernetes workshops that we run at Learnk8s](https://learnk8s.io/training) or this collection of past Twitter threads https://twitter.com/danielepolencic/status/1298543151901155330.

Until next time!