Hands On: Pull metrics into Kubernetes from anywhere and treat them generically with the Keptn Metrics Server
Tags: keptn, kubernetes, monitoring, observability
Published: 2024-01-03 06:05:07

## Metrics, Metrics, Everywhere

Metrics are everywhere because tool sprawl is real. If you work in anything beyond a startup, you'll most likely have multiple metric-storing backends. Multiple Prometheus instances, endless vendor solutions, homegrown databases etc.

## What if...

What if you could just say "get me a metric called `foo`" and magically, it appeared?

That's precisely what the [Keptn Metrics Server](https://keptn.sh/latest/docs/getting-started/metrics/) allows.

![How the Keptn Metrics Server Works](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/88bp4opcfzay683qt4nn.jpg)

## Install Prometheus
The first thing you'll need, of course, is at least one backend to store metrics. So install [Prometheus](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus) now:

```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prom-stack prometheus-community/prometheus
```

## Install Keptn

```
helm repo add keptn https://charts.lifecycle.keptn.sh
helm repo update
helm install keptn keptn/keptn -n keptn-system --create-namespace --wait
```

## Expose Prometheus and Get an Existing Metric

Now expose the Prometheus UI and get the name of a metric you would like to use.

Get the names of the services. Look for one called `prometheus-server` running on port `80` (that's the Prometheus user interface):

```
kubectl get svc
```

Port forward to access the UI on port `8080`:

```
kubectl -n default port-forward svc/prometheus-server 8080:80
```

Go to `http://localhost:8080` and in the search box, type `prometheus_`.

Choose a metric which has a non-zero value. For example:

```
prometheus_http_requests_total{code="200", handler="/-/ready", instance="localhost:9090", job="prometheus"}
```

Make a note of that PromQL. You'll need it later.

## Create KeptnMetricsProvider

A [KeptnMetricsProvider](https://keptn.sh/latest/docs/reference/crd-reference/metricsprovider/) is how you define **where** metrics are coming from. One `KeptnMetricsProvider` is created for each "backend".

So create one now. I'm deciding to call it `local-prometheus`:

**keptnmetricsprovider.yml**
```yaml
---
apiVersion: metrics.keptn.sh/v1alpha3
kind: KeptnMetricsProvider
metadata:
  name: local-prometheus
  namespace: default
spec:
  type: prometheus
  targetServer: "http://prometheus-server.default.svc.cluster.local:80"
```

Apply it:

```
kubectl apply -f keptnmetricsprovider.yml
```

See all KeptnMetricsProviders:

```
kubectl get keptnmetricsproviders
```

## Create a KeptnMetric

A [KeptnMetric](https://keptn.sh/latest/docs/reference/crd-reference/metric/) is created to define **what** you want to retrieve, and from where (which leverages the `KeptnMetricsProvider`) and how often the metric should be retrieved (`fetchIntervalSeconds`).

The `KeptnMetricsProvider` and `KeptnMetric` must be in the same namespace.

**keptnmetric.yml**
```yaml
---
apiVersion: metrics.keptn.sh/v1alpha3
kind: KeptnMetric
metadata:
  name: prometheus-http-requests-total
  namespace: default
spec:
  provider:
    name: local-prometheus
  query: 'prometheus_http_requests_total{code="200", handler="/-/ready", instance="localhost:9090", job="prometheus"}'
  fetchIntervalSeconds: 10
```

Show all Keptn Metrics with:

```
kubectl -n default get keptnmetrics
```

## That's it. Use any metric generically regardless of backend

All of your metrics are now available on-cluster to use for whatever you need. Horizontal Pod Autoscaling, FinOps etc. etc.