Prometheus Observability Platform: Application metrics
Tags: devops, kubernetes, promethe, metrics
Published: 2023-09-14 10:27:31

When creating our own applications, we need to use a metrics library to generate the metrics and then inside our application functions increment said metrics. With Go for example, we can use the Prometheus library. The metrics will then be exposed to the /metrics endpoint. If our application is inside a Kubernetes cluster with a prometheus-operator, we can use a ServiceMonitor to scrape its metrics. If we don’t have such a possibility, we can instead set up an application to send metrics straight to our long-term storage solution. For VictoriaMetrics, we can use the [github.com/VictoriaMetrics/metrics](https://github.com/VictoriaMetrics/metrics) library to send the metrics to VictoriaMetrics. Remember to add authentication logic into the section pushing the metrics to the long-term storage, if necessary.

## Demo
This example assumes that you have completed the following steps as the components from those are needed:
- [Prometheus Observability Platform: Prometheus](https://dev.to/polarsquad/prometheus-observability-platform-prometheus-1019)

Let's set up a hello-world Golang application in our cluster, and use ServiceMonitor to send its metrics to Prometheus.

First, we need to update our kube-prometheus-stack Helm deployment to pick up ServiceMonitor resources with a certain label attached. We need to pass the following value to our Helm chart:
```
prometheus:
  prometheusSpec:
    serviceMonitorSelector:
      matchExpressions:
      - key: app
        operator: Exists
```

I have converted that into json so that it can be passed to Helm:
```
helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace prometheus --reuse-values --set-json 'prometheus.prometheusSpec.serviceMonitorSelector={"matchExpressions":[{"key":"app","operator":"Exists"}]}'
```

This will update our kube-prometheus-stack to pick up ServiceMonitor resources from any namespace, as long as they have an app label attached.

Next, we are going to create a namespace for our hello-world application which is a simple Golang application exposing metrics via the [Prometheus module](https://github.com/prometheus/client_golang/tree/main/prometheus). We will borrow [this](https://github.com/okteto/go-prometheus-monitoring/blob/master/main.go) already-made application, which has logic defined to increment a metric called `hello_processed_total` each time the page is loaded. 

To create a namespace and a pod, we use the following commands:
```
kubectl create namespace hello-world
```
```
kubectl run hello-world --namespace=hello-world --image='okteto/hello-world:golang-metrics' --labels app=hello-world
```

Next, we need to create a service for the new pod:
```
cat <<'EOF' | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-world
  name: hello-world
  namespace: hello-world
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: hello-world
  type: ClusterIP
EOF
```

Now we can test that our application is working by port-forwarding it. We can also check what the `hello_processed_total` metric looks like:
```
kubectl port-forward -n hello-world services/hello-world 9090:8080
```

Now navigate to http://localhost:9090 and http://localhost:9090/metrics. You should see a metric called `hello_processed_total` with a number attached. Each reload of the page will increment this number.

![Metrics](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zbv5q9gyxtv62j2klkuy.png)

Next, we need to set up a ServiceMonitor to send these metrics to Prometheus:
```
cat <<'EOF' | kubectl create -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hello-world
  namespace: hello-world
  labels:
    app: hello-world
spec:
  selector:
    matchLabels:
      app: hello-world
  endpoints:
    - port: http
EOF
```

This ServiceMonitor will target services matching the label selector (`app=hello-world`) and will scrape the port called “http”.

Now, if we port-forward our Prometheus service, we should see a new service in the service discovery section:
```
kubectl port-forward -n prometheus services/kube-prometheus-stack-prometheus 9090:9090
```

Navigate to http://localhost:9090/service-discovery and you should see that there is a new service discovered with the name `serviceMonitor/hello-world/hello-world/0` and it should show 1/1 active targets.

We can now query the `hello_processed_total metric`:

![Metrics in Prometheus](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d1yo9jctopwjomapxjl8.png)

We have now achieved sending metrics from our custom app running in its own namespace into Prometheus.

Next part: [Prometheus Observability Platform: Grafana](https://dev.to/polarsquad/prometheus-observability-platform-grafana-40d3)