Monitoring the Kubernetes Nginx Ingress Controller with Prometheus and Grafana
Tags: kubernetes, grafana, prometheus, nginx
Published: 2020-09-20 19:22:06

![cover](https://dev-to-uploads.s3.amazonaws.com/i/t22ld8cbidyerzd6pidf.png)

[In a previous article](/posts/infrastructure/kubernetes-nginx-ingress-controller) I explained how we can set-up an Nginx Kubernetes Ingress Controller, but how can we now monitor this? This is what I would like to tackle in this article, on how we are able to utilize Prometheus and Grafana to start visualizing what is happening on our Ingress Controller.


## Prerequisites

After following [my previous article](/posts/infrastructure/kubernetes-nginx-ingress-controller) you will now have the following components running:
* An Nginx Ingress controller deployed in the `ingress-nginx` namespace
* A demo application that we can reach through a `ingress` route

If you do not have these 2 components running, I would recommend you to go check it out! 

## Setting up Prometheus

When we open the reference of [`ingress-nginx` online](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/monitoring.md) we can see that it should be quite straightforward to install prometheus.

Simply run the following to deploy and configure the Prometheus Server:

```bash
kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/
```

Once this is done (~2 min) you will see some output of everything that was created:

```bash
serviceaccount/prometheus-server created
role.rbac.authorization.k8s.io/prometheus-server created
rolebinding.rbac.authorization.k8s.io/prometheus-server created
configmap/prometheus-configuration-hct76d4c56 created
service/prometheus-server created
deployment.apps/prometheus-server created
```

Now on to the "hard" part - reconfiguring our Nginx.

> Note: at the time of writing (20-SEP-2020) the documentation was incomplete, making me spend 5h on the issue of reconfiguring nginx...

## Reconfiguring Nginx

Reconfiguring Nginx to allow metrics to be sent to Prometheus might sound as an easy task, but actually it isn't... It appears that the official documentation is lacking on this point and that a [recent pull request](https://github.com/kubernetes/ingress-nginx/pull/6024) requires the [ServiceMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md) to be used.

Installing the ServiceMonitor is a good option, but when checking this on the [official repository](https://github.com/prometheus-operator/prometheus-operator) it shows that this is still in "Beta" so it's not a really a good option to configure this one.

Luckily for us we can always check the values defined in our [helm chart](https://github.com/kubernetes/ingress-nginx/blob/master/charts/ingress-nginx/values.yaml), which shows that we can still utilize the [prometheus annotations](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus).

Therefor we upgrade our Helm chart through the following command:

```bash
helm upgrade ingress-controller ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.metrics.enabled=true \
  --set-string controller.podAnnotations."prometheus\.io/scrape"="true" \
  --set-string controller.podAnnotations."prometheus\.io/port"="10254"
```

When we validate this through the following command

```bash
helm get values ingress-controller --namespace ingress-nginx
```

We can see that our values are now set as they should:

```bash
controller:
  metrics:
    enabled: true
    service:
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
```

When I now open the Prometheus dashboard (remember: `kubectl get nodes` and get the external IP address, and open this with the prometheus port shown through `kubectl get svc -A`) and start typing `ng` it shows our metrics!

![](./prometheus.png)

## Installing Grafana

From here on, it's a smooth ride to the finish line! We can simply follow the [official documentation](https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/) and install grafana with:

```bash
kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/
```

Once this is done, we do the same as for prometheus and open the grafana dashboard in our browser (`kubectl get nodes`; `kubectl get svc -A`) where we can use `admin:admin` as our credentials and run the following steps:

1. Click "Add data source"
2. Select "Prometheus"
3. Enter the details (note: I used `http://CLUSTER_IP_PROMETHEUS_SVC:9090`)
4. Left menu (hover over +) -> Dashboard
5. Click "Import"
6. Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/grafana/dashboards/nginx.json
7. Click Import JSON
8. Select the Prometheus data source
9. Click "Import"

![](./grafana.png)

## Conclusion

In this article we learned how we can now start monitoring our earlier application its ingress controller. The goal of this is to in a next post be able to auto-scale the created infrastructure based on the incoming requests!

> **Note:** feel free to check the original post at my blog [https://xaviergeerinck.com](https://xaviergeerinck.com)