Kubernetes monitoring with Prometheus
Tags: kubernetes, docker, devops
Published: 2019-07-25 16:49:11


Microservices architecture is going to be one of the essential features in software development in the coming years. Packing a large monolithic application into small containers poses various advantages. One of the key advantages I can think of is that if certain things fail, then a part of the application will be down and it can auto-heal rather than crashing the whole application. This is one of the reasons that when Instagram, Facebook, and Whatsapp crashes, only certain functionalities stop working. The application, as a whole, is still working. This serves a great advantage. Also, when the demand for a particular service or component increases, the number of components can be increased or decreased easily. The small containers which I am referring here are the Docker containers and the application is deployed on Kubernetes. Kubernetes is a container orchestration software which manages the containers at various levels and enables the management of connections and endpoints of these containers.

[Kubernetes](https://kubernetes.io/?source=post_page) or K8s is an open-source self-healing application which manages the deployment, scaling, and operation of these containers. It was originally developed by Google but later on, donated to CNCF. Since Kubernetes provides tons of services, there needs to be an easier way to monitor the activities in the Kubernetes cluster. This is possible by [Prometheus](https://prometheus.io/). It is an open-source solution to monitor the metrics and manage the alerts in the system. It was developed by SoundCloud, but later joined CNCF as the second hosted project after Kubernetes. Prometheus provided a rich set of monitoring metrics and alert management system which helps the developers to monitor and get notified about any unusual activity or consumption.

## Prometheus Operator
CoreOs launched [Prometheus operator](https://github.com/coreos/prometheus-operator) to ease the process of integrating K8s with Prometheus. It preserves the configuration of both the K8s and Prometheus while installing and configuring the cluster. It provides easy monitoring for K8s services and deployments, along with managing Prometheus, Grafana and Alertmanager configuration.
When a new version of the application is deployed, K8s manages the creation of new pod and deletes the older one. Prometheus, on the other hand, constantly watches the K8s API and creates a new Prometheus configuration whenever it detects a change, based on the services/pods changes. It uses a ServiceMonitor, a CRD(Custom Resource Definition), to abstract the configuration to target.
## Installation
### Prerequisites
* Kubernetes
* Helm (Package installer for K8s)

### Steps
Install Prometheus operator in a different namespace. It is preferable to keep your monitoring containers in a separate namespace.
```
$ helm install stable/prometheus-operator --name prometheus-operator --namespace monitor

```
If everything got installed perfectly, you can see these pods available:
```
$ kubectl get pods -n monitor
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-operator-alertmanager-0          2/2     Running   0          13d
prometheus-operator-grafana-749b598b6c-t4r48             2/2     Running   0          13d
prometheus-operator-kube-state-metrics-d7b8b7666-zfqg5   1/1     Running   0          13d
prometheus-operator-operator-667dd7cbb7-hjbl6            1/1     Running   0          13d
prometheus-operator-prometheus-node-exporter-mgsqb       1/1     Running   0          13d
prometheus-prometheus-operator-prometheus-0              3/3     Running   1          13d
```
To run the dashboard, enter the following command and go to `http://localhost:9000`.
```
$ kubectl port-forward -n monitor prometheus-prometheus-operator-prometheus-0 9090
```

### Prometheus Dashboard
![](https://thepracticaldev.s3.amazonaws.com/i/ztrejj1mdkknfsto0664.png)
You can enter your query to get the results about any particular instance or even a graph of it as shown in the figure above.
To see the visual representation at each level, we use Grafana. It provides some great visual insights regarding the usage, health and other metrics. We can also add more custom metrics. We will get real-time analysis of the data.
```
$ kubectl port-forward $(kubectl get  pods --selector=app=grafana -n  monitor --output=jsonpath="{.items..metadata.name}") -n monitor  3000
```
Go to `http://localhost:3000` and enter ‘admin’ as username and ‘prom-operator’ as password. These are the available options:

![](https://thepracticaldev.s3.amazonaws.com/i/iih0cf7ss3bhy7ohgl0k.png)

You can get visual graphs by selecting any one of the options. Node-level metrics are shown here:
![](https://thepracticaldev.s3.amazonaws.com/i/66znm7td8s6tvx2n8jce.png)
We can configure alerts in many ways. We can access the dashboard to configure the AlertManager by going to `http://localhost:9093` after executing this command:
```
$ kubectl port-forward -n monitor alertmanager-prometheus-operator-alertmanager-0 9093
```
![](https://thepracticaldev.s3.amazonaws.com/i/9x1hllzbntj78nrjaruw.png)
It’ll look like this. Here you can add more alerts and can see the Slack API URL under the status tab. We can set the notifications on Slack, HipChat or even email. Some of the templates are available [here](https://prometheus.io/docs/alerting/notification_examples/?source=post_page).

## Conclusion
The use of microservices is going to increase soon and monitoring the metrics and alert notifications are going to be an essential part of it. Prometheus provides optimal monitoring services with easy installation services.