Testing Application Monitoring Locally with a Docker Composition
Tags: prometheus, tutorial, docker, monitoring
Published: 2021-04-14 13:37:35

---
title: Testing Application Monitoring Locally with a Docker Composition
published: true
description: How to create a monitoring stack on your workstation to be able to test/debug code that generates metrics in the application
tags: prometheus,tutorial,docker,monitoring
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pamidmpvpdpufa8dyt3g.jpg
series: Prometheus Operator For Developers
---

In the [previous post](https://dev.to/camptocamp-ops/implement-prometheus-metrics-in-a-flask-application-p18), we saw how to implement metrics in a simple Python Flask application. In this post we will see how to:
* start a local monitoring stack with a Docker composition
* configure Prometheus to scrape metrics from our application
* build a Grafana dashboard
* commit the Dashboard definition so other developers can also use it

![final-grafana-dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l8agspyl7u60gvuptmaj.png)


Before launching an application in a Kubernetes cluster, we need to make sure that it generates the right metrics. This will be done using a Prometheus stack running in a Docker composition.

This test Prometheus stack will be made of:
* [Prometheus](https://prometheus.io/): Core component that will:
    * scrape application metrics 
    * store metrics in a time series database
    * expose metrics for use with Grafana
* [Grafana](https://grafana.com/docs/grafana/latest/): Visualization of Prometheus metrics 

To be able to run this Docker composition, you will need to [install Docker](https://docs.docker.com/get-docker/) and [install docker-compose](https://docs.docker.com/compose/install/).

# Start Prometheus

Let's write a `docker-compose.yml` file that will start a Docker container for Prometheus:

```yaml
version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    ports:
      - "9090:9090"
```

*Be sure to use only spaces for indentation in the YAML file, not tabs.*

Start the Docker composition with:

```shell
docker-compose up -d
```

This composition starts Prometheus with the default configuration and allows access to the web interface on [http://localhost:9090/](http://localhost:9090/).
You will be redirected to the "Graph" page where you can query metrics using the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/examples/) format.

![prometheus-home](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8johj968a72mq5voiazq.png)


In the [Status → Targets](http://localhost:9090/targets) menu, you can find all exporters from which Prometheus retrieves metrics.

![prometheus-targets-menu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tujuneaq13su1j2miiim.png)


In the default setup, there is only one defined target, which points to the [metrics endpoint of Prometheus itself](http://localhost:9090/metrics).

![prometheus-default-target](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qb41nrewopvgkt6da8wy.png)


Now it's time to configure Prometheus to retrieve metrics from our application!

Prometheus is running in a container but needs to access the application that runs on the host. We will start the application with:

```shell
flask run --host 0.0.0.0
```

With the new `--host` option, our application is running on all network interfaces, including the interface used by containers. From a container, you can connect to the host with the special IP address `172.17.0.1` (the bridge address for the `172.17.0.0` network, linked to the host). So let's create a `prometheus.yml` file that contains the Prometheus configuration:

```yaml
scrape_configs:
  - job_name: view_buy
    metrics_path: /metrics
    static_configs:
      - targets:
        - 172.17.0.1:5000
```

* *line 2:* define the name of the Prometheus exporter; here we use the name of the application
* *line 5:* IP address and port of the application

*The `prometheus.yml` file must be in the same folder as `docker-compose.yaml`.*

We edit the `docker-compose.yaml` as follow to inject the Prometheus configuration in the container:

```yaml
version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
```

Let’s restart the Docker composition with:

```shell
docker-compose up -d
```

You should now be able to see a target that points to the application running on the host on the [target list page](http://localhost:9090/targets).

![prometheus-app-target](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/35dc5u6pas9vra53388c.png)


This target has a static label with the name of the application set in the `prometheus.yml` file: `job="view_buy"`.

We can now display all metrics from this target on the [Graph page](http://localhost:9090/graph?g0.range_input=1h&g0.expr=%7Bjob%3D%22view_buy%22%7D&g0.tab=1). In order to query the time series for our label, you just need to surround the label and value with curly braces: `{job="view_buy"}`.

![prometheus-target-metrics](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/id0c2r4939w3we5xeocg.png)


It's time to generate some traffic!

Open 3 terminals and run the following commands:

* `watch -n 1 "curl localhost:5000/view/product1"`
* `watch -n 2 "curl localhost:5000/view/product2"`
* `watch -n 3 "curl localhost:5000/buy/product1"`

On the [Graph page](http://localhost:9090/graph) of the Prometheus Web UI, you can query the `view_total` metric using the following PromQL expression: [`view_total`](http://localhost:9090/graph?g0.range_input=1h&g0.expr=view_total&g0.tab=1). A visual representation of the metrics is available in the [Graph tab](http://localhost:9090/graph?g0.range_input=1h&g0.expr=view_total&g0.tab=0), just under the "Execute" button.

![prometheus-graph](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fiwm85f6bo362r9km96o.png)


# Start Grafana

Add a new service to the Docker composition by editing the `docker-compose.yml` file:

```yaml
version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:7.5.2
    volumes:
      - ./datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
    ports:
      - "3000:3000"
```

We need to create the `datasource.yaml` file with the following content:

```yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    isDefault: true
    url: http://prometheus:9090/
```

This file configures the connection between Grafana and Prometheus. The hostname of the Prometheus container is `prometheus` because this is the name of the service in the Docker composition. Restart the Docker composition with:

```shell
docker-compose up -d
```

You can now access the [Grafana Web UI](http://localhost:3000/) using the default login/password `admin/admin`.

![grafana-login](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j18vvu0zedij46um4u58.png)


Skip the password change for now:

![grafana-skip](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vfndwrewmza84i3pw89r.png)

![grafana-explore](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oybg0p03azxgp881j6cl.png)


In the [Explore view](http://localhost:3000/explore), you can use the following expression to visualize the number of views for each product: [`view_total`](http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Prometheus%22,%7B%22exemplar%22:true,%22expr%22:%22view_total%22%7D%5D).

![grafana-promql](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1x12ucs9sco47bs3qh07.png)


The `view_total` metric is a *Counter*, a metric that always increases. As suggested by Grafana, it is better to display a [temporal derivative of the value](http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Prometheus%22,%7B%22exemplar%22:true,%22expr%22:%22rate%28view_total%5B5m%5D%29%22%7D%5D), which can be achieved using the `rate()` function.

Now we have more meaningful information about the number of views of products. The unit is *view per second*. For example, "product2" has 0.5 view per second, which makes sense as it corresponds to one view each 2 seconds.

Finally, in the legend, you can find the labels of each *time series*, one per product.

![grafana-rate](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wyleo29qtjkf5er8fw9w.png)


Let's create a dashboard now.

# Creating a Grafana Dashboard

In the Grafana web UI, you can [create a new dashboard](http://localhost:3000/dashboard/new). Choose "Add an empty panel". Then type the following expression: `rate(view_total[5m])` and validate with 'Shift+Enter'.

![grafana-add-panel](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6k0qso61x2gta3vm14k1.png)


To improve the legend, you can define the "Legend Format" and extract label values with the expression `{{<label name>}}`. For our application, we will use `{{product}}`.

![grafana-dashboard1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/obkxl7aieibcxpaxav1k.png)


The legend looks better as we now only have the name of the product. It's also possible to add a prefix and a suffix, using e.g.: `Product {{product}} views`.

Now you can click on the "Apply" button to save modifications of the dashboard. You can then add a new panel for purchases. Finally, click on the save 💾 button.

![grafana-dashboard2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2vobxqtc8aevfpovjcz7.png)


# Persisting Metrics and Dashboards

If you stop or restart the current Docker composition, you will lose all the metrics retrieved by Prometheus and the Dashboard you just built. To persist data, with this local setup, we will use Docker volumes.

We will first create a Docker volume to persist Prometheus metrics. The time series database is stored in `/prometheus/` inside the container so we will mount the volume on this path:

```yaml
version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus:/prometheus
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:7.5.2
    volumes:
      - ./datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
    ports:
      - "3000:3000"
volumes:
  prometheus:
```

* *line 7:* we add a volume to persist the content of `/prometheus` folder
* *last 2 lines:* a top level `volumes` key is used to define volumes used in the Docker composition

Finally, you can apply changes with `docker-compose.yml`. Metrics are now persisted across restarts of the Docker composition. Note that mounting the volume resets all metrics previously retrieved.

In order to persist the Grafana dashboard, we will create a file that contains the definition of the dashboard:

* Switch back to the [Grafana web ui](http://localhost:3000/),
* Open your dashboard
* Go to settings ⚙ and then "JSON Model"
* Copy the JSON definition of the dashboard
* Paste the copied content into a new `dashboard.json` file

![grafana-dashboard-settings](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/az5agpva942zzfjaxfmz.png)


![grafana-json-model](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5db0mmwgoxdx884it601.png)

![grafana-json-model-copy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y8g9810lcu281vm77d2m.png)



The last step is to mount this dashboard in the Grafana container.

Create a `dashboards.yaml` file with the following contents:

```yaml
apiVersion: 1

providers:
  - name: 'provisionned dashboards'
    orgId: 1
    folder: ''
    folderUid: ''
    type: file
    disableDeletion: false
    editable: true
    updateIntervalSeconds: 10
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
```

This file tells Grafana to load every dashboard defined in the `/var/lib/grafana/dashboards`. Finally, modify the Docker composition to include `dashboards.yaml` and `dashboard.json` in the Grafana container:

```yaml
version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus:/prometheus
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:7.5.2
    volumes:
      - grafana:/var/lib/grafana
      - ./datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
      - ./dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml
      - ./dashboard.json:/var/lib/grafana/dashboards/dashboard.json
    ports:
      - "3000:3000"

volumes:
    prometheus:
    grafana:
```

Update the Docker composition with `docker-compose up -d`. Your dashboard should be automatically loaded into Grafana.

![grafana-provisionned](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ztf19ogdtoq0t7qg3nub.png)

# Dashboard Modification Workflow

If you modify the dashboard, remember to copy the new JSON Model to the `dashboard.json` file and restart Grafana with `docker-compose restart grafana`. Now, your modifications are persisted as code in the `dashboard.json` and you can commit the dashboard definition within your project repository.

If someone wants to try your project, they just need to run the application and the monitoring stack with `docker-compose up -d`. They can then access your dashboard, see the evolution of metrics and create or modify graphs.

In this post, we were able to start a monitoring stack to retrieve metrics from our test application and build a dashboard to visualize the evolution of product views and purchases. We created several files that can be committed in Git and used later by other developers.

# Conclusion
You are now ready to launch a monitoring stack aside your application, whichever language you’re using. You can also integrate the application as a new service in the Docker composition to ease test and demo.
This Docker composition should help you get your application monitoring code ready for production, so you can now deploy it on a Kubernetes cluster, and avoid the usual pitfalls of a new Kubernetes deployment.

In the next post, we will deploy the application and its monitoring in a Kubernetes cluster and see how to:
* integrate the application monitoring with the Prometheus Operator
* create alerts based on application metrics
* package everything in a Helm Chart