Metrics with Prometheus StatsD Exporter and Grafana
Tags: metrics, statsd, prometheus, grafana
Published: 2019-07-31 15:46:04

---
title: Metrics with Prometheus StatsD Exporter and Grafana
published: true
description: 
tags: metrics, statsd, prometheus, grafana
cover_image: https://thepracticaldev.s3.amazonaws.com/i/kynf5f96m98kgn5gk1bx.png
---


This post explains how to convert [StatsD](https://github.com/statsd/statsd#statsd--) metrics using the Prometheus StatsD-exporter, and later visualising them with Grafana. 

I explain from the perspective of an application which is instrumented with StatsD but is monitored by a Prometheus server. As such, I will use the terminal as a pretend application emitting metrics **;** Prometheus will monitor the application and collect any metrics **;** and finally Grafana will be used to draw a graph by using Prometheus as a data source.

## Overview of Technologies Used
This section briefly covers why each technology is being used but to learn more, you can read about each using the links in the [Recommended Reading](#recommended-reading) section.

### Metrics
During a system's runtime, measurements taken of any of its properties are known as _metrics_. These metrics are typically in the form of _data points_ associated with some information to give it context. Metrics are things like the number memory bytes used, number of website visitors, the time a task took to complete, etc.

StatsD metrics allow such information to be recorded and represented using a [text format](https://github.com/b/statsd_spec#statsd-metrics-export-specification-v01) specification. Below are two StatsD metric lines each representing how long an API request has taken.

```bash
# metric format: <metric name>:<value>|ms
test_api_call.orders-api.timer./v1/orders:84|ms
test_api_call.orders-api.timer./v1/orders:95|ms
```
In the above example, two API calls to `GET /api/orders` are measured at 84 and 95 milliseconds each. The `|ms` is used with StatsD timer metrics. StatsD also supports other metric types like Counters, Gauges, Histograms and [more](https://github.com/statsd/statsd/blob/master/docs/metric_types.md#statsd-metric-types).

#### Why dot-delimited?
To record information about the metric, in this case the name of the API and its endpoint, the `<metric name>` part is _dot-delimited_ so those details can be captured.

![](https://thepracticaldev.s3.amazonaws.com/i/ytsp6k7bu0c591vz0hu1.png "Figure 1: StatsD metric in dotted notation")

_Later you will see how the Statsd-exporter will help us capture the dot-delimited parts we are concerned with._

### Prometheus for Monitoring
Prometheus can be used to monitor the performance of an instrumented system such as, a microservice, a host server, etc. The system will need to expose its metrics in order for Prometheus to collect them - this is also known scraping. The scraped metrics can then be visualised or queried using the Prometheus [expression-browser](https://prometheus.io/docs/visualization/browser/#expression-browser). Prometheus also has an [AlertManager](https://prometheus.io/docs/alerting/alertmanager/) which can send alerts about certain metrics being monitored - it has support for Slack, PagerDuty, email and [more](https://prometheus.io/docs/introduction/faq/#can-i-send-alerts).

The following diagram shows Prometheus scraping an application's metrics which are exposed on  the`/metrics` path. Grafana can then use these metrics for visualisations.

![](https://thepracticaldev.s3.amazonaws.com/i/seqaahl8z0q0hhgrslck.png "Figure 2: Prometheus scraping metrics from two applications")

### Statsd-exporter
The statsd-exporter is used to convert Statsd metrics into Prometheus metrics. This is because the StatsD [text-format](https://github.com/b/statsd_spec#statsd-metrics-export-specification-v01) represents metrics differently to Prometheus' [exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-format-example) which is represented like this:

```
<time series name>{<label name>=<label value>,...}
```

The following diagram shows how a StatsD metric, is converted to Prometheus metrics by using an exporter's mapping rules.

![](https://thepracticaldev.s3.amazonaws.com/i/2vvlwz5t26l5fu6dzl69.png "Figure 3: StatsD-exporter mapping configuration")
> _It is very common for a system to record its metrics in a format different to Prometheus, therefore there are many [Exporters](https://prometheus.io/docs/instrumenting/exporters) written which allow such metrics to be converted to Prometheus' time series notation_

#### Where did count, sum and quantile come from?
The StatsD-exporter automatically converts timer metrics into a [Prometheus summary](https://prometheus.io/docs/concepts/metric_types/#summary). The summary contains extra information like the count, sum and each quantile for the current observation. _By default the quantiles are 0.5, 0.9 and 0.99_.

In the image above, 10 API requests took 80 milliseconds each to respond. As such a summary is generated with these details:
- `_count` of 10.
- `_sum` of 0.799999999999999 for that observation.
- and each `_quantile` where 0.5 (50th percentile) is the median.

Now there is an understanding of what each technology does in relation to each other, we can get started implementing our test application.

---

## Configure Prometheus and Statsd-exporter
In this section the configuration files for both Prometheus and the statsd-exporter are created.

### Create the Directory and Files
Create a directory named `.prometheus` to contain the files `prometheus.yml` and `test-mapping.yml` as follows:
```bash
# I have created mine in $HOME/.prometheus/
mkdir ~/.prometheus && cd $_
touch ./{prometheus,test-mapping}.yml
```
Run `tree` or `find .` to verify the files.

### Prometheus Configuration
This gives us some basic configuration for the Prometheus server such as static service discovery, and how often to scrape each discovered service.

Add the following to the `prometheus.yml` file and save it.
```yaml
global:
  scrape_interval:		15s
  evaluation_interval:	15s

scrape_configs:
  # optional: this makes the metrics available to us about Promethus itself.
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # tells Prometheus to scrape metrics an address over port 9123
  - job_name: 'test_metrics'
    static_configs:
      - targets: ['host.docker.internal:9123'] # see statsd-exporter further down
        labels: {'host': 'test-app-001'} # optional: just a way to identify the system exposing metrics
```

_Prometheus also supports [dynamic service discovery](https://github.com/prometheus/prometheus/tree/master/discovery) which is useful if you deploy applications on AWS ECS or other scalable cloud solutions._

### Statsd-Exporter Configuration
This is the file used by the Prometheus [statsd-exporter](#) so it knows how to convert StatsD metrics into Prometheus metrics.

Add the following to the `test-mapping.yml` file and save it. 
```yaml
mappings:
    # usage: test_api_call.product-api.timer./v1/product
  - match: "test_api_call.*.timer.*"
    name: "test_api_call"
    labels:
        api_name: "$1"
        api_endpoint: "$2"
```

## Startup the Statsd Exporter
Before visualising any metrics with Grafana, we can check whether our Statsd metrics are being converted correctly.

To start, run the the official Prometheus [statsd-exporter](https://hub.docker.com/r/prom/statsd-exporter) Docker image, and then send it metrics over port 8125.

### Running
_Make sure you are inside the `.prometheus` directory._

```bash
docker run --name=prom-statsd-exporter \
	-p 9123:9102 \
	-p 8125:8125/udp \
	-v $PWD/test-mapping.yml:/tmp/test-mapping.yml \
	prom/statsd-exporter \
		--statsd.mapping-config=/tmp/test-mapping.yml \
		--statsd.listen-udp=:8125 \
		--web.listen-address=:9102
```

### What this does
- listens for metrics sent to an address on port `8125` - _e.g._ `echo "hello:1|c" | nc -w 0 -u 127.0.0.1 8125`
- converts Statsd metrics to Prometheus ones by using the `test-mapping.yml` file created earlier.
- expose the metrics for Prometheus to scrape - in this case over host port `9123` - container port `9102`.

After running the image you should see output like this:
![](https://thepracticaldev.s3.amazonaws.com/i/5hh4mcrkzrfzmv011932.png "Figure 4: StatsD-exporter Docker logs after startup")

## Startup Prometheus

Now the Prometheus server can be started so it can scrape the metrics exposed on port 9123.
> _Recall that the `prom-statsd-exporter` container, is exposing metrics on container port 9102 which were remapped to host port 9123._

To start, run the official [Prometheus](https://hub.docker.com/r/prom/prometheus) Docker image.

### Running
_Make sure you are inside the `.prometheus` directory._
```bash
docker run --name=prometheus \
	-p 9090:9090 \
	-v $PWD/prometheus.yml:/prometheus.yml \
	prom/prometheus \
		--config.file=/prometheus.yml \
	    --log.level=debug \
		--web.listen-address=:9090 \
		--web.page-title='Prometheus - Test Metrics Demo'
```

### What this does
- runs Prometheus server and tells it to scrape metrics from the targets defined in `prometheus.yml`
- allow us to visualise the metrics using the Prometheus browser and even query them using [PromQL](#)
- allows Grafana to use Prometheus as a data source over address port 9090

After running the image you should see output like this:
![](https://thepracticaldev.s3.amazonaws.com/i/xw8h1lwgymu21aju5mrp.png "Figure 5: Prometheus Docker logs after startup")

_The above informs us that the `prometheus.yml` file was loaded, and Prometheus knows about the two targets to scrape._


### Verify the Targets are Up and Running
Go to [localhost:9090/targets](http://localhost:9090/targets). Each one endpoint for each target should be `up`.

![](https://thepracticaldev.s3.amazonaws.com/i/3ljdorq2rdwmzlj4g1gg.jpg "Figure 6: Prometheus showing the targets it can scrape")

We can start sending metrics and prometheus will scrape them.

## Sending some Metrics
Now try sending a couple metric over localhost IP address and port 127.0.0.1:8125 using the lines below.
```bash
echo "test_api_call.orders-api.timer./v1/orders:89|ms" | nc -w 0 -u 127.0.0.1 8125
echo "test_api_call.orders-api.timer./v1/orders:68|ms" | nc -w 0 -u 127.0.0.1 8125
```

Now goto http://localhost:9123 to view the metrics. If you have updated your `hosts` file to resolve `host.docker.internal` to `localhost` then you can visit `http://host.docker.internal:9123` instead.

Search for `test_api_call` and you should see a Prometheus summary similar to the following.

![](https://thepracticaldev.s3.amazonaws.com/i/zbv3qxzvj9a2l4i072j8.jpg "Figure 7: Prometheus Scraped metrics")

---
## Visualise the API Call Metrics in Grafana
This final section looks at starting up Grafana on port 3000, adding a Prometheus data source, and creating a Graph to visualise the API call metrics.

Start by running the official [Grafana](https://hub.docker.com/r/grafana/grafana) Docker image.

```bash
docker run -d --name=grafana -p 3000:3000 grafana/grafana
```
The default _username_ and _password_ for Grafana are both _admin_ and _admin_ respectively.

### Adding a Prometheus Data Source to Grafana
To add a Prometheus data source and add a Graph you can follow the instructions at the Prometheus website - _instead of me repeating the same instructions here_.
- [Create a Prometheus data source](https://prometheus.io/docs/visualization/grafana/#creating-a-prometheus-data-source) 
- [Creating-a-prometheus-graph](https://prometheus.io/docs/visualization/grafana/#creating-a-prometheus-graph)

__Plotting the average request duration__
Within the the section _"Creating a Prometheus Graph"_ where it mentions 'Entering a Prometheus expression' into the Query field, use the expression `rate(test_api_call_sum[1m]) / rate(test_api_call_count[1m])` as seen below. This expression calculates the average request duration for the last minute using the Prometheus summary for `test_api_call`

Optionally, you can also update the legend field to `{{api_name}} - {{api_endpoint}}` as seen below also - doing so will display the legend like this: `orders-api - /v1/orders`.

![](https://thepracticaldev.s3.amazonaws.com/i/qsp44osw217xnqte0kbq.jpg "Figure 8: Grafana Prometheus Query and Legend")

### Emit Metrics Continuously
Now to save us having to send metrics manually - _just to update the graph more frequently_, we can write a simple command-line statement which will send a metric every second. This will be done in two shells, in order to simulate API calls to our fictional `orders-api` and `invoice-api`. 

Below the [shuf](https://explainshell.com/explain?cmd=shuf+-i+50-150+-n+1) command is used to generate a random range of milliseconds between 50 and 150.

__Shell 1: orders-api__
```bash
while true; do echo -n "test_api_call.orders-api.timer./v1/orders:$(shuf -i 50-150 -n 1)|ms" | \
	nc -w 1 -u 127.0.0.1 8125; done
``` 

__Shell 2: invoice-api__
```bash
while true; do echo -n "test_api_call.invoice-api.timer./v1/invoice:$(shuf -i 50-150 -n 1)|ms" | \
	nc -w 1 -u 127.0.0.1 8125; done
```
> _use **ctrl**+**c** to stop either loop_

### The Visualisation
After a few seconds you should see the graph drawing. And some time later - in my case 5 minutes your graph should look similar to the image below. _Make sure the Dashboard's refresh rate is at least 5 seconds._

![](https://thepracticaldev.s3.amazonaws.com/i/q6h9iruipr7vnt4gydmm.jpg "Figure 9: The API call metrics visualisation")

## How do I emit metrics from a real application?
There are StatsD client libraries written for Go, Node, Python, Perl  and more. See link below:
https://github.com/statsd/statsd/wiki#client-implementations
Some of the libraries also default the host `localhost` and port to `8125` so you can get up and running quickly.

### Node StatsD Client Example
This example uses the NPM module [node-statsd](https://github.com/sivy/node-statsd#node-statsd).
```js
const StatsD = require('node-statsd');
const statsdClient = new StatsD(); // defaults to localhost:8125

statsdClient.timing('example.response_time', 80); // record 80ms response time
statsdClient.increment('example.site_visit');     // record a new site visitor count +1
```

## Can I just emit metrics to Prometheus format instead of StatsD?
Yes, it is possible instrument an application to record metrics in Prometheus format. This approach eliminates the need to use a Prometheus exporter sidecar and the extra work of creating a mapping file.

To instrument your application with Prometheus, use one of the [official or unofficial](https://prometheus.io/docs/instrumenting/clientlibs) client libraries.

---

## Conclusion
Getting Statsd metrics out of an application (terminal for this demo) and into Grafana, only required the following:

- two configuration files.
- running statsd-exporter, Prometheus and Grafana.
- emitting metrics over a host address and port with UDP using the Statsd line protocol.
- setting Prometheus as a data source in Grafana and configuring a graph.

That was it! The majority of this post is mostly explaining some essentials which I think are very important to understand. Although the terminal is used as a pretend application 'to emit metrics', the principles are the same for a real application. So whether you write your application in Go, Node, Python, etc, you can just use a StatsD client library to emit metrics from it.

#### Recommended Reading
- Learn more about Prometheus - https://prometheus.io
- Prometheus Exporters - https://prometheus.io/docs/instrumenting/exporters
- Instrument an application using Prometheus itself - https://prometheus.io/docs/instrumenting/clientlibs 
- Statsd Github - https://github.com/statsd/statsd#statsd--

Thank you for reading!

[Cover image](https://www.flickr.com/photos/kirklewis/48423312857/)