Scratching an Itch with Prometheus
Published: 2020-01-27 22:09:28

---
title: Scratching an Itch with Prometheus
published: true
date: 2018-07-05 22:30:00 UTC
tags: 
canonical_url: https://braytonium.com/2018/07/05/scratching-an-itch-with-prometheus/
---

Not too long ago I became obsessed with Prometheus. I'd heard about it for a while, knew it was powerful, and couldn't quite understand how everything fit together. The documentation is extremely verbose for good reason but it took playing with it for a while for everything to click.[This post](https://ordina-jworks.github.io/monitoring/2016/09/23/Monitoring-with-Prometheus.html) is a rather concise and extensive overview that goes a long way in expressing the basic concepts to my developer brain. In their simplest form, exporters expose an HTTP endpoint of `/metrics` with the output being statistics in Prometheus' format. The real power of Prometheus comes when you expose your own `/metrics` endpoint and have Prometheus consume the statistics you generate.[This post is also a very good introduction](https://blog.alexellis.io/prometheus-monitoring/) with the section `Building your own exporter` being extremely valuable in describing just some of the possibilities.

After getting my bearings I started with a prototype with a simple premise "Why look at the usage graphs in Digital Ocean for each server independently? Why not have it in one location?"[How To Install Prometheus on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04) is a very good primer to get everything up and running quickly.

I've made a few modifications since working through the article:

- [Prometheus version 2.3.1](https://github.com/prometheus/prometheus/releases/tag/v2.3.1)
  - There have been massive perf improvements in v2.3.x.
- [node\_exporter version 0.16.0](https://github.com/prometheus/node_exporter/releases/tag/v0.16.0)
  - There are significant changes to the metrics naming conventions.
  - This exporter typically has the most coupling with Grafana dashboards and often requires altering them to work correctly.
- Use `prometheus:prometheus` for ownership of core prometheus processes like `prometheus` or `alertmanager`.
  - `sudo useradd --no-create-home --shell /bin/false prometheus`
- Use `prometheus-exporter:prometheus-exporter` for ownership of exporters. Exporters should possibly be more isolated but I feel it may be a case of YAGNI.
  - `sudo useradd --no-create-home --shell /bin/false prometheus-exporter`
- Set scrape\_interval to 1 minute: `scrape_interval: 1m`.
  - 15 seconds is still doable but I'm currently not concerned with very granular detail.
  - This reduces the load of making 4 calls per minute to just 1, reducing some overhead required for Prometheus and every exporter.

At $dayJob we've moved to provisioning servers using Laravel Forge, which has the possibility of utilizing exporters for mysqld, mariadb, postgres, memcached, redis, beanstalkd, nginx, php-fpm, and sendmail. I've opted to use [node\_exporter](https://github.com/prometheus/node_exporter), [mysqld](https://github.com/prometheus/mysqld_exporter), [nginx-vts-exporter](https://github.com/hnlq715/nginx-vts-exporter), [php-fpm](https://github.com/hipages/php-fpm_exporter), and [redis](https://github.com/oliver006/redis_exporter) respectively. To put the original premise into perspective, replicating the newer monitoring agent graphs in Digital Ocean only require `node_exporter`. A few of the exporters require very little setup, only setting a few configuration variables systemd service definitions. Other exporters like `nginx-vts-exporter` require building nginx from source.

I plan to introduce a series of posts that should aid in getting a very rudimentary implementation running. There is an abundant usage of Kubernetes in the Prometheus ecosystem, to the point that it almost seems required but fortunately it also just works(tm) in a traditional virtual machine without any real fuss.