Monitoring external services with Prometheus and Grafana
Tags: ruby, rails, monitoring, showdev
Published: 2020-03-23 05:07:21

---
title: Monitoring external services with Prometheus and Grafana
published: true
description: Simple guide on how to monitor external services HTTP requests of your Ruby application with Prometheus and Grafana.
tags: #ruby #rails #monitoring #showdev
---

## Preface

Time ago, web-applications depended only on self-written code or libraries. It was easy to diagnose any problem, plainly investigating how an application work.

Today, modern web-applications tend to have many external dependencies, such as microservices or external SaaS. It makes your system depend not only on the internal factors but on the reliability of external services as well. When something fails, it's essential to localize a problem and find out which side caused it.
 
That is why monitoring of external services call is crucial.

I work for [Amplifr](https://amplifr.com), the social media automation platform. Our Ruby core application depends on many external APIs: Facebook, Instagram, LinkedIn, Pinterest, [Bitly](https://bitly.com), Stripe, Amplitude, Intercom, and many others. 
We also are integrated with many self-hosted micro-services, such as [imgproxy](https://github.com/imgproxy/imgproxy) for resizing images, [browser screenshot generator](https://github.com/dsalahutdinov/screenshot), social services bots, and other internal domain-specific services.

When an application has so many integrations, it's handy to have a quick  traffic details overview:
 - how many requests application does
 - how much time consumes an average request
 - how many of the attempts fail

That was my second point to set monitoring up and write my own solution.

## Metrics

As most APIs work over HTTP, we will monitor the HTTP requests.

Meet my gem [yabeda-http_requests](https://github.com/yabeda-rb/yabeda-http_requests). It is the new one of the [Yabeda](https://github.com/yabeda-rb) monitoring solution by [Evil Martians](https://evilmartians.com/). If you are not familiar with it, here is the [great article](https://evilmartians.com/chronicles/meet-yabeda-modular-framework-for-instrumenting-ruby-applications) to start.

In a nutshell, Yabeda - is the open-source Ruby framework for collection metrics of Ruby processes (Rack/Rails, Sidekiq,p Puma) and exporting them into Prometheus (or any other adapter).

`yabeda-http_requiests` wraps all the external HTTP requests with the help of [sniffer gem](https://github.com/aderyabin/sniffer). Sniffer patches [all popular HTTP libraries](https://github.com/aderyabin/sniffer/tree/master/lib/sniffer/adapters) and logs performing requests. By using middleware, `yabeda-http_requests` gem calculates and stores metrics. Then metrics could be exported with `yabeda-prometheus` gem.

To get started with it you need:
 - to add the gem into your dependencies (Gemfile)
 - set up metrics exporter for Prometheus
 - tune up Prometheus to scrape the metrics endpoint
 - import predefined [dashboard](https://grafana.com/grafana/dashboards/11959) into your Grafana config

You could find all the details `yabeda-http_requests`  [at the README.md](https://github.com/yabeda-rb/yabeda-http_requests#usage).

Let's have a look at providing metrics & charts.


## What to monitor?

The first release of `yabeda-http_requests` includes some basics:

### RPM

The basic metric is RPM (request per minute). It shows how many HTTP request application does.

Formally, `yabeda-http_requests` collect a `http_request_count` as counter. It is separated by quest host, port and method:
```
# HELP http_response_total A counter of the total number
# of external HTTP responses.
http_response_total{host="twitter.com",port="443",method="GET",status="200"} 131.0
http_response_total{host="dev.to",port="443",method="GET",status="200"} 131.0
```

 Than, using [rate](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate) function:
```
sum by (host) (rate(http_response_total[1m]) )
```
renders the RPM chart:

![Request per minute (by host)](https://dev-to-uploads.s3.amazonaws.com/i/u3j9smoqyxk04bel6043.png)


It is easy to regroup the same chart by status:

![Request duration chart](https://dev-to-uploads.s3.amazonaws.com/i/9hy5732zxlchmuahfqr2.png)

❗ "RPM by status" chart uses another metric, which is called `http_response_total` which logs the HTTP responses.


### Request duration

When you want to investigate which one of the dependent components causes a slow-down, there is a histogram duration data and the corresponding chart. It shows how many times consumes the slowest request:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/d6ixo7quxvib6lewblb3.png)

## Example Application

This section is for practical approach lovers. Here is the [example of web-application](https://github.com/yabeda-rb/yabeda-http_requests/tree/master/example). It shows how all things work together: web-application, metrics collector, exporter, and Prometheus & Grafana.

The example has a docker-compose config for automatic setup. Just run the following:

```bash
$ git clone https://github.com/yabeda-rb/yabeda-http_requests
$ cd yabeda-http_requests/example
$ docker-compose up
```

This command runs the rack-application, Sidekiq, Prometheus, and Grafana.
Rack application adds random web scrapping jobs to the sidekiq. Navigate to `http://localhost:9292` to enqueue some jobs.

"Sidekiq instance" runs background scrapping tasks, collects metrics, and exposes an endpoint with data for the Prometheus. 
Here is the basic sidekiq configuration, showing how the sidekiq process run its web-server:

```ruby
# sidekiq config
Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch('REDIS_URL') }

  # configures all the Yabeda metrics
  ::Yabeda.configure!

  # expose 9394 port with /metrics path for prometheus
  ::Yabeda::Prometheus::Exporter.start_metrics_server!
end
```

Navigate to `http://localhost:9394/metrics` to see metrics endpoint, and you will see the metrics values formatted as text:

```
# TYPE http_request_total counter
# HELP http_request_total A counter of the total number of external HTTP requests.
http_request_total{host="twitter.com",port="443",method="GET"} 16922.0
http_request_total{host="dev.to",port="443",method="GET"} 17083.0
http_request_total{host="amplifr.com",port="443",method="GET"} 451.0
http_request_total{host="www.sitepoint.com",port="443",method="GET"} 428.0
# TYPE http_response_total counter
# HELP http_response_total A counter of the total number of external HTTP responses.
http_response_total{host="twitter.com",port="443",method="GET",status="200"} 16910.0
http_response_total{host="dev.to",port="443",method="GET",status="200"} 17080.0
http_response_total{host="amplifr.com",port="443",method="GET",status="301"} 451.0
http_response_total{host="www.sitepoint.com",port="443",method="GET",status="200"} 428.0
http_response_total{host="twitter.com",port="443",method="GET",status="503"} 1.0
# TYPE http_response_duration_milliseconds histogram
# HELP http_response_duration_milliseconds A histogram of the response                                                duration (milliseconds).
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="0.5"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="1"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="2.5"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="5"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="10"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="25"} 0.0
...
```


Prometheus scrapes `http://sidekiq:9394/metrics` periodically and stores the metrics into its database.

Here is the overview diagram of how it works together:
![HTTP requests monitoring example diagram](https://dev-to-uploads.s3.amazonaws.com/i/yykdbbpezp7ivllfy88j.png)

Grafana visualizes the data. Navigate to `http://localhost:3000/d/OGd-oEXWz/yabeda-external-http-requests`. Use `admin/foobar` to login and see the template dashboard:

![Yabeda External HTTP requests Grafana dashboard](https://dev-to-uploads.s3.amazonaws.com/i/yw2vtcaiw8x9myke9h24.png)

❗You could import prepared Grafana dashboard  [here](https://grafana.com/grafana/dashboards/11959).


## Conclusion

The monitoring of external HTTP calls is essential. It is handy to an overview of your outgoing application traffic. It also helps to diagnose slow-down issues of dependent services.

Yabeda framework and the `yabeda-http_requests` let to quickly set up metrics and visualize them using a predefined Grafana dashboard template.

If you have any other ideas, please do not hesitate to contribute: by writing an Issue, sending PR, or leaving the comment.

Thank you for reading.