Prometheus: Alertmanager’s alerts receivers and routing based on severity level and tags
Tags: monitoring, prometheus, alertmanager
Published: 2019-03-26 12:19:02

---
title: Prometheus: Alertmanager’s alerts receivers and routing based on severity level and tags
published: true
tags: Monitoring,Prometheus,Alertmanager
canonical_url: https://rtfm.co.ua/en/prometheus-alertmanagers-alerts-receivers-and-routing-based-on-severity-level-and-tags/
---

![](https://rtfm.co.ua/wp-content/uploads/2017/09/prometheus-logo.png)We have three working environments – _Dev_, _Stage_, _Production_.

Also, there are a bunch of alerts with different severities – _info_, _warning_ и _critical_.

For example:

```
...
- name: SSLexpiry.rules
  rules:
  - alert: SSLCertExpiring30days
    expr: probe_ssl_earliest_cert_expiry{job="blackbox"} - time() < 86400 * 30
    for: 10m
    labels:
      severity: info
    annotations:
      summary: "SSL certificate warning"
      description: "SSL certificate for the {{ $labels.instance }} will expire within 30 days!"
...
```

Alerts are sent to Slack and [OpsGenie](https://rtfm.co.ua/prometheus-opsgenie-i-alertmanager-uvedomleniya-v-pochtu-sms-telefon/).

The task is depending on environment and severity level – send to Slack only, or Slack + OpsGenie.

OpsGenie in its turn depending on severity level will do:

- for the _warning_ – will send an email plus notification to its mobile application
- for the _critical_ – email plus notification to its mobile application plus bot’s call to a mobile

Thus the whole logic looks like next:

- _Dev_
  - all messages independent on severity – send to Slack only
- _Staging_:
  - _info_ – Slack only
  - _warning_ и _critical_ – Slack + OpsGenie and set _warning_ priority for OpsGenie _(P3)_
- _Production_
  - _info_ – Slack only
  - _warning_ и _critical_ – Slack and OpsGenie and set _critical_ priority for OpsGenie _(P1)_

To break down messages between Slack and OpsGenie we gave three [receivers](https://prometheus.io/docs/alerting/configuration/#%3Creceiver%3E) configured and in _warning_ and _critical_ receivers – priorities _P3_ or _P1_ will be set for the OpsGenie:

```
...
receivers:
  - name: 'default'
    slack_configs:
      - send_resolved: true
        title_link: 'https://monitor.example.com/prometheus/alerts'
        title: '{{ if eq .Status "firing" }}:confused:{{ else }}:dancing_panda:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        text: "{{ range .Alerts }}*Priority*: `{{ .Labels.severity | toUpper }}`\nMonitoring host: {{ .Labels.monitor }}\n{{ .Annotations.description }}\n{{ end }}"
  - name: 'warning'
    slack_configs:
      - send_resolved: true
        title_link: 'https://monitor.example.com/prometheus/alerts'
        title: '{{ if eq .Status "firing" }}:disappointed_relieved:{{ else }}:dancing_panda:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        text: "{{ range .Alerts }}*Priority*: `{{ .Labels.severity | toUpper }}`\nMonitoring host: {{ .Labels.monitor }}\n{{ .Annotations.description }}\n{{ end }}"
    opsgenie_configs:
      - priority: P3
  - name: 'critical'
    slack_configs:
      - send_resolved: true
        title_link: 'https://monitor.example.com/prometheus/alerts'
        title: '{{ if eq .Status "firing" }}:scream:{{ else }}:dancing_panda:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        text: "{{ range .Alerts }}*Priority*: `{{ .Labels.severity | toUpper }}`\nMonitoring host: {{ .Labels.monitor }}\n{{ .Annotations.description }}\n{{ end }}"
    opsgenie_configs:
      - priority: P1
```

And the routing itself is done in the [`route`](https://prometheus.io/docs/alerting/configuration/#%3Croute%3E) block:

```
...
route:
  group_by: ['alertname', 'cluster', 'job', 'env']
  repeat_interval: 24h
  group_interval: 5m
  # capture All Dev + All INFO
  receiver: 'default'
  routes:
    # capture All WARN to the 'warning' with P3
    - match:
        severity: warning
      receiver: warning
      routes:
      # forward Dev WARN to the 'default'
      - match_re:
          env: .*(-dev).*
        receiver: default
    # capture All CRIT to the 'critical' with P1
    - match:
        severity: critical
      receiver: critical
      routes:
      # forward Stage CRIT to the 'warning'
      - match_re:
          env: .*(-stage).*
        receiver: warning
      # forward Dev CRIT to the 'default'
      - match_re:
          env: .*(-dev).*
        receiver: default
...
```

Here we the ‘`default`‘ route set – all alerts didn’t match for other rules below will be sent via this route, which will send only Slack notification.

The additional routes are described:

1. in `match` catch alerts using the `severity: warning` tag
2. in the nested route using `match_re` the `env` will be checked – if it as “_-dev_” value then it will send back to the `default` receiver
3. all other alerts with the _warning_ level will be sent back and will go thru the `receiver: warning` receiver

Similarly, rules on the next level will be applied – catch alerts with the `severity: critical` and check them:

1. if `env: .*(-stage).*` – then go to the `warning` receiver
2. if `env: .*(-dev).*` – then go to the `default` receiver
3. everything other (only _env == production_ and _severity == critical_ are left) – will go thru the `critical` receiver

Using such an approach you can write rules using any tags and use any nested levels to check conditions and select next routes for alerts.

### Similar posts

- <small>03/26/2019</small> [Prometheus: Alertmanager – send alerts to a “/dev/null”](https://dev.to/setevoy/prometheus-alertmanager--send-alerts-to-a-devnull-he-temp-slug-751082)<small> (0)</small>  
- <small>03/06/2019</small> [Prometheus: blackbox-exporter probe_http_status_code == 0 and its debug](https://dev.to/setevoy/prometheus-blackbox-exporter-probehttpstatuscode--0-and-its-debug-41mc)<small> (0)</small>  
- <small>08/15/2018</small> [Prometheus: проверка отсутствия метрик – avg_over_time()](https://rtfm.co.ua/prometheus-proverka-otsutstviya-metrik-avg_over_time/)<small> (0)</small>  
- <small>03/10/2019</small> [Prometheus: RTFM blog monitoring set up with Ansible – Grafana, Loki, and promtail](https://dev.to/setevoy/prometheus-rtfm-blog-monitoring-set-up-with-ansible--grafana-loki-and-promtail-58i)<small> (0)</small>