Prometheus Alertmanager best practices
Tags: prometheus, devops, monitoring
Published: 2023-02-09 09:59:54

---
title: Prometheus Alertmanager best practices
published: true
description: Woke up at 5AM because of a random alert? Do you know that Prometheus Alertmanager can help you with that? Learn more in this article

tags: prometheus, devops, monitoring
cover_image: https://sysdig.com/wp-content/uploads/BlogImages-PrometheusAlertmanagerBestPractices-featured.png
canonical_url: https://sysdig.com/blog/prometheus-alertmanager/
# Use a ratio of 100:42 for best results.
---
<!-- wp:paragraph -->
<p>Have you ever fallen asleep to the sounds of your on-call team in a Zoom call? If you’ve had the misfortune to sympathize with this experience, you likely understand the problem of <strong>Alert Fatigue</strong> firsthand.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>During an active incident, it can be exhausting to tease the upstream root cause from downstream noise while you’re context switching between your terminal and your alerts.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This is where <strong>Alertmanager</strong> comes in, providing a way to mitigate each of the problems related to Alert Fatigue.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In this article, you will learn:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li><a href="#alert-fatigue">What Alert Fatigue is</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#alertmanager">What AlertManager is</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#routing">Routing</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#inhibition">Inhibition</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#silencing-throttling">Silencing and Throttling</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#grouping">Grouping</a></li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li><a href="#notification-template">Notification Template</a></li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2 id="alert-fatigue">Alert Fatigue</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>Alert Fatigue</strong> is the exhaustion of frequently responding to unprioritized and unactionable alerts. This is not sustainable in the long term. Not every alert is so urgent that it should wake up a developer. Ensuring that an on-call week is sustainable must prioritize sleep as well.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>Was an engineer woken up more than twice this week?</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>Can the resolution be automated or wait until morning?</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>How many people were involved?</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Companies often focus on response time and how long a resolution takes but how do they know the on-call process is not contributing to burn out?</p>
<!-- /wp:paragraph -->

<!-- wp:table {"className":"is-style-stripes"} -->
<figure class="wp-block-table is-style-stripes"><table><tbody><tr><td><strong>Pain Point</strong></td><td><strong>Feature</strong></td><td><strong>Alertmanager</strong></td></tr><tr><td>Send alerts to the right team</td><td>Routing</td><td>Labeled alerts are routed to the corresponding receiver</td></tr><tr><td>Too many alerts at once</td><td>Inhibition</td><td>Alerts can inhibit other alerts (e.g., Datacenter down alert inhibits downtime alert)</td></tr><tr><td>False positive on an Alert</td><td>Silencing</td><td>Temporarily silence an alert, especially when performing scheduled maintenance</td></tr><tr><td>Alerts are too frequent</td><td>Throttling</td><td>Customizable back-off options to avoid re-notifying too frequently</td></tr><tr><td>Unorganized alerts</td><td>Grouping</td><td>Logically group alerts by labels such as 'environment=dev' or 'service=broker'</td></tr><tr><td>Notifications are unstructured</td><td>Notification Template</td><td>Standardize alerts to a template so that alerts are structured across services</td></tr></tbody></table></figure>
<!-- /wp:table -->

<!-- wp:heading -->
<h2 id="alertmanager">Alertmanager</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Prometheus <strong>Alertmanager</strong> is the open source standard for translating alerts into alert notifications for your engineering team. <a href="https://prometheus.io/docs/alerting/latest/alertmanager/" target="_blank" rel="noreferrer noopener">Alertmanager</a> challenges the assumption that a dozen alerts should result in a dozen alert notifications. By leveraging the features of Alertmanager, dozens of alerts can be distilled into a handful of alert notifications, allowing on-call engineers to context switch less by thinking in terms of incidents rather than alerts.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="routing">Routing</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>Routing</strong> is the ability to send alerts to a variety of receivers including Slack, Pagerduty, and email. It is the core feature of Alertmanager.</p>
<!-- /wp:paragraph -->

<!-- wp:preformatted -->
<pre class="wp-block-preformatted">route:
  receiver: slack-default            # Fallback Receiver if no routes are matched
  routes:
    - receiver: pagerduty-logging
      continue: true
    - match:
      team: support
      receiver: jira
    - match:
      team: on-call
      receiver: pagerduty-prod</pre>
<!-- /wp:preformatted -->

<!-- wp:image {"id":65945,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image size-large"><a href="https://sysdig.com/wp-content/uploads/BlogImages-PrometheusAlertmanagerBestPractices-diagram1-1170x351.png"><img src="https://sysdig.com/wp-content/uploads/BlogImages-PrometheusAlertmanagerBestPractices-diagram1-1170x351.png" alt="Prometheus alertmanager diagram 1" class="wp-image-65945" title="image_tooltip"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Here, an alert with the label {team:on-call} was triggered. Routes are matched from top to bottom with the first receiver being <code>pagerduty-logging</code>, a receiver for your on-call manager to track all alerts at the end of each month. Since the alert does not have a <code>{team:support}</code> label, the matching continues to <code>{team:on-call} </code>where the alert is properly routed to the pagerduty-prod receiver. The default route, slack-default, is specified at the top of the routes, in case no matches are found.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="inhibition">Inhibition</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>Inhibition</strong> is the process of <strong>muting downstream alerts </strong>depending on their label set. Of course, this means that alerts must be systematically tagged in a logical and standardized way, but that's a human problem, not an Alertmanager one. While there is no native support for warning thresholds, the user can take advantage of labels and inhibit a warning when the critical condition is met. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This has the unique advantage of supporting a warning condition for alerts that don't use a scalar comparison. It's all well and good to warn at 60% CPU usage and alert at 80% CPU usage, but what if we wanted to craft a warning and alert that compares two queries? This alert triggers when a node has more pods than its capacity.</p>
<!-- /wp:paragraph -->

<!-- wp:code {"language":"yaml"}-->
<pre class="wp-block-code"><code>(sum by (kube_node_name) (kube_pod_container_status_running)) &gt; 
on(kube_node_name) kube_node_status_capacity_pods</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>We can do exactly this by using inhibition with Alertmanager. In the first example, an alert with the label <code>{severity=critical}</code> will inhibit an alert of <code>{severity=warning}</code> if they share the same region, and alertname.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In the second example, we can also inhibit downstream alerts when we know they won't be important in the root cause. It might be expected that a Kafka consumer behaves anomalously when the Kafka producer doesn't publish anything to the topic.</p>
<!-- /wp:paragraph -->

<!-- wp:code {"language":"yaml"}-->
<pre class="wp-block-code"><code>inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: &#91;'region','alertname']
  - source_match:
      service: 'kafka_producer'
    target_match:
      service: 'kafka_consumer'
    equal: &#91;'environment','topic']
</code></pre>
<!-- /wp:code -->

<!-- wp:image {"id":65946,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image size-large"><a href="https://sysdig.com/wp-content/uploads/BlogImages-PrometheusAlertmanagerBestPractices-diagram2-1170x429.png"><img src="https://sysdig.com/wp-content/uploads/BlogImages-PrometheusAlertmanagerBestPractices-diagram2-1170x429.png" alt="Prometheus alertmanager diagram 2" class="wp-image-65946" title="image_tooltip"/></a></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2 id="silencing-throttling">Silencing and Throttling</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now that you've woken up at 2 a.m. to exactly one root cause alert, you may want to acknowledge the alert and move forward with remediation. It’s too early to resolve the alert but alert re-notifications don’t give any extra context. This is where silencing and throttling can help.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Silencing</strong> allows you to temporarily snooze an alert if you're expecting the alert to trigger for a scheduled procedure, such as database maintenance, or if you've already acknowledged the alert during an incident and want to keep it from renotifying while you remediate.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Throttling</strong> solves a similar pain point but in a slightly different fashion. Throttles allow the user to tailor the renotification settings with three main parameters:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>group_wait</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>group_interval</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>repeat_interval</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:image {"id":65947,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image size-large"><a href="https://sysdig.com/wp-content/uploads/BlogImages-PrometheusAlertmanagerBestPractices-diagram3-1170x468.png"><img src="https://sysdig.com/wp-content/uploads/BlogImages-PrometheusAlertmanagerBestPractices-diagram3-1170x468.png" alt="Prometheus alertmanager diagram 3" class="wp-image-65947" title="image_tooltip"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>When Alert #1 and Alert #3 are initially triggered, Alertmanager will use <code>group_wait</code> to delay by 30 seconds before notifying. After an initial alert has been triggered, any new alert notifications are delayed by group_interval . Since there was no new alert for the next 90 seconds, there was no notification. Over the subsequent 90 seconds however, Alert #2 was triggered and a notification of Alert #2 and Alert #3 was sent. In order to not forget about the current alerts if no new alert has been triggered, <code>repeat_interval</code> can be configured to a value, such as 24 hours, so that the currently triggered alerts send a re-notifications every 24 hours.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="grouping">Grouping</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>Grouping</strong> in Alertmanager allows multiple alerts sharing a similar label set to be sent at the same time- not to be confused with Prometheus grouping, where alert rules in a group are evaluated in sequential order. By default, all alerts for a given route are grouped together. A <code>group_by</code> field can be specified to logically group alerts.</p>
<!-- /wp:paragraph -->

<!-- wp:code {"language":"yaml"}-->
<pre class="wp-block-code"><code>route:
  receiver: slack-default            # Fallback Receiver if no routes are matched
  group_by: &#91;env]
  routes:
    - match:
        team: on-call
      Group_by: &#91;region, service]
      receiver: pagerduty-prod
</code></pre>
<!-- /wp:code -->

<!-- wp:image {"id":65948,"sizeSlug":"large","linkDestination":"custom"} -->
<figure class="wp-block-image size-large"><a href="https://sysdig.com/wp-content/uploads/BlogImages-PrometheusAlertmanagerBestPractices-diagram4-1170x819.png"><img src="https://sysdig.com/wp-content/uploads/BlogImages-PrometheusAlertmanagerBestPractices-diagram4-1170x819.png" alt="Prometheus alertmanager diagram 4" class="wp-image-65948" title="image_tooltip"/></a></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Alerts that have the label {team:on-call} will be grouped by both region and service. This allows users to immediately have context that all of the notifications within this alert group share the same service and region. Grouping with information such as <code>instance_id</code> or <code>ip_address</code> tends to be less useful, since it means that every unique <code>instance_id</code> or <code>ip_address</code> will produce its own notification group. This may produce noisy notifications and defeat the purpose of grouping.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If no grouping is configured, all alerts will be part of the same alert notification for a given route.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="notification-template">Notification Template</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Notification templates offer a way to customize and standardize alert notifications. For example, a notification template can use labels to automatically link to a runbook or include useful labels for the on-call team to build context. Here, <code>app</code> and <code>alertname</code> labels are interpolated into a path that links out to a runbook. Standardizing on a notification template can make the on-call process run more smoothly since the on-call team may not be the direct maintainers of the microservice that is paging.</p>
<!-- /wp:paragraph -->

<!-- wp:html -->
<div class="sysdig-box" style="background-color: #f8f8f8; padding: 2em;">
<h2 style="margin-top: 0;"><em>Manage alerts with a click with Sysdig Monitor</em></h2>
<p>As organizations grow, maintaining Prometheus and Alertmanager can become difficult to manage across teams. Sysdig Monitor makes this easy with Role-Based Access Control where teams can focus on the metrics and alerts most important to them. We offer a turn-key solution where you can manage your alerts from a single pane of glass. With Sysdig Monitor you can spend less time maintaining Prometheus Alertmanager and spend more time monitoring your actual infrastructure. Come chat with industry experts in monitoring and alerting and we'll get you up and running.</p>
<figure><a href="https://sysdig.com/wp-content/uploads/Prometheus-Alertmanager-CTA-1170x530.png"><img src="https://sysdig.com/wp-content/uploads/Prometheus-Alertmanager-CTA-1170x530.png" alt="Alert Monitoring in Sysdig Monitor" width="1438" height="899" class="aligncenter size-full wp-image-53165"></a></figure><p></p>
<p>Sign up now for a <a href="https://sysdig.com/company/free-trial-monitor/" class="gtmbgmonpop" target="_blank" rel="noopener">free trial of Sysdig Monitor</a></p>
</div>
<!-- /wp:html -->