Prometheus: Alertmanager with SendGrid and Slack API
Tags: kubernetes, monitoring
Published: 2021-03-28 21:54:16

I've been working in with Prometheus a lot, it's about 3 years now when I changed my mind from Nagios / Zabbix monitoring to a metrics monitoring. I was developing lot of different Perl, Python and Shell scripts to monitor infrastructures and applications via SNMP, NPRE o NSCA "protocols".

I discovered Kubernetes in their second year of life, and then Prometheus. You can read a lot of different Prometheus implementations like adding a Grafana, maybe doing some bigger Prometheus with Cortex or Thanos, but what is really useful to me on my personal projects? Setting up Alertmanager.

Im here I will show you how to manage AlertManager. At first I installed it with Helm and configured a custom values.yaml file, but let's make it simple. I will show you how to edit YAMLs and later how to custom Helm charts :D

# AlertManager
## Configure AlertManager

AlertManager is the tool that sends all notifications via mail or API. Prometheus server has all the alert rules, when an alert is triggered by a rule, Alertmanager will send the notification.

First step is to configure Alertmanager config file (alertmanager.yml)

In here we could add some customization parameters like:

- **Global**: Where we can set our notification sender by SMTP (SendGrid, Gmail...), API (Slack, WeChat..) or simply set the timeouts.

- **Route**: We set a sending logic here, how the mails and to who the notifications will be sent.

- **Inhibit Rules**: Add another set of rules to set diffferent route rules to different receivers

- **Receivers**: Here we set the different receivers. Maybe we need to send notifications to slack, mail, or both. In here we need to add as much receviers that we want to.

In this sample I will set up an API (via Slack) and a mail (via SendGrid)

```yaml
global:
  # Sendgrid SMTP properties.
  smtp_smarthost: 'smtp.sendgrid.net:587'
  smtp_from: 'Alertmanager <alertmanager@cosckoya.io>'
  smtp_auth_username: 'apikey'
  smtp_auth_password: '<API KEY HERE>'

  # Slack API properties
  slack_api_url: 'https://hooks.slack.com/services/<API TOKEN HERE>'

  receivers:
  - name: mail
    email_configs:
    - to: "admins@mail.me"
      headers:
        Subject: "Alert ({{ .Status }}): {{ .CommonLabels.severity }} {{ .CommonAnnotations.message }} ({{ .CommonLabels.alertname }})"
      html: |
        Greetings,
        <p>
        You have the following firing alerts:
        <ul>
        {{ range .Alerts }}
        <li>{{.Labels.alertname}} on {{.Labels.instance}}</li>
        <li>Labels:</li>
        <li>{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}</li>
        <li>{{ end }}Annotations:</li>
        <li>{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}</li>
        <li>{{ end }}---</li>
        {{ end }}
        </ul>
        </p>

  - name: slack     
    slack_configs:
    - channel: '#alerting'
      send_resolved: true
      icon_url: https://avatars3.githubusercontent.com/u/3380462
      title: |-
       [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
       {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
         {{" "}}(
         {{- with .CommonLabels.Remove .GroupLabels.Names }}
           {{- range $index, $label := .SortedPairs -}}
             {{ if $index }}, {{ end }}
             {{- $label.Name }}="{{ $label.Value -}}"
           {{- end }}
         {{- end -}}
         )
       {{- end }}
      text: >-
       {{ range .Alerts -}}
       *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
  
       *Description:* {{ .Annotations.description }}
  
       *Details:*
         {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
         {{ end }}
       {{ end }}

  route:
    group_wait: 10s
    group_interval: 5m
    receiver: mail
    repeat_interval: 10s
    routes:
    - match:
        severity: High
      repeat_interval: 1m
      receiver: slack       
```
This parameters will setup AlertManager to send notifications to the declared "routes"; mail as default and Slack API when the match case is setted up.

Now we have to setup some alerts. These ones are just for demostration and testing that the alert triggers works.

```yaml
  groups:
  - name: deadman.rules
    rules:
    - alert: deadman
      expr: vector(1)
      for: 10s
      labels:
        severity: Critical
      annotations:
        message: Dummy Check
        summary: This is a dummy check meant to ensure that the entire lerting pipeline is functional.
  - name: pod.rules
    rules:
    - alert: PodLimit
      expr: kubelet_running_pods > 0
      for: 15m
      labels:
        severity: High
      annotations:
        message: 'Kubelet {{$labels.instance}} is running {{$value}} pods.'
```

Last Step is to setup "alerting" in the prometheus server config file (prometheus.yml):
```yaml
[...]
  alerting:
    alertmanagers:
      - static_configs:
        - targets:
          - prometheus-alertmanager:80
[...]      
```
---

If you managed to set it up correctly, then you will see a "gatling gun" Alertmanager notifications in your mail and Slack channel.

Enjoy!