React faster: Forward Prometheus Alerts to Teams
Tags: monitoring, sre, prometheus
Published: 2021-11-05 08:54:00

Many companies use Microsoft Teams. Receiving alerts in Teams allows you to react faster. It makes working on an incident visible, as you can answer the message. Let's add it to our monitoring stack!

Head over to [GitHub](https://github.com/ablx/monitoring_stack) to get the full code.


## Teams configuration
[Alertmanager](https://verbosemode.dev/react-faster-alertmanager-to-teams-fa888bcd7c01) does not support Teams out of the box. You must use webhooks to achieve this. The webhook is used by prom2teams to write messages.

Add a channel to the team which should receive the notifications. Click on the three dots beside the team name and choose  `Add channel`.

![Add a channel in Teams](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jzk6s03on1vb3fcur8u5.png)

To add the webhook, click on the dots beside the channel and click on `Connectors`.

![Add connector in Teams](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pe7awkfw3m498l9y5onc.png)


Configure an `Incoming Webhook`. Teams shows the webhook URL once you click `Create`. Copy this URL in the next step.


![Configure Webhook in Teams](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/laxa51ru0bhronz8y4q8.png)



## Add prom2teams to the stack
[prom2teams](https://github.com/idealista/prom2teams) will use the webhook to send  messages to Teams.

Add prom2teams to your `docker-compose.yml`:

```yml
  prom2teams:
    image: idealista/prom2teams:2.7.0
    restart: unless-stopped
    environment:
      PROM2TEAMS_CONNECTOR: "url from the webhook"
    ports:
      - 8089:8089
```

## Update your alerts
The [Alertmanager](https://verbosemode.dev/react-faster-alertmanager-to-teams-fa888bcd7c01) uses labels to decide which alert goes to which notification channel. Change the `prometheus/alerts.yml` to contain the following:

```yml

groups:
  - name: DemoAlerts
    rules:
      - alert: InstanceDown 
        expr: up{job="services"} < 1 
        for: 1m
        labels: # labels and annotations are new
          severity: low
        annotations:
          summary: 'Alert with low severity.'

      - alert: InstanceDownCritical
        expr: up{job="services"} < 1 
        for: 1m 
        labels:
          severity: high
        annotations:
          summary: 'Alert with high severity.'

```

Now you have two alerts with different labels.

## Configure the Alertmanager
Change the `alertmanager/alertmanager.yml` so that the [Alertmanager](https://verbosemode.dev/react-faster-alertmanager-to-teams-fa888bcd7c01) is aware of prom2teams.


```yml
route:
  group_by: [ alertname ]
  receiver: 'mail' # default receiver
  repeat_interval: 24h
  routes:
    - receiver: 'teams'
      repeat_interval: 12h
      matchers:
        - severity="medium"

    - receiver: 'teams'
      repeat_interval: 4h
      matchers:
        - severity="high"


receivers:
  - name: 'mail'
    email_configs:
      - smarthost: 'yourmailhost.com:465'
        auth_username: 'yourmail@yourmailhost.com'
        auth_password: "your mail password"
        from: 'yourmail@yourmailhost.com'
        to: 'someonesmail@yourmailhost.com'
        require_tls: false

  - name: 'teams'
    webhook_configs:
      - url: "http://prom2teams:8089"
        send_resolved: true
```

Now, the [Alertmanager](https://verbosemode.dev/react-faster-alertmanager-to-teams-fa888bcd7c01) can publish to two channels. 
`matchers` tell the Alertmanager which channel to use.

The Alertmanager sends alerts with `medium` severity to Teams. This is repeated every four hours - until you fix it!

The Alertmanager uses email if no route matcher matches.

You can use the [routing tree editor](
https://www.prometheus.io/webtools/alerting/routing-tree-editor/) to check if the configuration is what you need. Add labels and values of your alert to see which route they will take.

![Routing Tree Editor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9n9s2hked22jf34g86rd.png)


## Check the result

Start the stack with `docker-compose up` and wait until the alerts fire.

You will get a notification from Teams for the `InstanceDownCritical` alert:

![Alert in Teams](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a9jasyd14pveiu04w9x0.png)

And an email for the `InstanceDown` alert:

![Mail alert](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wb40tjyehsa5n0pm1ijc.png)




Which notification channels do you want to see next? Leave me a message!
Head over to [GitHub](https://github.com/ablx/monitoring_stack) to get the full code.

**If this article was helpful for your, please consider to buy me a coffee :-)**
[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/Y8Y1EKYJI)