Alerts and Alert Manager in Prometheus
Tags: monitoring, alertmanager, prometheu, gmail
Published: 2023-08-01 04:42:30

In the last [blog](https://t.co/mONwe4DRVY) we walked through the process of setting up a Prometheus server with Grafana dashboard. In this blog we will be setting up the alertmanager in the prometheus server.
## Structure of an alertmanager and the prometheus server

![alertmanager1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9mdun8pt7zunvx1gk23k.png)

## Defining rules for prometheus server
- In the last blog we created a `alerts.rules` config files, this file will contain all the rules for our prometheus server 
```bash
cd /opt/prometheus/
vim alerts.rules
```
```bash
groups:
  - name: Linux-disk-space
    rules:
     - alert: Linux-DiskSpaceAlert
       expr:  (node_filesystem_avail_bytes/node_filesystem_size_bytes)*100 < 20
       for: 5m
       annotations:
        summary: "Disk space is running low"
        description: 'The available disk space on {{ $labels.instance }} is less than 80percent'


  - name: Linux-memory-utilisation
    rules:
     - alert: Linux-MemoryUsage
       expr: ((node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes)*100 ) < 20
       for: 2m
       annotations:
        Summary: "High RAM Usage"
        description: 'Memory usage on {{ $labels.instance }} is greater than 80percent'

  - name: Linux-up
    rules:
     - alert: Linux-up-status
       expr: up{job='node-exporter'} < 1
       for: 2m
       annotations:
        Summary: "Linux machine/exporter down"
        description: '{{ $labels.instance }} metrics could not be fetched machine may be down'

  - name: Windows-memory-utilisation
    rules:
     - alert: Windows-MemoryUsage
       expr:  ((windows_os_physical_memory_free_bytes/windows_cs_physical_memory_bytes)*100 ) < 20
       for: 2m
       annotations:
        Summary: "High RAM Usage"
        description: 'RAM usage on {{ $labels.instance }} is higher than 80 percent'

  - name: Windows-diskspace-utilisation
    rules:
     - alert: Windows-diskspace-utilisation
       expr: (windows_logical_disk_free_bytes/windows_logical_disk_size_bytes)*100 < 20
       for: 2m
       annotations:
        Summary: "Disk Space running low"
        description: 'Diskspace utilisation on {{ $labels.instance }} is higher than 80 percent'

  - name: Windows-up
    rules:
     - alert: Windows-up-status
       expr: up{job='win-exporter'} < 1
       for: 2m
       annotations:
        Summary: "Windows machine/exporter down"
        description: '{{ $labels.instance }} metrics could not be fetched machine may be down'
```

## Setting the alertmanager
- for this walkthrough we will be setting up gmail alerts using gmail smtp server

### 1. setting up gmail smtp server
To set up a gmail smtp server for prometheus we need a gmail account say `sender@gmail.com` on which 2-step verification is enabled

![step-1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kwephk0abfi90ss88ex0.png)

![step-2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7stqye98t0224zvdmo71.png)

![step-3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2npne6oq8f2a56j1ouwa.png)

- your app password is your `secret-key` which will be used in `alertmanager.yml` as `auth_password`
![step-4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l3d1t63mc9933jikvllq.png)


### 2. config file for alertmanager
```bash
cd /opt/prometheus/
touch alertmanager.yml
vim alertmanager.yml
```
```bash
global:
  resolve_timeout: 1m

route:
  receiver: 'gmail-alert-notification'
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 1m

# sender@gmail is the gmail account on which you have generated your app password
receivers:
- name: 'gmail-alert-notification'
  email_configs:
    - to: 'receiver@gmail.com'
      from: 'sender@gmail.com'
      smarthost: smtp.gmail.com:587
      auth_username: 'sender@gmail.com'
      auth_identity: 'sender@gmail.com'
      auth_password: 'secret-key'
```

### 3. installing the alertmanager docker image
We will be installing the docker image of alertmanager on the same machine on which we installed the prometheus server
```bash
docker run -d -p 9093:9093 -v /opt/prometheus/alertmanager.yml:/etc/alertmanager/alertmanager.yml --restart unless-stopped --name alertmanager prom/alertmanager
```
- verify that alertmanager is running by visiting `alertmanager-ip:9093` in your browser

## Adding rules and alertmanager in the prometheus
```bash
cd /opt/prometheus/
vim prometheus.yml
```
```bash
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['ubuntu_id:9100']

  - job_name: 'win-exporter'
    static_configs:
      - targets: ['windows_ip:9182']

rule_files:
  - "/etc/prometheus/alerts.rules"

alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - "ubuntu_ip:9093"
# ubuntu_ip is the IP on which the alertmanager container is installed, use localcost:9093 if same as prometheus server
```
```bash
# if prometheus docker is already running run the following command
docker restart prometheus

# if not run the following
docker run -d -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/alerts.rules:/etc/prometheus/alerts.rules --restart unless-stopped --name prometheus prom/prometheus
```
Your prometheus and alertmanager is ready and can be verified by visiting `prometheus_ip:9090` and `alertmanager_ip:9093` on your browser
> I hope this blog helped you understand how to setup prometheus with alertmanager. Stay connected for more such blogs