Prometheus & Grafana Installation with Node exporter,Alert Manager,SMTP service configuration for Monitoring
Tags: linux, devops, prometheus, grafana
Published: 2023-11-02 16:05:43

## **Prometheus Installation**

- Do the apt Update
`apt update -y && apt upgrade -y`

- Download prometheus

```
wget https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz

```
`tar -xvf prometheus-2.47.2.linux-amd64.tar.gz`

- Rename prometheus 2.47.2 to prometheus-files
`mv prometheus-2.47.2.linux-amd64 prometheus-files`

- Add prometheus user
`useradd --no-create-home --shell /bin/false prometheus`

- Create directories and addgroup
`mkdir /etc/prometheus`
`mkdir /var/lib/prometheus`
`groupadd prometheus`

- Change the ownership of the above created directories
`chown prometheus:prometheus /etc/prometheus`
`chown prometheus:prometheus /var/lib/prometheus`

- Copy the prometheus and promtool to /etc/prometheus and provide ownership for the same
`cp -r prometheus-files/consoles /etc/prometheus`
`cp -r prometheus-files/console_libraries /etc/prometheus`
`chown -R prometheus:prometheus /etc/prometheus/consoles`
`chown -R prometheus:prometheus /etc/prometheus/console_libraries`

- Create the prometheus.yml file and set the configuration
`vim /etc/prometheus/prometheus.yml`

```
root@prometheus-1:/etc/prometheus# cat prometheus.yml 
global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'prometheus_server'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.122.181:9100']

```

- Change the ownership to prometheus user
`chown prometheus:prometheus /etc/prometheus/prometheus.yml`

- Create and setup a prometheus service file
`vim /etc/systemd/system/prometheus.service`

```
root@prometheus-1:/etc/systemd/system# cat prometheus.service 
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
       --config.file /etc/prometheus/prometheus.yml \
       --storage.tsdb.path /var/lib/prometheus/ \
       --storage.tsdb.retention.time=7d \
       --storage.tsdb.retention.size=8GB \
       --web.console.templates=/etc/prometheus/consoles \
       --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

```

- Reload start and enable the prometheus

```
systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus
systemctl status prometheus
```

- Access the prometheus on port 9090


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jrenivl0umvndd0dm80t.png)

## **Grafana Installation**

- Do apt update and download the Grafana GPG key
`apt update -y`

```
wget -q -O - https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null
```

```
root@prometheus-1:/# apt update -y
Hit:1 http://in.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://in.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://in.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://in.archive.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
36 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@prometheus-1:/# wget -q -O - https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > /dev/null

```

- Install APT Grafana Repo

```
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
```

```
root@prometheus-1:/# echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main
root@prometheus-1:/# apt update
Hit:1 http://in.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://in.archive.ubuntu.com/ubuntu jammy-updates InRelease
Get:3 https://packages.grafana.com/oss/deb stable InRelease [5,983 B]         
Hit:4 http://in.archive.ubuntu.com/ubuntu jammy-backports InRelease           
Hit:5 http://in.archive.ubuntu.com/ubuntu jammy-security InRelease
Get:6 https://packages.grafana.com/oss/deb stable/main amd64 Packages [163 kB]
Fetched 169 kB in 2s (70.4 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
36 packages can be upgraded. Run 'apt list --upgradable' to see them.

```

- Install Grafana 
`apt install grafana -y`

- Start and enable the Grafana service
```
systemctl start grafana-server.service
systemctl enable grafana-server.service
systemctl status grafana-server.service
```

- Find Grafana version

```
root@prometheus-1:/# grafana-server -v
Version 10.2.0 (commit: 895fbafb7a, branch: HEAD)

```

- Access the grafana dashboard on port 3000
login as user:admin & pass:admin and chage the credientials.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yk23i0ugkwiiitflygoc.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mk0i9grvsqthcvv3ol4d.png)

## **Node Exporter Installation**

- Download the node exporter

```
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
```

`tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz`

- move the node exporter folder to /usr/local/bin
`mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/`

- create the group and add the user
`useradd -rs /bin/false node_exporter`
`groupadd node_exporter`

- Create the Node exporter service file
`vim /etc/systemd/system/node_exporter.service`

```
root@prometheus-1:/etc/systemd/system# cat node_exporter.service 
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

```

- reload, Enable & start the Node exporterservice

```
systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter
systemctl status node_exporter
```

- Access the metrics on port 9100
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8oz059xz63ti54ckua39.png)

## **Add Data source**

- You can get the data source from below link
[(https://grafana.com/grafana/dashboards/?plcmt=footer)]
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f200y3wiitmc42f7n0io.png)

- You can add the ID or upload the JSON package on the grafana dashboard.

 **Home >Dashboard>New>Importdashboard**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jg1qwtd4l4xnii6log9e.png)

## **Alert Manager Installation on Prometheus**

- apt update and download the Alert manager and copy the alert manager file to /usr/local/bin

```
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
```

`tar -xvf alertmanager-0.26.0.linux-amd64.tar.gz`

`cd alertmanager-0.26.0.linux-amd64/`

`cp -r . /usr/local/bin/alertmanager`
- Create the alert manager service file

`vim /etc/systemd/system/alertmanager.service`

```
root@prometheus-1:/etc/systemd/system# cat node_exporter.service 
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
root@prometheus-1:/etc/systemd/system# cat alertmanager.service 
[Unit]
Description=Prometheus Alert Manager Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/alertmanager/alertmanager \
    --config.file=/usr/local/bin/alertmanager/alertmanager.yml \
    --cluster.advertise-address="192.168.122.181:9093"

[Install]
WantedBy=multi-user.target

```

- Check the Alert manager with amtool

```
/usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
```

```
root@prometheus-1:~# /usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
Checking '/usr/local/bin/alertmanager/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 1 receivers
 - 0 templates

```

- Reload,start & enable the Alert manager service

```
systemctl daemon-reload
systemctl start alertmanager.service
systemctl enable alertmanager.service
```

- Access the Alert manager on port 9093

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ucar55ilt5rkprb6s6ih.png)

## **Add alert rules**

- adding alert managerconfiguration on prometheus.yml file which we created on /etc/prometheus/

**add the below parameters before the scarpe_configs**

```
# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - localhost:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "/etc/prometheus/rules/*.yml"
```
**SMTP Configure for Alert manager**

- To proceed with these setup we need to enable 2-factor authentication on your "Gmail" account and create the "app password" for (prometheus)and copy the key to the below auth password paramater.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zhbnji25avhn9lfjtrlf.png)
- create and add the paramaters on alertmanager.yml 
`vim /usr/local/bin/alertmanager/alertmanager.yml`

- Fill the "FROM","TO" email and add the auth key generated from the user gmail account.
```
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 24h
  receiver: 'email'
receivers:
- name: 'email'
  email_configs:
  - to: "recepient mail address"@gmail.com
    from: '"usermail address"@gmail.com'
    smarthost: smtp.gmail.com:587
    auth_username: '"usermail address"@gmail.com'
    auth_identity: '"usermail address".com'
    auth_password: '(Key we copied from the appsecurity tab)'
    send_resolved: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

```

## **Alert Manager Rule**

- Create the directories to add the rules.

`mkdir /etc/prometheus/rules`

`vim /etc/prometheus/rules/alert-rules.yml`

```
groups:
- name: alert-rules
  rules:
  - alert: ExporterDown
    expr: up == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      description: 'Metrics exporter service for {{ $labels.job }} running on {{ $labels.instance }} has been down for more than 5 minutes.'
      summary: 'Exporter down (instance {{ $labels.instance }})'

  - alert: HostOutOfDiskSpace
    expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 15 and ON (instance, device, mountpoint) node_filesystem_readonly == 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of disk space (instance {{ $labels.instance }})
      description: "Disk is almost full (< 15% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: HostOutOfMemory
    expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 15
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of memory (instance {{ $labels.instance }})
      description: "Node memory is filling up (< 15% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: HostHighCpuLoad
    expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 85
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host high CPU load (instance {{ $labels.instance }})
      description: "CPU load is > 85%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yojthjsx5sudgf198bbj.png)

## **To check and find the status validation of alertmanager.yml and rules with amtool & promtool**

```
/usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
```

```
promtool check rules /etc/prometheus/rules/alert-rules.yml

```
```
root@prometheus-1:~# /usr/local/bin/alertmanager/amtool check-config /usr/local/bin/alertmanager/alertmanager.yml
Checking '/usr/local/bin/alertmanager/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 1 receivers
 - 0 templates

root@prometheus-1:~# promtool check rules /etc/prometheus/rules/alert-rules.yml
Checking /etc/prometheus/rules/alert-rules.yml
  SUCCESS: 4 rules found

```
**Restart all the services like prometheus,node exporter,Alert manager,grafana**

- We have verified the disk space and added file.img to check the alertmanager and rules.

```
root@prometheus-1:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              392M  1.3M  390M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   12G  7.3G  3.4G  69% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/vda2                          2.0G  251M  1.6G  14% /boot
tmpfs                              392M  4.0K  392M   1% /run/user/1000
root@prometheus-1:~# fallocate -l 2G file.img
root@prometheus-1:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              392M  1.3M  390M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   12G  9.3G  1.4G  88% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/vda2                          2.0G  251M  1.6G  14% /boot
tmpfs                              392M  4.0K  392M   1% /run/user/1000
root@prometheus-1:~# ls

```
-Grafana Dashboard status.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z6uzsjf9lql8y04t4gd0.png)