Monitoring Systems with Prometheus and Grafana: A Comprehensive Guide
Tags: devops, monitoring, dashboard
Published: 2023-07-25 12:22:21

In this blog we will be walking through a step-by-step guide to build your monitoring system for your devices. In this guide we will be setting up a Prometheus server and Grafana dashboard to analyze your systems resource utilization. </br>
We will be deploying our Prometheus and Grafana in docker containers because they are lightweight and easy to manage.

## Why is System Monitoring Important?
System monitoring is crucial for several reasons, as it provides valuable insights and benefits that contribute to the overall health, performance, and security of IT infrastructures and applications. Here are some key reasons why system monitoring is important:

- Early Detection of Issues
- Performance Optimization
- Capacity Planning
- Security and Compliance
- Troubleshooting and Root Cause Analysis
- SLA and KPI Tracking
- Resource Optimization and Cost Reduction
- Trend Analysis and Planning

## What is Prometheus
Prometheus is a monitoring and alerting toolkit designed to collect and store time-series data from various sources, such as applications, services, and system components. It is particularly well-suited for monitoring highly dynamic and distributed environments. Prometheus scrapes data from designated endpoints at regular intervals, allowing users to monitor the health and performance of their systems. It uses a query language called PromQL to extract and analyze data. Prometheus also supports alerting based on predefined rules, enabling users to receive notifications when certain conditions are met.

## What is Grafana
Grafana is a powerful visualization and analytics platform that works seamlessly with Prometheus and other data sources. It allows users to create custom dashboards and visualizations using various data sources, including Prometheus. Grafana provides a user-friendly interface for exploring and understanding complex data sets. It supports a wide range of charts, graphs, and panels, making it easy to display data in a meaningful way. Grafana is commonly used to create real-time monitoring dashboards, performance reports, and executive summaries, helping teams and organizations gain valuable insights into their systems' health and performance

## Setting up the environment for Prometheus and Grafan
- enabling ssh
```bash
sudo apt-get install openssh-server
sudo systemctl enable ssh
sudo systemctl start ssh
sudo ufw allow ssh
sudo ufw enable
sudo ufw status
```
- installing docker
```bash
sudo apt install docker.io
```
## Configuring Prometheus and Grafana server on ubuntu
- create a directory to store all the configuration files which will be mounted to the docker container
```bash
cd /opt/
mkdir prometheus
cd prometheus
touch prometheus.yml
touch alerts.rules
```
> alerts rules and alert manager will be discussed in the next post

- create docker container for prometheus
```bash
docker run -d -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/alerts.rules:/etc/prometheus/alerts.rules --restart unless-stopped --name prometheus prom/prometheus
```

- creating a node exporter
a node exporter helps in retrieving the metrics of a Linux based system and send it to prometheus
```bash
docker run -d -p 9100:9100 -v --name node-exporter prom/node-exporter
```

- create docker container for grafana
```bash
docker run -d --name=grafana -p 3000:3000 grafana/grafana
```
> now open any browser and search for `ubuntu_ip:9090`, `ubuntu_ip:3000` and `ubuntu_ip:9100/metrics` for checking whether the prometheus, grafana and node exporter are running or not
> ubuntu_ip is the IP address of the machine where the docker container is created

> if you want to monitor your windows machine you can install a windows exporter from [here](https://github.com/prometheus-community/windows_exporter/releases/tag/v0.23.1). After installing you can start the service to start the windows exporter

### Configuring Prometheus Server
```bash
vim /opt/prometheus/prometheus.yml
```
`prometheus.yml`
```bash
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['ubuntu_id:9100']
  # ubuntu_ip is the IP address where node exporter is installed

  # for windows exporter
  - job_name: 'win-exporter'
    static_configs:
      - targets: ['windows_ip:9182']
  # windows_ip is the IP address of the window machine
```
```bash
docker restart prometheus
```

- now you will be able to use PromQL to gain insights of you Linux machine by visiting `ubuntu_ip:9090`

![Prometheus UI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ktg4gjj3bi3erdkjoa73.png)

### Configuring Grafana Dashboard
open `ubuntu_ip:3000` on your browser 

![Grafana login](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wfi9s0s1uyceca9kqx4x.png)

now use the default username and password `admin` and `admin` and after that add prometheus to your datasource

![Grafana Home](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/31pv6m3fqxxxhhnyanwn.png)

![Grafana Datasource](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yd2i3v54cn4o4p5liomo.png)
After adding the datasource build a dashboard for your node exporter using the following library [Grafana Dashboards](https://grafana.com/grafana/dashboards/)

> Thanks for reading the blog I hope it was helpful for you. In the next blog we will be configuring our Prometheus Server to send alerts based on some rules.