Monitoring Linux instances with Prometheus and Grafana
Tags: devops, performance, go, webdev
Published: 2023-01-20 23:52:29

If you want to know how is everything going in your servers, you must monitor them, yeah, sometimes read htop or similar commands output can be hard to understand, there are a lot of alternatives, and today I am going to show you how to set up a monitoring suite to always know what is going on your servers, linux instances, and more.

In this tutorial, I will show you how to mount a complete, flexible, and scalable monitoring system, using two great (and free) tools.

## What are we going to use?
**Prometheus** is a powerful, open-source and easy-to-use monitoring solution, it is going to help us to collect and fetch a lot of information from your Linux instances.
**Grafana** on the other hand, is the most popular technology to create monitoring dashboards in a fast and easy way, also it has a lot of plugins and integrations, therefore, if it exists, you can create a Grafana panel for it.

Let's start!

### Installing node_exporter
First of all, we will need to install a Prometheus light-weight daemon in all your servers that you want to monitor, this daemon is node_exporter.
Let's install it.
Connect to your server

```
$ ssh user@target-instance
```

**Download the node_exporter files**

```
$ wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
Uncompress and rename the folder
$ tar xvf node_exporter-1.0.1.linux-amd64.tar.gz
$ mv node_exporter-1.0.1.linux-amd64 ne_files
```

**Create a new user**

```
$ sudo useradd --no-create-home -s /bin/false node_exporter
Change the node_exporter binary ownership to your new user
$ sudo chown node_exporter:node_exporter ./ne_files/node_exporter
```
Let's add node_exporter to your local binaries

```
$ sudo mv ./ne_files/node_exporter /usr/local/bin/
```
Now we are adding node_exporter as a new service
Create a service file for `node_exporter` with

```
$ sudo vim /etc/systemd/system/node_exporter.service
```
Then, add this to the new file, do not forget to save your changes.

```
[Unit]
Description=node_exporter
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
```
Reload the services daemon to see your new service

```
$ sudo systemctl daemon-reload
```
Start it!

```
$ sudo systemctl start node_exporter
```
Ok, but how do I know it is working?
You can use:

```
$ sudo systemctl status node_exporter
```
Or go to your favorite browser and navigate to
`http://<target-instance>:9100/metrics`

If everything is working, you should see something similar to

```
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.000108492
go_gc_duration_seconds{quantile="0.25"} 0.000108492
go_gc_duration_seconds{quantile="0.5"} 0.000108492
go_gc_duration_seconds{quantile="0.75"} 0.000108492
go_gc_duration_seconds{quantile="1"} 0.000108492
go_gc_duration_seconds_sum 0.000108492
go_gc_duration_seconds_count 1
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 8
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.846384e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.689864e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.445623e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
```
Remember, you need to repeat this process within every Linux instance that you want to monitor.

### Installing Prometheus
Once all your target instances are running node_exporter, we will install Prometheus to scrape and understand your server's metrics. 
You can install it either in your `localhost` or on another server. In this tutorial, we are installing Prometheus in a new server that will monitor everything. Don't worry if you want to install the monitoring suite in your `localhost`; both processes are very similar.

Connect to your monitoring server

```
$ ssh user@<monitoring-server>
```
Download the Prometheus files from here

```
$ wget https://github.com/prometheus/prometheus/releases/download/v2.23.0/prometheus-2.23.0.linux-amd64.tar.gz
```
Now, let's `untar` and rename the Prometheus files

```
$ tar xvf prometheus-2.23.0.linux-amd64.tar.gz
$ mv prometheus-2.23.0.linux-amd64 p_files
```
Create a new user to use Prometheus

```
$ sudo useradd --no-create-home -s /bin/false prometheus
```
Change the Prometheus binaries ownership

```
$ sudo chown prometheus:prometheus p_files/prometheus
$ sudo chown prometheus:prometheus p_files/promtool
```
Move Prometheus binaries to your local binaries

```
$ sudo mv p_files/prometheus /usr/local/bin
$ sudo mv p_files/promtool /usr/local/bin
```
Create folders to store Prometheus configuration

```
$ sudo mkdir /etc/prometheus
$ sudo mkdir /var/lib/prometheus
```
Do not forget to change the directories ownership

```
$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus
```
Copy configurations to your new folders

```
$ sudo cp -r p_files/consoles /etc/prometheus
$ sudo cp -r p_files/console_libraries /etc/prometheus
```
As you might guess, yes, you have to change the directories ownership

```
$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
```
Create a prometheus.yml file within `/etc/prometheus` folder

```
$ sudo vim /etc/prometheus/prometheus.yml
```
Add this to `prometheus.yml`

```
global:
 scrape_interval: 10s
scrape_configs:
 - job_name: 'prometheus'
 scrape_interval: 5s
 static_configs:
 - targets: ['<target-instance-1>:9090', '<target-instance-2:9100>']
```
This is a basic Prometheus configuration file. You must add every instance with the node_exporter daemon running in the "targets" array.
Once the `prometheus.yml` file is ready, change its ownership

```
$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
```
Now, let's create a Prometheus service file

```
$ sudo vim /etc/systemd/system/prometheus.service
```
Add the following configuration to `prometheus.service`

```
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
 --config.file /etc/prometheus/prometheus.yml \
 --storage.tsdb.path /var/lib/prometheus/ \
 --web.console.templates=/etc/prometheus/consoles \
 --web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
```
Reload the service daemon

```
$ sudo systemctl daemon-reload
```
It's time to run your server!

```
$ sudo systemctl start prometheus
```
Check your service status

```
$ sudo systemctl status prometheus
```
Now, **let's check the Prometheus Web UI**

Open your favorite browser and navigate to
`http://<monitoring-server-ip>:9090/`
Now, you should see something like this,


![Prometheus web ui](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bn2irp523toohdlwwu99.png)


Let's run a simple query to see if Prometheus is retrieving your instances metrics correctly.
Copy the following query into Prometheus Expression input, then click `Execute`.

```
sum by (mode, instance) (irate(node_cpu_seconds_total{job="Prometheus",mode="user"}[30s]))*100/4
```
Then, you should see a similar output


![Query output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7ztp7zm0exdkuw7htx19.png)


As you can see, Prometheus also lists all the instances you added in the targets array before.


### Installing Grafana
We have installed node_exporter in your instance to export its metrics.
Then we installed Prometheus on a different server to interpret the instance's metrics. Now, we will install Grafana to create dashboards and charts to make the information easier to read.
The Grafana installation is pretty short. If you want (or need) to install it using the standalone binaries, following the previous steps should give you an idea.
Grafana has a "ready to download and install" deb package; using the following snippet should install Grafana as a new service.

```
$ sudo apt-get install -y adduser libfontconfig1
$ wget https://dl.grafana.com/oss/release/grafana_7.3.4_amd64.deb
$ sudo dpkg -i grafana_7.3.4_amd64.deb
```
Once Grafana installation is done, run the service with

```
$ sudo systemctl start grafana-server
```
Now open a new tab on your browser and navigate to
`http://<monitoring-server>:3000`
You should see a login screen.

![Grafana login screen](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fx36ps9ugvg8assh2ych.png)


The default username and password are always "admin" after you log in, Grafana will ask you to change your admin password. Remember to use a strong password, please.

Now, let's go to setup a new Data Source to fetch information from our Prometheus service.

Click the gear icon on the left bar, then click the Data Sources section.


![Data sources config](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mchveyqli6vr22ayp1kw.png)


Then click the Add data source button


![Data Sources page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pa1h2m3w6d8j5hy5s8j4.png)


As you might guess, yes, we will add a new Prometheus data source


![Add Prometheus](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ybplw4qcblrglrkccvfa.png)

You should see a similar screen


![Data Source configuration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xwstvih3msltz3ywe6io.png)

Set the HTTP URL to use your Prometheus service, by default it is set to localhost, which is true in this case, because Grafana and Prometheus are running in the same host.

Then scroll down and click on "Save and test"


![Save the data source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6hfy20g710bznaxo0bnk.png)

Now, navigate to Manage dashboard sections, click the dashboard button on the left bar, then click on "Manage" button.


![Prometheus Data Source is already working](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7eyo1sy65f0fv7qb6vto.png)

In this screen, click on Import


![Dashboards Page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ippktbm69c2eg1tq4uw.png)


You should get this screen, as you can see, there are some ways to import dashboards, click on Upload JSON file.


![Import a dashboard JSON file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p6cbw7nt2h46g6o4wx5x.png)

You can find a lot of examples on internet or, if you prefer, you can create your own custom dashboard. For this tutorial we are using [this](https://grafana.com/api/dashboards/159/revisions/5/download) dashboard, developed by **Grafana** Labs.
Download the [JSON](https://grafana.com/api/dashboards/159/revisions/5/download) file and select it at the file explorer, then this window should be displayed.
Note: You can find a lot of dashboards [here](https://grafana.com/grafana/dashboards/).


![Dashboard configuration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/43f0tjykf0rroptfaz2m.png)

Just keep going with the dashboard settings, remember to choose the right Prometheus dashboard at the "Prometheus" dropdown, then, click on import.


![Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ro4ld6rpno28f4vesy5l.png)


And now, your dashboard is ready!
At the upper left corner, is a "Host" dropdown, there you can select any of your instances added to the Prometheus configuration file.
Yes, those are a lot of charts, Grafana allows you to remove or add as many as you want, so, get fun creating your own monitoring dashboard!
Thanks for reading, this is my first tutorial/blog ever! I will appreciate any feedback or comment.