Unlocking Linux Host Performance Insights with Prometheus Monitoring (Part-1)
Tags: prometheus, linux, monitoring, devops
Published: 2023-07-14 07:19:26

## Table Of Contents
 * [Introduction](#introduction)
 * [Importance of monitoring Linux host metrics](#Importance of monitoring Linux host metrics)
 * [How Prometheus can help?](#how prometheus help)
 * [What is Node Exporter and its role ?](#What is Node Exporter and its role ?)
 * [Setting up Prometheus server to monitor Linux hosts](#Setting up Prometheus server to monitor Linux hosts)
 * [Install and Configure Node Exporter on the  Server]
(#Install and Configure Node Exporter on the  Server)
 

#Introduction:<a name="introduction"></a>

Welcome to part 1 of our 2-part blog series on Linux host monitoring with Prometheus. This series is designed for DevOps professionals, monitoring enthusiasts, and Prometheus Certified Associate aspirants.

In this series, we'll explore the importance of monitoring Linux host metrics and how Prometheus can help. We'll cover topics such as exporting metrics through node exporter and how prometheus scrapes metrics.

Stay tuned for part 2, where we'll dive into practical implementation, configuration, and creating dashboards with grafana for monitoring Linux hosts with Prometheus.

Get ready to elevate your DevOps and monitoring skills as we unlock the power of Linux host monitoring with Prometheus. Let's begin this journey together.

#Importance of monitoring Linux host metrics:<a name="Importance of monitoring Linux host metrics"></a>

Monitoring Linux host metrics is crucial for optimizing performance, planning capacity, troubleshooting issues, ensuring security, and proactively maintaining systems. By tracking metrics like CPU, memory, disk, and network usage, you can identify problems, make data-driven decisions, and ensure the smooth operation of your Linux environment.

#How Prometheus can help?<a name="how prometheus help"></a>

Prometheus is a powerful monitoring tool that simplifies the process of monitoring Linux host metrics. It collects data from various sources using lightweight exporters like **Node Exporter** and stores it in a time-series database. With PromQL, you can query and analyze the collected metrics, enabling you to retrieve specific Linux host metrics and gain insights into system performance. 

#What is Node Exporter and its role ?<a name="What is Node Exporter and its role ?"></a>



![Prometheus Server scraping metrics from Node Exporter service running on linux hosts](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/58vul3p9tktczjj60gvx.gif)
**Prometheus Server scraping metrics from Node Exporter service running on Linux hosts**

Linux doesn't provide native Prometheus metrics out of the box. However, Node Exporter comes to the rescue by bridging this gap. Node Exporter is an exporter specifically designed for Linux systems, serving as a reliable source for collecting Linux host metrics.

Node Exporter runs as a service on the Linux host, exposing a wide range of metrics related to system resources, network, disk, CPU, memory, and more. It periodically collects these metrics from the Linux kernel and other system sources, making them available for scraping by Prometheus.

By installing and configuring Node Exporter on Linux hosts, you can unlock a wealth of Linux-specific metrics that would otherwise be inaccessible to Prometheus. This includes detailed information about CPU utilization, memory usage, disk I/O, network traffic, and other crucial indicators of system performance.

Node Exporter acts as a bridge between the Linux operating system and Prometheus, ensuring that Linux host metrics can be easily monitored and analyzed using Prometheus' powerful features. It provides a standardized and reliable way to expose Linux-specific metrics, enabling seamless integration with the Prometheus ecosystem.

With Node Exporter, system administrators and DevOps teams can gain valuable insights into the performance and health of their Linux hosts, allowing them to proactively optimize resource allocation, troubleshoot issues, and ensure the smooth operation of their Linux-based infrastructure.

#Setting up Prometheus server to monitor Linux hosts<a name="Setting up Prometheus server to monitor Linux hosts"></a>

   
###Download and install Prometheus


#####1.Create the `prometheus` user

```
sudo useradd -M -r -s /bin/false prometheus
```
#####2.Create prometheus directories

```
sudo mkdir /etc/prometheus /var/lib/prometheus
```
#####3. Then download latest binary archive for Prometheus.
```
mkdir -p /tmp/prometheus && cd /tmp/prometheus

curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -
```
#####4. Untar the the downloaded binary
```
tar xvf prometheus*.tar.gz
prometheus-2.45.0.linux-amd64/
prometheus-2.45.0.linux-amd64/LICENSE
prometheus-2.45.0.linux-amd64/prometheus.yml
prometheus-2.45.0.linux-amd64/console_libraries/
prometheus-2.45.0.linux-amd64/console_libraries/prom.lib
prometheus-2.45.0.linux-amd64/console_libraries/menu.lib
prometheus-2.45.0.linux-amd64/consoles/
prometheus-2.45.0.linux-amd64/consoles/node-overview.html
prometheus-2.45.0.linux-amd64/consoles/node-cpu.html
prometheus-2.45.0.linux-amd64/consoles/index.html.example
prometheus-2.45.0.linux-amd64/consoles/node.html
prometheus-2.45.0.linux-amd64/consoles/node-disk.html
prometheus-2.45.0.linux-amd64/consoles/prometheus-overview.html
prometheus-2.45.0.linux-amd64/consoles/prometheus.html
prometheus-2.45.0.linux-amd64/promtool
prometheus-2.45.0.linux-amd64/prometheus
prometheus-2.45.0.linux-amd64/NOTICE
```
#####5. Move the binary files to /usr/local/bin/ directory and set their ownership to `prometheus` user.

```
sudo mv prometheus promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/{prometheus,promtool}
```
#####6. Move the files from the tmp archive to the appropriate locations, and set ownership on these files and directories to the `prometheus` user

```
sudo cp -r prometheus-2.45.0.linux-amd64/{consoles,console_libraries} /etc/prometheus/

sudo cp prometheus-2.45.0.linux-amd64/prometheus.yml /etc/prometheus/prometheus.yml

sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
```
#####7. Run Prometheus in the foreground to make sure everything is set up correctly so far:

```
prometheus --config.file=/etc/prometheus/prometheus.yml
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zm7pmpq6h8opcjupjpwa.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y9fvj2lp8jp8m7bi42sn.png)
#####8. Configure Prometheus as a systemd Service

Create a systemd unit file for Prometheus:

```
sudo vim /etc/systemd/system/prometheus.service
```

```
[Unit] Description=Prometheus Time Series Collection and Processing Server Wants=network-online.target After=network-online.target
 
[Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --storage.tsdb.path /var/lib/prometheus/ \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries 

[Install] WantedBy=multi-user.target
```
#####9. Make sure systemd picks up the changes we made:

```
sudo systemctl daemon-reload
```

#####10. Start the Prometheus service:
```
sudo systemctl start prometheus
```

#####11. Enable the Prometheus service so it will automatically start at boot:

```
sudo systemctl enable prometheus
```

#####12. Verify the Prometheus service is healthy:

```
sudo systemctl status prometheus
```
We should see its state is active (running).

#####13. Make an HTTP request to Prometheus to verify it is able to respond:

curl localhost:9090

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x8706tvu63vw38t5sokv.png)

#####14. In a new browser tab, access Prometheus by navigating to http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090 (replacing <PROMETHEUS_SERVER_PUBLIC_IP> with the IP listed on the lab page). We should then see the Prometheus expression browser.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kr6grcr124409bo48dkb.png)

#Install and Configure Node Exporter on the  Server<a name="Install and Configure Node Exporter on the  Server"></a>

#####1. Create a user and group that will be used to run Node Exporter

```
sudo useradd -M -r -s /bin/false node_exporter
```

#####2. Get the node exporter binary
```
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz
```
#####3. Extract the Node Exporter binary:
```
tar -xvzf node_exporter-1.6.0.linux-amd64.tar.gz
```
#####4. Copy the Node Exporter binary to the appropriate location
```
sudo cp node_exporter-1.6.0.linux-amd64/node_exporter /usr/local/bin/
```
#####5. Set ownership on the Node Exporter binary
```

sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
```

#####6. Create a systemd unit file for Node Exporter:

```
sudo vim /etc/systemd/system/node_exporter.service
```
#####7.Define the Node Exporter service in the unit file
```
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
```

#####8. Make sure systemd picks up the changes we made:
```
sudo systemctl daemon-reload
```
#####9. Enable the node_exporter service so it will automatically start at boot:
```
sudo systemctl enable node_exporter
```
#####10. Test that your Node Exporter is working by making a request to it from localhost:
```
curl localhost:9100/metrics
```
#####11.Configure Prometheus to Scrape Metrics from the Acme Web Server

Edit the Prometheus config file:
```
sudo vim /etc/prometheus/prometheus.yml
```
Locate the scrape_configs section and add the following beneath it (ensuring it's indented to align with the existing job_name section):

```
...

  - job_name: 'LimeDrop Web Server'
    static_configs:
    - targets: ['10.0.1.102:9100']

...
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7kkleogeud3n8io4qa9u.png)

#####11. Restart Prometheus to load the new config:
```
sudo systemctl restart prometheus
```
#####12. Navigate to the Prometheus expression browser in your web browser using the public IP address of your Prometheus server:
```
<PROMETHEUS_SERVER_PUBLIC_IP>:9090
```
#####13.In the expression field (the box at the top of the page), paste in the following query to verify you are able to get some metric data from the LimeDrop web server:
```
node_filesystem_avail_bytes{job="Acme Web Server"}
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0z010gsqd5p07a7j33qa.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lmitx3x1esxpquo8q84p.png)


In the second part of our blog series on Linux host monitoring with Prometheus, we will delve into more advanced concepts of Node Exporter and explore how to create insightful dashboards and queries.