Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3
Tags: prometheus, observability, cloudnative, opensource
Published: 2023-08-21 16:48:56

In my previous article, we looked at how we can quickly set up Prometheus and Node Exporter.

{% embed https://iamunnip.hashnode.dev/prometheus-quick-setup-on-amazon-ec2-ubuntu-part-2 %}

But in that setup, Prometheus and Node Exporter processes are running in the foreground and it won't start the process after a reboot of the VM.  
We are going to fix that issue in this article.

## Prerequisites

* Setup an EC2 instance of type t2.small
    
* Ubuntu 22.04 LTS as AMI
    
* 30 GB of hard disk space
    
* Open ports 22 for SSH, 9090 for Prometheus and 9100 for Node Exporter
    

## Installation

### Prometheus

Login to your EC2 instance

```bash
$ ssh -i prometheus.pem ubuntu@3.80.133.119
```

```bash
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy
```

Create a new user for managing the Prometheus process  
Upon executing the command it will create a user called prometheus without a home directory and shell as /bin/false

```bash
$ sudo useradd --shell /bin/false --no-create-home prometheus
```

Create a new directory for storing our Prometheus configuration file and update the ownership of the directory

```bash
$ sudo mkdir -p /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus
```

Create another directory for storing Prometheus time series data and update the ownership of the directory

```bash
$ sudo mkdir -p /var/lib/prometheus

$ sudo chown prometheus:prometheus /var/lib/prometheus
```

Download and extract the latest Prometheus release from their GitHub page

```bash
$ wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz
```

```bash
$ tar -xzvf prometheus-2.46.0.linux-amd64.tar.gz
```

Change the directory and install Prometheus and Promtool binaries

```bash
$ cd prometheus-2.46.0.linux-amd64

$ sudo install prometheus /usr/local/bin

$ sudo install promtool /usr/local/bin
```

Change the permission of these binaries to prometheus user

```bash
$ sudo chown prometheus:prometheus /usr/local/bin/prometheus

$ sudo chown prometheus:prometheus /usr/local/bin/promtool
```

Check the version of prometheus or promtool binaries

```bash
$ prometheus --version
prometheus, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:31:24
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels
```

```bash
$ promtool --version
promtool, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:31:24
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels
```

Copy the consoles and console\_libraries directory which is used for dashboarding and visualization

```bash
$ sudo cp -rv consoles /etc/prometheus

$ sudo cp -rv console_libraries /etc/prometheus
```

Change the ownerships of the directories recursively to prometheus user

```bash
$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles

$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
```

Copy the configuration file to the above-mentioned directory and change its ownership to prometheus user

```bash
$ sudo cp prometheus.yml /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
```

Create a systemd unit file for the Prometheus service

```bash
$ sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
	--config.file /etc/prometheus/prometheus.yml \
	--storage.tsdb.path /var/lib/prometheus \
	--web.console.templates /etc/prometheus/consoles \
	--web.console.libraries /etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
```

Start the prometheus process and enable the process at boot

```bash
$ sudo systemctl daemon-reload

$ sudo systemctl start prometheus

$ sudo systemctl enable prometheus
```

Check the prometheus service status

```bash
$ sudo systemctl is-active prometheus
active

$ sudo systemctl is-enabled prometheus
enabled
```

Open http://3.80.133.119:9090 in the browser and you can see the Prometheus expression browser


![prom-1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zjotngacehe1bqetkyrz.png)

### Node Exporter

Create a new user for managing the node exporter process

```bash
$ sudo useradd --shell /bin/false --no-create-home node_exporter
```

Download and extract the latest Node Exporter release from their GitHub page

```bash
$ wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

$ tar -xzvf node_exporter-1.6.1.linux-amd64.tar.gz
```

Change the directory and install the node\_exporter binary

```bash
$ cd node_exporter-1.6.1.linux-amd64

$ sudo install node_exporter /usr/local/bin
```

Change the ownership of the binary to node\_exporter user

```bash
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
```

Check the version of the node\_exporter binary

```bash
$ node_exporter --version
node_exporter, version 1.6.1 (branch: HEAD, revision: 4a1b77600c1873a8233f3ffb55afcedbb63b8d84)
  build user:       root@586879db11e5
  build date:       20230717-12:10:52
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo osusergo static_build
```

Create a systemd unit file for the Node Exporter service

```bash
$ sudo vi /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
```

Start the node\_exporter process and enable the process at boot

```bash
$ sudo systemctl daemon-reload

$ sudo systemctl start node_exporter

$ sudo systemctl enable node_exporter
```

Check the status of the node\_exporter process

```bash
$ sudo systemctl is-active node_exporter
active

$ sudo systemctl is-enabled node_exporter
enabled
```

Open http://3.80.133.119:9100/metrics in your browser to view all the metrics


![node-1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1d5bhojhxunyyzkd09zo.png)

## Configuration

### Prometheus

Now we have installed Prometheus and Node Exporter on the server and we need to modify the Prometheus configuration file to add new targets to scrape metrics

In the installation step, we already copied a default Prometheus configuration file to the /etc/prometheus location. You can edit this file or replace it with the below contents.

Here I have added a new job called "node\_exporter" to scrape metrics from the same VM, instead of using localhost I have used the private IP of the instance

```bash
$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "node_exporter"
    static_configs:
      - targets: ["172.31.93.117:9100"]
```

Validate the configuration file using the promtool

```bash
$ promtool check config /etc/prometheus/prometheus.yml

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
```

Restart the Prometheus service to effect new configuration changes

```bash
$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus
```

Open the expression browser and navigate to **Status** -&gt; **Targets** to view the newly added "node_exporter" target


![prom-2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bjvowuazmp554bg92pri.png)

We can view an example metric called "node\_memory\_Active\_bytes" using the expression browser


![prom-3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i4tgt8kmbn516d316pp6.png)

Select the Graph option to view it as a graph


![prom-4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d4pn81lnqvgj4svh2mq7.png)

That's all for now

## Reference

[https://prometheus.io/docs/prometheus/latest/getting\_started/](https://prometheus.io/docs/prometheus/latest/getting_started/)

[https://kodekloud.com/courses/prometheus-certified-associate-pca/](https://kodekloud.com/courses/prometheus-certified-associate-pca/)