Prometheus - HTTPS & Authentication - Part 4
Tags: prometheus, observability, cloudnative, opensource
Published: 2023-08-22 15:31:52

> In this article, we will look how we can configure HTTPS and Authentication on both Prometheus and Node Exporter

## Prerequisites

In my previous article, we looked at how we can set up Prometheus and Node Exporter as systemd services on an Ubuntu instance.

{% embed https://iamunnip.hashnode.dev/prometheus-installation-on-amazon-ec2-ubuntu-part-3 %}

But in the above setup, we were accessing the Prometheus expression browser and Node Exporter metrics endpoints via HTTP and there was no authentication enabled.  
We are going to address these issues in this article.

## HTTPS

### Node Exporter

Create a new directory for storing the Node Exporter configuration file and change its ownership to "node\_exporter" user

```bash
$ sudo mkdir -p /etc/node_exporter

$ sudo chown node_exporter:node_exporter /etc/node_exporter
```

Create a configuration file for Node Exporter and change its ownership to "node\_exporter" user

```bash
$ sudo touch /etc/node_exporter/node_exporter.yml

$ sudo chown node_exporter:node_exporter /etc/node_exporter/node_exporter.yml
```

Generate a certificate and key using OpenSSL

```bash
$ sudo openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout prom.key -out prom.crt -subj "/C=US/ST=California/L=Oakland/O=MyOrg/CN=localhost" -addext "subjectAltName = DNS:localhost"

$ ls
prom.crt  prom.key
```

Copy the certificate and key files to the Node Exporter configuration directory and change their ownership to "node\_exporter" user

```bash
$ sudo cp prom.* /etc/node_exporter

$ sudo chown node_exporter:node_exporter prom.crt

$ sudo chown node_exporter:node_exporter prom.key
```

Add the tls\_server\_config details to the configuration file

```bash
$ sudo vi /etc/node_exporter/node_exporter.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key
```

Update the systemd unit file of node\_exporter service to include the above configuration file

```bash
$ sudo vi /etc/node_exporter/node_exporter.yml

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
        --web.config.file /etc/node_exporter/node_exporter.yml

[Install]
WantedBy=multi-user.target
```

Restart the node\_exporter service and verify its status

```bash
$ sudo systemctl restart node_exporter

$ sudo systemctl status node_exporter
```

The metrics endpoint is now accessible via HTTPS

![node-2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zqba4xwmnbklzxywauv9.png)

Open Prometheus expression browser and navigate to **Status** -&gt; **Targets**, we can see our node\_exporter target is showing down

![prom-7](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m1x43xx5icsomhxb4ecx.png)

### Prometheus

Copy the certificate and key files to the Prometheus configuration directory and change its ownership to "prometheus" user

```bash
$ sudo cp prom.* /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus/prom.crt

$ sudo chown prometheus:prometheus /etc/prometheus/prom.key
```

Update the Prometheus configuration file to include scheme and tls\_config for the "node\_exporter" job

```bash
$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "node_exporter"
    scheme: https
    tls_config:
      ca_file: prom.crt
      insecure_skip_verify: true
    static_configs:
      - targets: ["172.31.81.113:9100"]
```

Validate the configuration file using the promtool

```bash
$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
```

Restart the prometheus service to take effect the new configuration changes

```bash
$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus
```

Open Prometheus expression browser and navigate to **Status** -&gt; **Targets**, we can see our node\_exporter target is showing up

![prom-8](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fr6dy6hble38e9eb0kck.png)

Now we have enabled secure communication between the Prometheus server and Node Exporter but still our Prometheus expression browser is using an HTTP connection

Create a new configuration file for configuring HTTPS connection and change its ownership to "prometheus" user

```bash
$ sudo touch /etc/prometheus/webconfig.yml

$ sudo chown prometheus:prometheus /etc/prometheus/webconfig.yml
```

Add the tls\_server\_config details to the newly created configuration file

```bash
$ sudo vi /etc/prometheus/webconfig.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key
```

Update the systemd unit file of prometheus service to include the above configuration file

```bash
$ sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries \
    --web.config.file /etc/prometheus/webconfig.yml

[Install]
WantedBy=multi-user.target
```

Now we can access the Prometheus expression browser using HTTPS

![prom-11](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2s5eqtm2a4r8nl1tqef5.png)

## Authentication

Install the apache2 utils package to generate a password

```bash
$ sudo apt update

$ sudo apt install apache2-utils
```

Generate the password using the htpasswd tool

```bash
$ htpasswd -nBC 16 admin | tr -d ':\n'
New password:
Re-type new password:
admin$2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O
```

### Node Exporter

Update the Node Exporter configuration file to include basic authentication

```bash
$ sudo vi /etc/node_exporter/node_exporter.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key
basic_auth_users:
  admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O
```

Restart the node\_exporter service and verify its status

```bash
$ sudo systemctl restart node_exporter

$ sudo systemctl status node_exporter
```

Now access the Node Exporter metrics endpoint and it will show a login prompt

![node-3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nlfokvo386li7muquuxj.png)

Open Prometheus expression browser and navigate to **Status** -&gt; **Targets**, we can see our node\_exporter target is showing down

![prom-9](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8tmlz7072lueitk44od3.png)

### Prometheus

Update the Prometheus configuration file to include basic authentication for the "node\_exporter" job

```bash
$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "node_exporter"
    scheme: https
    tls_config:
      ca_file: prom.crt
      insecure_skip_verify: true
      basic_auth:
        username: admin
        password: Password!
    static_configs:
      - targets: ["172.31.81.113:9100"]
```

Validate the configuration file using the promtool

```bash
$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
```

Restart the prometheus service to effect new configuration changes

```bash
$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus
```

Open the Prometheus expression browser and navigate to Status -&gt; Targets, we can see our node\_exporter target is showing up

![prom-10](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1j3ndzkjz2rbdbapbz4d.png)

Now we have enabled basic authentication between the Prometheus server and Node Exporter but we need to enable authentication for the Prometheus server

Update the below configuration file to include basic authentication on the Prometheus server

```bash
$ sudo vi /etc/prometheus/webconfig.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key

basic_auth_users:
  admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O
```

Restart the prometheus service for the new changes and check its status

```bash
$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus
```

Now access the Prometheus expression browser and it will show a login prompt

![prom-12](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mzrut58n40cju1bq7qmz.png)

That's all for now

## Reference

[https://prometheus.io/docs/prometheus/latest/configuration/https/](https://prometheus.io/docs/prometheus/latest/configuration/https/)

[https://kodekloud.com/courses/prometheus-certified-associate-pca/](https://kodekloud.com/courses/prometheus-certified-associate-pca/)