Adding Loki From DigitalOcean's Loki 1-Click Installer To Grafana In Your DigitalOcean 1-Click Prometheus Monitoring Stack
Tags: kubernetes, cloud, monitoring, devops
Published: 2021-12-18 12:32:10

## TL;DR

🍬 Set your Loki HTTP URL to:

```
http://loki.loki-stack.svc.cluster.local:3100
```

## Contents
- [Prerequisites](#prerequisites)
- [Introduction](#introduction)
- [Add Loki As A Data Source](#add-loki-as-a-data-source)
- [How Does It Work?](#how-does-it-work)
- [Troubleshooting](#troubleshooting)
  - [502: Bad Gateway](#502-bad-gateway)
  - [Data source connected, but no labels received](#data-source-connected-but-no-labels-received)

_Note:_ This guide assumes you do not have another Loki service running, and that the Loki service installed by DigitalOcean is called `loki`.

## Prerequisites
- A DigitalOcean account.
- A kubernetes cluster set up on DigitalOcean.
- Prometheus Monitoring Stack already installed **as a 1-click application**.
- Loki stack already installed **as a 1-click application**.
- Administrator privileges in the Grafana dashboard which exists in the `kube-prometheus-stack` namespace.
- DNS resolver (comes as standard with DigitalOcean's kubernetes service).
- You're using the defaults of the 1-click installers.
- You can port forward the Grafana service using `kubectl` _(needs to be installed independently on your local machine, and connected to your cluster)_ to access your Grafana dashboard, e.g: ```
kubectl port-forward svc/kube-prometheus-stack-grafana -n kube-prometheus-stack 3000:80
```
  - _Note:_ This example assumes port `3000` is available on your machine. If it's already in use, change the `3000:80` part of the command to something else, followed by `:80`, e.g. `5000:80`.
  - You can then access your Grafana dashboard by navigating to: `http://localhost:3000` in your browser, or whatever you've set your local port as.

## Introduction

DigitalOcean allows you to install a Prometheus monitoring stack using a 1-click installation option, alongside a Loki stack.

The problem is, you end up with two sets of Grafana dashboards that are disjoint from each other; one in the `kube-prometheus-stack` namespace created by the Prometheus Monitoring Stack's 1-click installer, and one in the `loki-stack` namespace created by the Loki 1-click installer.

My goal was to have the Grafana dashboard in the `kube-prometheus-stack` display logs from the Loki service running under the `loki-stack` namespace.

Here's how I did it...

## Add Loki As A Data Source

1. From your browser, navigate to the Grafana dashboard which exists in the `kube-prometheus-stack` namespace (see [prerequisites](#prerequisites) if you're not sure how). On the side navigation panel of the Grafana UI, hover over ⚙ `Configuration` to reveal a sub menu. Inside this sub menu, select `Data sources`. ![Adding a data source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/htnru2w62syuci4qwgnu.png)

2. Click on the `Add data source` button. ![Clicking on add data source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ymhup8trqitlxarncuxw.png) 

3. Find `Loki` as a data source and click `Select`. _Note: You may need to search for_ `Loki` _in the search bar to find it._ ![Selecting Loki as a data source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ikrufd1dvquwad84lh0.png) 
 
4. The only field you need to enter is the `URL` field. Use the following value: ```
  http://loki.loki-stack.svc.cluster.local:3100
  ``` Afterwards, click `Save & test` to apply your settings. ([Didn't work?](#troubleshooting)) ![Loki data source configuration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/svktm5gmkbcx648d5i52.png
) _Note_: If you have a previous Loki data source added, rename this new source aptly, then proceed with your chosen name for step 8.

5. **If you already have a dashboard created, skip this and go to step 6.**
Create a new dashboard by hovering over the ➕ `Create` menu item on the left navigation bar. In the sub menu revealed, select `Dashboard`. ![Creating a new dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0z61rwhictubboh6uj2w.png)

6. Click the `Add panel` icon on the top right of the dashboard. ![Add panel button](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iyf72cq6vfwxo6yypb3u.png)

7. Select `Add an empty panel`. ![Add an empty panel button](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6xts8vd3exiuie1zk95u.png)

8. Configure the data source to be `Loki` (or the name given to the data source in step 4). ![Data source set to Loki](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cznypglplb27t17kpu29.png)

9. Click on the visualisation type to change it from the default (in this case, _time series_), to `Logs`.
  i. ![Visualisation option](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/erbrxxkfyevxnlhbebtz.png)
  ii. ![Logs option](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1oqppgr192e1u8kws4p9.png)

10. Enter a query to read the _stdout_ of a pod, for example: ```
{app="loki"}
``` ![An example query](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7aboj2tmbfph72b4tfnv.png)

11. _(Optional)_ Consider reducing the number of data points being requested to something more modest, e.g. 100. ![Query options](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ccmjrywwn860eyuv4jv8.png)

12. _(Optional)_ If your logs are timestamped, you may be able to group them and format the timestamp. ![Log formatting options](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uvyei4mv7gwzeafo3u3m.png)
 

12. Apply your changes. ![Applying changes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hq2nz1o13yfertwn9zjl.png
)

If all goes well, you should be able to see your desired logs using Loki, as installed by DigitalOcean's 1-click installer. 🥳 ![A dashboard with Loki](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n6ezkm1kje2xyjpef9jl.png)

## How Does It Work?

The Loki 1-Click installer creates a service called `loki` in the `loki-stack` namespace, with an endpoint which the DNS resolver can interpret using the following syntax:

```
<service_name>.<namespace>.svc.cluster.local
```
Loki is running inside a pod where port 3100 is being used. Thus, the complete syntax is:

```
<service_name>.<namespace>.svc.cluster.local:<port>
```

## Troubleshooting

I'm not an expert. 🤷‍♂️

### 502: Bad Gateway

Translation: Loki is not accessible using the URL you've provided. That probably means your installation is different to what the 1-click installer provides, or DigitalOcean has changed the way it installs Loki after this article was written.

### Data source connected, but no labels received

The most likely culprits are:

- You're pointing to the wrong service. Be sure to double check the name of the service using `kubectl get svc -n loki-stack`, and verify that a service with the name `loki` exists.
- The Loki service has a different endpoint. Verify that the endpoint with the name `loki` exists using `kubectl get endpoints -n loki-stack`.
- A firewall policy. GG well met. ☠ Just kidding, but I'm no expert on kubernetes or its network policy configurations.