Kubernetes Logging with Grafana Loki & Promtail in under 10 minutes
Tags: kubernetes, grafana, observability, promtail
Published: 2023-02-21 08:49:59

## What is the goal?

After completing this lab, we will have consolidated all the logs generated in our Kubernetes cluster in a tidy, neat, real-time dashboard in Grafana.

## What are we going to need?

We are going to need a:

- Kubernetes cluster.
- Grafana installation.
- Grafana Loki installation.
- Promtail agent on every node of the Kubernetes cluster.

> If you don’t have a Kubernetes cluster already in place, follow this guide to get started quickly with a containerised variant based on [K3S](https://k3s.io/):
> [Provision a Highly Available K3S Cluster with K3D](https://akyriako.medium.com/provision-a-high-availability-k3s-cluster-with-k3d-a7519f476c9c)
> or this one if you want a full-blown environment based on virtual machines in the cloud or on-premises:
> [Install Kubernetes 1.26 on Ubuntu 20.04](https://akyriako.medium.com/install-kubernetes-on-ubuntu-20-04-f1791e8cf799)

## What is Grafana?

[Grafana ](https://grafana.com/) is an analytics and interactive visualisation platform. It provides a rich variety of charts, graphs, and alerts and connects to plead of supported data sources as Prometheus, time-series databases or the known RDBMs. It allows you to query, visualise, create alerts on your metrics regardless where they are stored.

> You have to think it as the equivalent of Kibana in the ELK stack.

The installation is fairly simple and we are going to perform it via _Helm_. If you haven’t Helm already installed on your workstation you can do it either with brew if you are working on **MacOS**:

```
brew install helm
```

or with the following bash commands if you are working on **Debian/Ubuntu** Linux:

```
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null

sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list

sudo apt-get update
sudo apt-get install helm --yes
```

Putting this behind, we can now install the Helm chart for Grafana:

```
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install grafana grafana/grafana --namespace grafana --create-namespace
```

Let's see what's provisioned so far:

```
kubectl get all -n grafana
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jwjuzpf446q161pgradp.png)

> `service/grafana` service would be of type `ClusterIP`in a vanilla installation, in my case I am already using MetalLB as a network loadbalancer in my cluster and I have patched the service as of type `LoadBalancer`. Feel free to ignore this, we are going to port-forward this service later.

## What is Grafana Loki & Promtail?

[Grafana Loki](https://grafana.com/oss/loki/) is a logs aggregation system, more specifically as stated in their website: ”_is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream._” It’s a fairly new open source project that was started in 2018 at Grafana Labs.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m4755w0pxv4y9bzmpuq5.png)

Loki uses Promtail to aggregate logs. [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) is a **logs collector agent** that collects, (re)labels and ships logs to Loki. It is built specifically for Loki — an instance of Promtail will run on every Kubernetes node. It uses the exact same service discovery as _Prometheus _and support similar methods for labeling, transforming, and filtering logs before their ingestion to Loki.

Loki **doesn’t index** the actual text of the logs. The log entries are grouped into streams and then indexed with labels. In that way, Loki not only reduces the overall costs but additionally reduces the time between ingestion of log entries and their availability in queries.

It comes with its own query language, _LogQL_, which can be used from its own command-line interface or directly from Grafana. Last but not least, it can tightly integrate with the Alert Manager of Prometheus — though, the last two are _out of the scope of this article_.

> You have to think it as the equivalent (not 1–1 but in bigger context) of Elasticsearch in the ELK stack.

Loki, consists of multiple [components/microservices](https://grafana.com/docs/loki/latest/fundamentals/architecture/deployment-modes/):

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/klqq0gd9tw11xbe13bos.png)

that can be deployed in _3 different modes_:

- [Monolithic](https://grafana.com/docs/loki/latest/fundamentals/architecture/deployment-modes/#monolithic-mode) mode, all of Loki’s microservice components run inside a single process as a single binary.
- [Simple Scalable](https://grafana.com/docs/loki/latest/fundamentals/architecture/deployment-modes/#simple-scalable-deployment-mode) mode, if you want to separate read and write paths.
- [Microservices](https://grafana.com/docs/loki/latest/fundamentals/architecture/deployment-modes/#microservices-mode) mode, every Loki component runs as a distinct processes.

The scalable installation requires a S3 compatible object store such as AWS S3, Google Cloud Storage, Open Telekom Cloud OBS or a self-hosted store such as [MinIO](https://min.io/). In the _monolithic _deployment mode only the filesystem can be used for storage. 

> For more information concerning how to configure Loki's storage consult this [link](https://grafana.com/docs/loki/latest/installation/helm/configure-storage/).

In this lab, we are going to use the microservices deployment mode with [Open Telekom Cloud OBS](https://open-telekom-cloud.com/en/products-services/core-services/object-storage-service) as Loki’s storage. The installation (and essentially the configuration) of Loki and Promtail is performed by two distinct and independent charts.

First let’s download the default chart values for every chart and make the necessary changes. For Loki (given that you chose as well to go with the `loki-distributed` chart)

```
helm show values grafana/loki-distributed > loki-distributed-overrides.yaml
```

If you are planning to go with an S3 compatible storage and not with the filesystem, make the following changes to your chart values:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xfnum3l7xysczhxsk1da.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sgzvdvptay96wj25ld9r.png)

> The format of S3 endpoint is: 
`s3://{AK}:{SK}@{endpoint}/{region}/{bucket}
`

Next let's enable and configure the compactor:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zclecz5izp8ys38d8jrf.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/noj7eq0tbx1qtkouig30.png)

Loki values are now set, let’s install it and move to Promtail:

```
helm upgrade --install --values loki-distributed-overrides.yaml loki grafana/loki-distributed -n grafana-loki --create-namespace
```

```
helm show values grafana/promtail > promtail-overrides.yaml
```

Get all the components that we installed from Loki chart:

```
kubectl get all -n grafana-loki
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pqw4owp28d8y5yz7f819.png)

We are going to need the endpoint of Loki’s gateway as the designated endpoint that Promtail will use in order to push logs to Loki. In our case that would be `loki-loki-distributed-gateway.grafana-loki.svc.cluster.local`, so let’s add it in the Promtail chart values:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h04mc624p9b4ade8pyow.png)

We are ready to now to deploy Promtail. Run the command and wait for a bit till all pods come to a Ready state.

```
helm upgrade --install --values promtail-overrides.yaml promtail grafana/promtail -n grafana-loki
```

## Configure Grafana Data Sources & Dashboard

All the deployments now are completed. It is time we set up our Grafana. As we saw before Grafana has a simple service, let’s then port-forward it and access the Grafana directly from `http://localhost:8080/`:

```
kubectl port-forward service/grafana 8080:80 -n grafana
```

> Of course you are free to expose this service in a different way either with assigning to it an external IP by a Load Balancer or as an ingress route via the Ingress solution of choice.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uyt7jk6dmjqdgr6w9ygy.jpeg)

You are going to need those credentials in order to login. Default user is admin, but the password will need a bit of work to be retrieved. Get all the `Secrets`in the grafana namespace:

```
kubectl get secrets -n grafana
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w98icwahyna7372rewg5.png)

This is where our password lives. Let’s extract it and decode it:

```
kubectl get secret grafana -n grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```

We are now in. As next we need to add Grafana Loki as a data source:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b75gywvo5pkwebbx20ui.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ss8ykogixzq0m056ykk1.png)

As URL, use the endpoint of the Grafana Loki gateway service: `http://loki-loki-distributed-gateway.grafana-loki.svc.cluster.local`. Test, save and exit.

Last step, now we need to add a dashboard in order to see eventually our logs. At the very beginning you can step on the shoulders of existing ones and then tailor them according to your needs. A good stepping stone is this [one](https://grafana.com/grafana/dashboards/15141-kubernetes-service-logs/). Copy the dashboard template ID from the web page:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xgiuwjigp7dj9nu7xnsy.png)

and in your Grafana environment, choose to Import a new Dashboard:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8vl7n8lfqcoo9bkl8owg.png)

Paste the template ID we just acquired and load the dashboard:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ceb90laiobkkxlu76ydj.png)

Now all the puzzle pieces should come together and you should be able to see logs from your Kubernetes workloads directly into your Grafana interface as an almost real-time experience:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nw7d3vdaq63xjwqjvo42.png)

## Summary

Definitively, when it comes to Kubernetes monitoring and observability this is only scratching the surface, but nevertheless it is a robust first step you can fulfil with almost minimum effort and in less than 10 minutes.

Stay tuned for more Kubernetes topics.

Article originally posted at [Medium](https://akyriako.medium.com/kubernetes-logging-with-grafana-loki-promtail-in-under-10-minutes-d2847d526f9e).