Loki: Effective Logging and Log Aggregation with Grafana
Tags: devops, webdev, tutorial, opensource
Published: 2023-12-29 13:09:25


If you are a programmer or application administrator, you surely understand how important logs are in the infrastructure. Through them, you can monitor what's happening under the hood of your application, detect issues, and analyze system performance and behavior in real time. However, what happens when the volume of logs increases and analysis becomes more complex?

## Introducing Loki

This is where Loki comes in - an advanced log aggregation system developed by Grafana Labs. Loki aims to simplify effective and user-friendly collection and storage of logs.

Loki is designed to resemble Prometheus, allowing for quick and intuitive data filtering. Compared to Prometheus, the creators of Grafana decided to forego storing metrics in favor of storing logs themselves. In my opinion this decision stemmed from the desire to break monitoring tools into smaller, more modular parts. Instead of creating an all-in-one tool, they chose an approach based on several smaller tools. This allows for utilizing only the features that are relevant to a specific project, providing greater flexibility and control over the project.

A key feature of Loki is that it indexes only labels and metadata for each log message. This means that the full log contents are not stored. On one hand, this improves the tool's performance and reduces maintenance costs. On the other hand, this approach might somewhat limit analysis. However, it's worth keeping in mind that everything comes at a cost.

![Loki 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yf8yqen4hq4spb5b9sfb.png)

## Loki's Technological Stack

In practice, Loki consists of three key components:

1. **Promtail** - This is an agent that operates in a "push" configuration rather than "pull." This means its task is to acquire logs and send them to Loki, instead of waiting for queries from Loki. Multiple Promtail agents can run on a single machine, collecting and forwarding logs.
2. **Loki** - Loki is responsible for storing our logs. It indexes only the labels and metadata of each message, enabling efficient filtering and searching. Loki collects and provides log data.
3. **Grafana** (or another visualization system) - This component is used to interact with Loki. Grafana queries Loki to perform filtering and select the desired results from the logs. These data can then be visualized as charts or other graphics.

The operational flow of this tool can be simplified using the following diagram:

![Loki 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/taim9krk9ig6jygh5q0b.png)

With the theoretical understanding in place, let's move on to the practical aspect.

## Deploying Loki

To demonstrate the tool's functionality, we will use the Docker platform. Specifically, we will use a pre-made docker-compose file available [here](https://ruanbekker.medium.com/logging-with-docker-promtail-and-grafana-loki-d920fd790ca8), while making certain modifications. This file will allow us to sequentially launch Grafana, Loki, Promtail, and an Nginx server.

Here's the content of the docker-compose file after making certain changes:

```yaml
version: '3.4'
  
networks:
	app:
  
services:
	nginx-app:
		container_name: nginx-app
		image: nginx
		labels:
		logging: "promtail"
		logging_jobname: "containerlogs"
		ports:
			- 8080:80
		networks:
			- app
	  
	grafana:
		image: grafana/grafana:latest
		ports:
			- 3000:3000
		volumes:
			- ./config/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yaml
		environment:
			- GF_AUTH_ANONYMOUS_ENABLED=true
			- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
			- GF_AUTH_DISABLE_LOGIN_FORM=true
		networks:
			- app
	  
	loki:
		image: grafana/loki:latest
		ports:
			- 3100:3100
		command: -config.file=/etc/loki/local-config.yaml
		networks:
			- app
	  
	promtail:
		image: grafana/promtail:latest
		container_name: promtail
		volumes:
			- ./config/promtail.yaml:/etc/promtail/docker-config.yaml
			- /var/lib/docker/containers:/var/lib/docker/containers:ro
			- /var/run/docker.sock:/var/run/docker.sock
		command: -config.file=/etc/promtail/docker-config.yaml
		depends_on:
			- loki
		networks:
			- app
```

Additionally, we have created two configuration files. The first of them is responsible for the data source in Grafana, while the second one governs Promtail settings.

File `grafana-datasources.yml`:

```yaml
apiVersion: 1
  
datasources:
	- name: Loki
		type: loki
		access: proxy
		url: http://loki:3100
		version: 1
		editable: false
		isDefault: true
```

File `promtail.yaml`:

```yaml
server:
	http_listen_port: 9080
	grpc_listen_port: 0
  
positions:
	filename: /tmp/positions.yaml
  
clients:
	- url: http://loki:3100/loki/api/v1/push
  
scrape_configs:
	- job_name: flog_scrape
		docker_sd_configs:
			- host: unix:///var/run/docker.sock
			refresh_interval: 5s
			filters:
			- name: label
			values: ["logging=promtail"]
		relabel_configs:
			- source_labels: ['__meta_docker_container_name']
			regex: '/(.*)'
			target_label: 'container'
			- source_labels: ['__meta_docker_container_log_stream']
			target_label: 'logstream'
			- source_labels: ['__meta_docker_container_label_logging_jobname']
			target_label: 'job'
```

After running the `docker-compose up` command, containers for Grafana, Loki, Promtail, and Nginx will be created. Upon accessing the local address with port 3000, you will have the ability to create visualizations of your logs.

![Loki 3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kw0wuespz8v81papa468.png)

As we can see, with this approach, we can retrieve logs from a container with a specific name, in our case, it will be `nginx-app`.

## Types of Logs

There are several different types of logs, including:

1. **Event Log:** An event log is a high-level record that logs information about network traffic and interactions, such as login attempts, failed authentication attempts, and application-related events.
2. **Server Log:** A server log is a text file containing records of actions related to a specific server over a defined period of time.
3. **System Log (syslog):** A system log, also known as syslog, is a record of system events. It includes messages about system startup, changes to the system, unexpected shutdowns, errors, warnings, and other important processes. Windows, Linux, and macOS operating systems generate syslog messages.
4. **Authorization Logs and Access Logs:** Authorization logs and access logs contain lists of individuals or bots who accessed specific applications or files.
5. **Change Logs:** Change logs provide a chronological list of modifications made to an application or file.
6. **Availability Logs:** Availability logs track system performance, uptime, and availability.
7. **Resource Logs:** Resource logs provide information about connectivity issues and capacity limitations.
8. **Threat Logs:** Threat logs contain information about system traffic, files, or applications that match a predefined security profile within a firewall.

The method of logging data will depend on the application architecture and programming language. For example, for applications written in Django or Flask, you can use a library called `python-logging-loki`. Adding logs to Loki could look like the following:

```python
import logging
import logging_loki

handler = logging_loki.LokiHandler(
    url="https://my-loki-instance/loki/api/v1/push", 
    tags={"application": "my-app"},
    auth=("username", "password"),
    version="1",
)

logger = logging.getLogger("my-logger")
logger.addHandler(handler)
logger.error(
    "Something happened", 
    extra={"tags": {"service": "my-service"}},
)
```

## Summary

In the discussed article, we learned about a tool called Loki, created by Grafana Labs, which is useful for effective log management in programming environments. We discovered that Loki is a system designed for collecting and processing various types of logs. Through its innovative "logs as queries" (log-as-data) approach, Loki enables efficient storage of vast amounts of logs while minimizing resource consumption.

Additionally, we discussed various types of logs that can be collected and analyzed using Loki.