Loki: Getting Started
Tags: logaggregator, observability, beginners, tutorial
Published: 2023-08-06 23:22:44

The realm of open source log aggregators has seen significant growth over the years, with key players like [ElasticSearch](https://www.elastic.co/elastic-stack) (2010) and [Graylog](https://graylog.org/) (2016) dominating the field. More recently, [Grafana Loki](https://grafana.com/oss/loki/) (2019) has emerged as a compelling addition to this ecosystem. 

This article aims to facilitate a rapid onboarding process for log shipping using [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/), log aggregation with [Loki](https://grafana.com/oss/loki/), and data visualization through [Grafana](https://grafana.com/oss/grafana/). Collectively, this powerful trio of tools is commonly referred to as the PLG (Promtail-Loki-Grafana) stack.

## The Server System

These instructions will work on Debian or Ubuntu system.

### Using Vagrant (Virtualbox)

If you have Intel system, you can use [Vagrant](https://www.vagrantup.com/) with [Virtualbox](https://www.virtualbox.org/) to quickly bring up virtual servers on your workstation.  Follow the instructions from respect sites to install these tools.

When ready, create a file called `Vagrantfile` with the following contents:

```ruby
Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu2204"
  config.vm.network "forwarded_port", guest: 3000, host: 3000, host_ip: "127.0.0.1"
  config.vm.network "forwarded_port", guest: 3100, host: 3100, host_ip: "127.0.0.1"
end
```

**NOTE**: For this setup, the configuration will utilize `3000` for [Grafana](https://grafana.com/oss/grafana/) and `3100` for [Loki](https://grafana.com/oss/loki/) on the localhost. If any services are currently running on these ports, you'll have to either halt those services or modify the host: key in the provided above configuration to use an alternative port.

When ready, you can bring up the system and log into the system with the following:

```bash
vagrant up
vagrant ssh
```

### Installing Loki

On your desired system, such as the [Vagrant](https://www.vagrantup.com/) managed virtual machine above, you can run the following to install PLG:

```shell
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key \
  https://apt.grafana.com/gpg.key

# add a repository for stable releases
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" \
  | sudo tee -a /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install -y promtail loki grafana
```

### Verify Services are running

```bash
sudo service loki status
sudo service grafana-server status
```

If any of these services are stopped, you can run:

```bash
sudo service loki start
sudo service grafana-server start
```

## The Client Workstation

### Visualization

You can access the login page by opening a web browser on your system and navigating to http://127.0.0.1:3000. Simply use the default credentials, admin for both the username and password. Upon login, you will be prompted to set a new password for the admin account.

After logging in, you can click on Explore, Loki, and then select on of the logs by clicking on Label.

![Screen Shot of Grafana UI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ujyociv0oapsro55x8uy.png)

### Ship More Logs with Promtail

When setting up a new system, there may not be many interesting logs initially. However, your host workstation system is likely to have accumulated a wealth of interesting logs over time.

If you are running the server as a virtual machine using the previously mentioned `Vagrantfile` configuration, you can easily ship logs from the laptop to `localhost:3100`, where [Loki](https://grafana.com/oss/loki/) is up and running. This allows you to leverage the abundant logs from your host workstation for analysis and visualization.

You can install [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) locally as well. If the host system is running Debian or Ubuntu, you can run this:

```bash
# add a repository for stable releases
sudo wget -q -O /usr/share/keyrings/grafana.key \
  https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" \
  | sudo tee -a /etc/apt/sources.list.d/grafana.list

# install promtail
sudo apt-get update && sudo apt-get install -y promtail
```

By default, the configuration file `/etc/promtail/config.yml` will set up Promtail to ship logs to the URL: `http://localhost:3100/loki/api/v1/push`. This configuration will function correctly if you are using [Vagrant](https://www.vagrantup.com/) with [Virtualbox](https://www.virtualbox.org/) to run [Loki](https://grafana.com/oss/loki/). However, if you are not running the [Loki](https://grafana.com/oss/loki/) server locally, you will need to modify the address (currently set to `localhost`) in the configuration accordingly.

As an option you can modify the configuration with the following:

```yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      host: my_workstation
      agent: promtail
      __path__: /var/log/*log
```

And later restart the service:

```bash
sudo systemctl restart promtail
```