Running ELK (Elastic Logstash Kibana) on Docker
Tags: docker
Published: 2020-09-21 03:50:04

ELK (Elastic Logstash Kibana) are a set of software components that are part of the Elastic stack.

# What does ELK do ?
To explain in layman terms this what each of them do

* **Elasticsearch** is primarily a data store
* **Logstash** is a data parsing software that stores the data in Elasticsearch in a desired format
* **Kibana** is the UI that can be used to query / visualize the data that is stored in Elasticsearch

To get an in-depth understanding of what they do and how they work I would recommend [Beginner's Guide To Elastic Search](https://dev.to/lisahjung/beginner-s-guide-to-elasticsearch-4j2k)

# How do I run ELK ?

Since each of the above components are separate pieces of software one way of running them is to head to the installation instructions and run each one of them separately. 

An easier and a more convenient way to run them would be using Docker.

Most likely if you find yourself experimenting with this stack you would want to run all these 3 together. What better way to achieve than using `docker` and `docker-compose`

## Docker Compose

At the time of writing this post I was experimenting with ELK stack version `6.6`. Hence the following `docker-compose.yml` refers to image versions `6.6`.

{% gist https://gist.github.com/anandsunderraman/736bceba231a4dce7522bfa924062449 file=docker-compose.yml %}

If you notice the above gist it references a directory by name `logstash-conf`. The contents of this directory is a `logstash` configuration file that dictates how the data needs to be parsed.

The contents of this file would be:

{% gist https://gist.github.com/anandsunderraman/736bceba231a4dce7522bfa924062449 file=jsonlogs-pipeline.conf %}

### What are we configuring in Logstash ?

The following section says we will get the input for logstash via `beats` which is another software in `Elastic` which I will attempt to explain in another post.

We configure to obtain data via port `5044` and we expect the data to be in `json` format
```
input {
    beats {
        port => "5044"
        codec => "json"
    }
}
```

Here we state that we are using the `json` plugin in logstash and attempt to extract `json` data from the `message` field in our log message. I know this sounds a bit cryptic but hope you take the leap of faith with me on this.
```
filter {
  json {
    source => "message"
  }
}
```

Finally we have the output. We basically are passing on the data to elasticsearch to store the data in an index that is defined by `"%{[fields][project]}-%{[fields][application]}-%{+YYYY.MM.dd}"`

```
output {
    elasticsearch {
        hosts => "${ELASTIC_HOST}"
        index => "%{[fields][project]}-%{[fields][application]}-%{+YYYY.MM.dd}"
        codec => json
    }
}
```

### How to run it ?

* Start `docker` on your local machine
* Run `docker-compose up` in the directory where you have the `docker-compose.yml`

### How do I navigate to Kibana ?

* Point your browser to `http://localhost:5601/` 
* Note this is based on the port `5601` provided for the `kibana` image on the `docker-compose.yml`

# Conclusion

The main goal of this tutorial was to demonstrate how to get the ELK stack running using docker