An ELK stack from scratch, with Docker
Tags: monitoring, elk, elasticsearch, kibana
Published: 2020-04-09 13:55:41

---
title: An ELK stack from scratch, with Docker
published: true
date: 2016-09-28 12:00:00 UTC
tags: monitoring, elk, elasticsearch, kibana
canonical_url: https://d3rwan.github.io/d3rwan-blog/posts/elk-from-scratch
---

Or how to deploy a complete monitoring architecture in few minutes.

## Monitor

In few years, monitoring has become an indispensable activity in the IT world. Technical or functional, whether it provides metrics, charts, or other KPIs, monitoring its system has become a must-have.

In the rest of the article, we will learn how to deploy a monitoring architecture from scratch, in few minutes, thanks to Docker.

## The stack

First off, we will use the ELK stack, which has become in a few years a credible alternative to other monitoring solutions (Splunk, SAAS …).

It is based on the following software:

- **E** as _Elasticsearch_, search engine which provide full text search & analytics,
- **L** as _Logstash_, an ETL for retrieving data from heterogeneous sources, transforming them and sending them to _Elasticsearch_,
- **K** as _Kibana_, which provide an UI for exploring data, and create interactive dashboards

But also :

- **R** as \*Redis, \*an upstream broker which will serve as buffer in case of latency of the system, while avoiding excessive congestion in case of a peak,
- **C** as _Curator_, a tool to manage our index
- \*\*B \*_as \*Beats_, client-side agent to send the logs/metrics to our stack

![The complete architecture](https://d3rwan.github.io/d3rwan-blog/media/elk/stack-elk.jpeg)_The complete architecture_

## Deploy

We will use Docker containers for each stack component.

- [Elasticsearch](https://hub.docker.com/_/elasticsearch/) (5.1.2),
- [Logstash](https://hub.docker.com/_/logstash/) (5.1.2),
- [Kibana](https://hub.docker.com/_/kibana/) (5.1.2),
- [Redis](https://hub.docker.com/_/redis/) (3.2.6),
- [Curator](https://hub.docker.com/r/bobrik/curator/) (4.0.4)

Services and interactions are described in a docker-compose.yml file:

{% gist https://gist.github.com/d3rwan/66311cf71c8398aae0be32e47f270bab %}

A viable configuration is also available on my github account : [docker_elk_stack](https://github.com/d3rwan/docker_elk_stack).

## Hello, world

Based on this repository, we will deploy a functional stack:

```
# clone repo & build images
git clone [https://github.com/d3rwan/docker_elk_stack](https://github.com/d3rwan/docker_elk_stack)
cd docker_elk_stack
docker-compose build

# run (daemon)
docker-compose up -d

# show logs
docker-compose logs
```

After startup, you should be able to access Kibana (port 5601).

Then, we will deploy a basic example web app (NGinx serving HTML + Filebeat agent to send log in our stack)

```
# build image
docker build ./webapp -t dockerelkstack_webapp

# run (daemon)
docker run --network dockerelkstack_logging --link redis:redis -p 80:80 -d --name webapp dockerelkstack_webapp

# show logs
docker logs webapp
```

After startup, you should be able to access the web app (port 80).

![Example web app capture (source: [https://github.com/sbilly/joli-admin](https://github.com/sbilly/joli-admin))](https://d3rwan.github.io/d3rwan-blog/media/elk/joliadmin.png)_Example web app capture (source: [https://github.com/sbilly/joli-admin](https://github.com/sbilly/joli-admin))_

After few minutes browsing, returning to Kibana. An index (logstash-\*) is now available.

![Kibana: Index pattern configuration screen (here, logstash-\* index is available)](https://d3rwan.github.io/d3rwan-blog/media/elk/configure-pattern.png)_Kibana: Index pattern configuration screen (here, logstash-\* index is available)_

After creating index, we can now exploring our web app logs (Discover tab), create visualizations (Visualize tab) and dashboards (Dashboard tab).

![Kibana: Exploring the raw logs](https://d3rwan.github.io/d3rwan-blog/media/elk/response.png)_Kibana: Exploring the raw logs_

![Kibana: Analytic dashboard](https://d3rwan.github.io/d3rwan-blog/media/elk/analytics.png)_Kibana: Analytic dashboard_

And… voilà! In just few minutes, we set up, an operational monitoring stack.