Deploy Kafka + Filebeat + ELK - Docker Edition - Part 2
Tags: docker, kafka, elk, devops
Published: 2021-07-18 11:49:32

##Introduction
This article is the last part of a two part series where we will deploy ELK stack using docker/docker-compose.
In this article, we will be configuring Logstash, Elasticsearch and Kibana. If you haven't gone through the previous article, check out this [link](https://dev.to/dhingrachief/deploy-kafka-filebeat-elk-docker-edition-part-1-3m77)

##Logstash
Logstash is a server-side data processing pipeline that consumes data from different sources and send it to elasticsearch. We touched on its importance when comparing with filebeat in the previous article.
Now to install logstash, we will be adding three components 
* a pipeline config - logstash.conf
* a setting config - logstash.yml
* docker-compose file

Pipeline configuration will include the information about your input (kafka in our case), any filteration that needs to be done, and output (aka elasticsearch).
Create a folder named pipeline and add this configuration file to it.

###Logstash pipeline - logstash.conf
```
input {
    kafka{
        bootstrap_servers => "KAFKA_SERVERS_IP:KAFKA_SERVERS_PORT"
        topics => ["applogs"]
    }
}
filter {
    json {
        source => "message"
      }
}
## Add your filters / logstash plugins configuration here
output {
        elasticsearch {
    hosts => ["ELASTICSEARCH_IP:9200"]
    user => 'elastic'
    password => 'somesecretpassword'
    }

```
As you can see, In the input section, we are listening to kafka on topic **applogs**.
In my case, I have added a filter which will jsonify the data that comes under the key named "message". There are several plugin [filters](https://www.elastic.co/guide/en/logstash/6.8/filter-plugins.html) to choose from.
The output is redirected towards elastic search, with username password configured for authentication.
Create a folder named settings and add this configuration file to it

###logstash settings - logstash.yml
```
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
path.logs: /var/log/logstash
config.reload.automatic: true
log.level: debug
xpack.monitoring.enabled: false

```
The docker-compose file looks like this: 

###logstash docker-compose 
```
version: '2'

services:
  logstash:
    image: docker.elastic.co/logstash/logstash:5.6.3
    ports:
      - "10000:10000"
    volumes:
      - ./settings/:/usr/share/logstash/config/
      - ./pipeline/:/usr/share/logstash/pipeline/
    container_name: logstash                          
```
Here the configuration files mentioned above have been mounted.

To run the above file 
```
docker-compose up -d
```
Now, You will get an error as we have not setup elastic search yet. Now lets go ahead and install Elasticsearch and Kibana


## Elasticsearch and Kibana
Kibana is an open source user interface that helps you monitor and visualise data, which in our case, is being provided by elasticsearch.
Elasticsearch, as the name suggests, is a search and analytics engine for different types of data.

### Elasticsearch and Kibana  docker-compose 
```
version: "2"
services:
 elasticsearch:
  image: "docker.elastic.co/elasticsearch/elasticsearch:7.13.2"
  container_name: elasticsearch
  environment:
   - discovery.type=single-node
   - cluster.routing.allocation.disk.threshold_enabled=true
   - cluster.routing.allocation.disk.watermark.low=65%
   - cluster.routing.allocation.disk.watermark.high=70%
   - xpack.security.enabled=true
   - xpack.security.audit.enabled=true
   - ELASTIC_PASSWORD=somethingsecret
   - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
   - bootstrap.memory_lock=true

  ulimits:
    memlock:
      soft: -1
      hard: -1
  volumes:
    - ./data:/usr/share/elasticsearch/data
  ports:
   - "9200:9200"
  networks:
   - eknetwork

 kibana:
  depends_on:
   - elasticsearch
  image: "docker.elastic.co/kibana/kibana:7.13.2"
  container_name: kibana
  ports:
   - "5601:5601"
  environment:
   - ELASTICSEARCH_URL=http://localhost:9200
   - ELASTICSEARCH_USERNAME=elastic
   - ELASTICSEARCH_PASSWORD=somethingsecretpassword
  networks:
   - eknetwork

networks:
 eknetwork:
  driver: bridge
```

Make sure to add same elasticsearch password as the one you have provided in your logstash pipeline configuration file.

To run the above file 
```
docker-compose up -d
```

Now that all our processes are running, You would need to configure Kibana's source from the UI so that you are able to see data on the UI. If you have done this setup on the server and want to configure UI immediately, I would suggest you to create a firewall rule for your public IP on port 5601 

```
ufw allow from YOUR_PUBLIC_IP to any port 5601
```

Now go to your favorite browser, type server's ip and port and you will see something like this:

![Home page of Kibana](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zmqoyrr5dscizw9tlyn3.png)
Enter the credentials as specified in the docker-compose above and you should be able to enter the application.

And Voila! Your setup is complete. Just don't forget to add log-rotation to your docker instances.
I have added all the configuration and docker file [here](https://github.com/kdsama/ELK)