Elasticsearch Logstash Kibana
Tags: elasticsearch, logstash, kibana
Published: 2020-06-13 10:07:06

The goal of this post is to load CSV (Comma Separated Values) file into Logstash which will process the data and give it to Elasticsearch for data storage and use Kibana to query and visualize the data. To achieve this, the process is shown in the image below:

![process](https://dev-to-uploads.s3.amazonaws.com/i/ecr74l6y788q8h0h420j.png)

Note: This article assumes that the reader has no prior knowledge of Elasticsearch, Logstash, Kibana and I will try to explain the basics to get it up and running. This post is not for advance working of Elasticsearch, Logstash, Kibana.

So let's get started

For those of you who are unaware of the [Elasticsearch](https://www.elastic.co/elasticsearch/), [Logstash](https://www.elastic.co/logstash), [Kibana](https://www.elastic.co/kibana) below is a quick intro of all three. When used together it is also known as the ELK stack. 

**Elasticsearch:**
![Elasticsearch Intro](https://dev-to-uploads.s3.amazonaws.com/i/6udmx2hpwung5ife27rp.png)

**Logstash:**
![Logstash Intro](https://dev-to-uploads.s3.amazonaws.com/i/93q0w7oh9huv8qbw2j2o.png)

**Kibana:**
![Kibana Intro](https://dev-to-uploads.s3.amazonaws.com/i/cgd6l4i52we7lztyojrf.png)

You can read more about it from [here] (https://www.elastic.co/what-is/elk-stack)

Now having the understanding of the ELK stack lets dive in. First, you will need to download [Elasticsearch](https://www.elastic.co/downloads/elasticsearch), [Logstash](https://www.elastic.co/downloads/logstash), [Kibana](https://www.elastic.co/downloads/kibana)

Note that the downloads will be in compressed folder. Extract all three (Elasticsearch, Logstash, Kibana) folders in your system.

Open Command Prompt in Windows or Terminal in Mac, go to the path where the folders are extracted. 
(For this tutorial I'm using Windows operating system and I have extracted the folders in `"D:\2020\Logstash\"`. At the time of this post the latest version for all three (Elasticsearch, Logstash, Kibana) is 7.7.1)

To run elasticsearch. Go to the bin folder of elasticsearch and then type elasticsearch, as shown below:
```
cd D:\2020\Logstash\elasticsearch-7.7.1-windows-x86_64\elasticsearch-7.7.1\bin
D:\2020\Logstash\elasticsearch-7.7.1-windows-x86_64\elasticsearch-7.7.1\bin>elasticsearch
```
![elasticsearch cmd](https://dev-to-uploads.s3.amazonaws.com/i/au2s0co156cg91y4fssx.png)

By default elasticsearch runs on port 9200 of localhost. Once the elasticsearch is loaded and running, as shown in below screenshot.

![elasticsearch cmd running](https://dev-to-uploads.s3.amazonaws.com/i/2a5tx3v5xpegcsmnj9x8.png)

You can now hit the http://localhost:9200 in your browser to check elasticsearch details. The URL returns a JSON file with details like name, cluster_name, cluster_uuid and version details.

![elasticsearch url](https://dev-to-uploads.s3.amazonaws.com/i/xtg070umuh0zhkwulx0i.png)

To run kibana. Go to the bin folder of kibana and then type kibana, as shown below:
```
cd D:\2020\Logstash\kibana-7.7.1-windows-x86_64\kibana-7.7.1-windows-x86_64\bin
D:\2020\Logstash\kibana-7.7.1-windows-x86_64\kibana-7.7.1-windows-x86_64\bin>kibana
```
![kibana cmd](https://dev-to-uploads.s3.amazonaws.com/i/tx04kwv0joqklsxo5yki.png)

By default kibana runs on port 5601 of localhost. Once the kibana is loaded and running, as shown in below screenshot.

![kibana cmd running](https://dev-to-uploads.s3.amazonaws.com/i/1fg4ynuef102dtur9tqk.png)

You can now hit the http://localhost:5601/app/kibana#/home in your browser to check kibana details. The URL returns the homepage of kibana.

![kibana home](https://dev-to-uploads.s3.amazonaws.com/i/sa896jq34azc9h6dztvy.png)

Next, to run logstash, we need to supply a config file as a parameter.  So we'll need to create the config file. 

Structure of a Config File
A Logstash config file has a separate section for each type of plugin you want to add to the event processing pipeline. For example:

```
# This is a comment. You should use comments to describe
# parts of your configuration.
input {
  ...
}

filter {
  ...
}

output {
  ...
}
```
Each section specifies the configuration options for one or more plugins. If you specify multiple filters, they are applied in the order of their appearance in the configuration file.

You can read more about it from [here](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html)

[Click here for more configuration examples](https://www.elastic.co/guide/en/logstash/current/config-examples.html)

For CSV file, you can create a CSV file or download it from [Kaggle.com](https://www.kaggle.com). In this post I've taken a sample file from [movies-on-netflix-prime-video-hulu-and-disney](https://www.kaggle.com/ruchi798/movies-on-netflix-prime-video-hulu-and-disney/data).
Note that these files may contain billions of data/ records which makes a perfect example to use it in elasticsearch.

![kaggle movies](https://dev-to-uploads.s3.amazonaws.com/i/5j4mm4ufp3z5u5efllar.png)

Example of the CSV file which I've used for this example:
![csv](https://dev-to-uploads.s3.amazonaws.com/i/x6q2hhmajiqg74bopqkp.png)

I've created a new folder named `data` and pasted the csv file in it. The name of the CSV file is movies.csv. Then create a new file with .config extension. I have named it `logstash_movies.config`. Note that these files need not be under the same folder and can have different names as well, as we will be specifying their paths with their file name, but for convenience I will put them under the same folder. Paste the below content in the config file

```
input {
    file {
        path => "D:/2020/Logstash/data/movies.csv"
        start_position => "beginning"
        sincedb_path => "NUL"
    }
}

filter {
    csv {
        separator => ","
        columns => ["Srno","ID","Title","Year","Age","IMDb","Rotten Tomatoes","Netflix",
        "Hulu","Prime Video","Disney+","Type","Directors","Genres","Country","Language",
        "Runtime"]
    }
    mutate { convert => ["Year", "integer"]} 
    mutate { convert => ["IMDb", "float"]} 
    mutate { convert => ["Runtime", "integer"]} 
}

output {
    elasticsearch {
        hosts => "localhost"
        index => "movies"
        document_type => "streaming_movies"
    }
    stdout {}
}
```
The input section configures the `path` to the CSV file, `start_position` specifies where to begin reading the data example from beginning or end of file (by default if you don't set, it is set to end of file), `sincedb_path` specifies the path to store the data, set to null if data storing is not required, etc. 

The filter section configures the csv and mutate filters. The csv section specifies the `separator`, `columns`, etc. By default all columns are set datatype as string but you can mutate it by converting it to the specified datatype like integer, float etc.

The output section configures elasticsearch which specifies 
host - set to localhost or the actual I.P address where the ELS stack is hosted, 
index - it is the index parameter used to identify the records for the given data. All data is fetched/ retrieved using this index parameter. We'll discuss more about index parameter later of this post, 
document_type - each record is a document.

Note that if you are using Mac O.S, change the following line:
Windows: `sincedb_path => "NUL"`
MAC: `sincedb_path => "/dev/null"`

Now that our config file is ready, we can run logstash. 

To run logstash . Go to the bin folder of logstash and then type logstash followed by `-f` and the config file path with name, as shown below:

```
cd D:\2020\Logstash\logstash-7.7.1\logstash-7.7.1\bin
D:\2020\Logstash\logstash-7.7.1\logstash-7.7.1\bin>logstash -f "D:\2020\Logstash\data\logstash_movies.config"
```

![logstash cmd](https://dev-to-uploads.s3.amazonaws.com/i/q6tg1fubo2era5hz0o9r.png)

If everything is in order then logstash will process the CSV file and feed it to elasticsearch. Once the CSV file is being read, it will log it to the console, as below: 
 
![logstash cmd running](https://dev-to-uploads.s3.amazonaws.com/i/djbkd7rp6e36lennjn8a.png)

Note that all three consoles must be running, do not close any of the consoles, as shown in the screenshot below:

![elasticsearch_kibana_logstash_cmd](https://dev-to-uploads.s3.amazonaws.com/i/en5wpzbc5obocsq2ay9n.png)

In order to visualize data, we need to create the index pattern. This index pattern is the same parameter which we specified in the config file.

From the home page of kibana, head to [`Management`](http://localhost:5601/app/kibana#/managemen) by navigating for the left pane, as shown below:

![kibana management](https://dev-to-uploads.s3.amazonaws.com/i/zfikaomio9qoobfz7lo6.png)

Under management on the left side, you will have two options Elasticsearch and Kibana. Under Kibana, click on the [`Index Patterns`](http://localhost:5601/app/kibana#/management/kibana/index_patterns). You will see as the screenshot below:

![index pattern](https://dev-to-uploads.s3.amazonaws.com/i/ysgzvxnmdt3j6n7m9p3v.png)

As seen above, there are no search patterns created. We need to create a index pattern. Click the `+ Create index pattern button` on the top right corner. Creating an index pattern is a two step process. 
In the first step, specify the index pattern name as `movies*`. You should able to see `Success! ...` in green text. Then click on `Next step`. 

![create index 1](https://dev-to-uploads.s3.amazonaws.com/i/zb5u2i3muk5io37bsczy.png)

In the second step, In `Configure settings`under Time Filter field name, you can select `I don't want to use the Time Filter` and click on `Create index pattern`.

![create index 2](https://dev-to-uploads.s3.amazonaws.com/i/ppcm3va0k39cstkqn0ul.png)

Now you should be able to see the newly created index pattern under `index pattern`, as below:

![movies index pattern](https://dev-to-uploads.s3.amazonaws.com/i/uhpj0czpoc59659fkjg3.png)

Here you get to see the columns names along with their datatypes, format, searchable, aggregatable etc.

Once the index pattern is create, you are now ready to visualize the data. But before that you can explore few options in kabana like `Discover` and `Dev Tools`

Under [`Dev Tools`](http://localhost:5601/app/kibana#/dev_tools/console) you can query the following:

movies get all endpoint:
```
GET /movies
```
![get movies](https://dev-to-uploads.s3.amazonaws.com/i/5t80co8jl0xsopc49y7r.png)


movies search endpoint:
```
GET /movies/_search
{
  "query": {
    "match_all": {}
  }
}
```

![get movies search](https://dev-to-uploads.s3.amazonaws.com/i/8tl9qevgwdlfeqod2y3j.png)

movies count endpoint:
```
GET /movies/_count
```
![get movies count](https://dev-to-uploads.s3.amazonaws.com/i/asgzc8talg89w446hn49.png)

Under [`Discover`](http://localhost:5601/app/kibana#/discover) you get to see the data records:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/6iwa4irzyd43sshe0lbb.png)

Following is the screenshot with the record expanded:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/5jo6mlrpgsj4saakqqls.png)

Now lets take a look at kibana visualization

Head to [`Visualize`](http://localhost:5601/app/kibana#/visualize) page, you should able to see as below:

![visualize](https://dev-to-uploads.s3.amazonaws.com/i/t7xgze0azvyasaz5mqg9.png)

Click on `+ Create new visualization`. A pop up screen will ask you select a visualization type, lets select `Pie`. 

![visualization new](https://dev-to-uploads.s3.amazonaws.com/i/8c3yxy4hdzuv48x9cspw.png)

On selecting `Pie`, the pop up will ask you to choose a source, it is the index pattern, select the index pattern: movies*

![visualization create pie](https://dev-to-uploads.s3.amazonaws.com/i/tgq4p6qvi99m3k21gaom.png)

On the right side, you will see `Buckets`, click on `Add` and select `Split slices`

![visualization pie](https://dev-to-uploads.s3.amazonaws.com/i/9k1970vwptz48pfk71se.png)

Next, select `Aggregation` to `Terms` and select any field of your choice. In this case I've selected `Country.keyword`. This will populate the pic chart by countries, as shown below:

![kibana_visualization_country](https://dev-to-uploads.s3.amazonaws.com/i/k7jsxw76j0wx77lsxt2z.png)

You can also do some customization under `Options` tab. Example: you can change the pie from donut, show or hide tooltip, label settings etc.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/kpx193jnzk5wa9hkhnlc.png)

Finally, after making the desired changes, you can save the visualization, as below:

![kibana_visualization_save](https://dev-to-uploads.s3.amazonaws.com/i/b5p603z9a2ichirt9dxb.png)

The saved visualization can then be used in dashboard.

Finally, Dashboard

We can create advanced dashboard like below screenshot from their official site, but we'll create a simple dashboard:

![advanced dashboard](https://dev-to-uploads.s3.amazonaws.com/i/hrytl8yppqhph186yr2v.jpg)

Dashboard contains visualization which we created in the previous step. To create a dashboard, head to [`Dashboards`](http://localhost:5601/app/kibana#/dashboards) 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/psq5bje8b1bt99hh6uvr.png)

Click on the `+ Create new dashboard` button

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/tjnuiuk9zwbcn30tye13.png)

On the next screen, you will asked to choose panels, which are nothing be the visualization which you have created, or you have an option to create a new one.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/r9tpn3s7436a003x6ov0.png)

Then save the dashboard:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/2srds1ssnyukvhsfoa1m.png)

Our final dashboard looks as below:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/v17amht3kh7nsyh1ar7k.png)

You can find the config file and the CSV file on my github page [MilburnGomes](https://github.com/MilburnGomes/elk-stack)

**See also**
https://www.elastic.co/

That's it for the ELK stack, hope you have enjoyed and understood it as well.

Thanks for reading!