Create a dashboard to visualize application logs in Kibana
Tags: elasticsearch, kibana, log, aspnetcore
Published: 2021-04-29 20:21:46

In this article, I'm going to show you how to create a dashboard in Kibana to visualize application logs, and of course if you are using Elasticsearch to store your application logs.

####Step 1- Setup Elasticsearch and Kibana
I use docker to run an instance of Elastic and Kibana. 
- Create `docker-compose.yml` file and add the following content:
```yml
version: "3.0"
services:
  elasticsearch:
    container_name: es-container
    image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
    environment:
      - xpack.security.enabled=false
      - "discovery.type=single-node"
    networks:
      - es-net
    ports:
      - 9200:9200
  kibana:
    container_name: kb-container
    image: docker.elastic.co/kibana/kibana:7.12.0
    environment:
      - ELASTICSEARCH_HOSTS=http://es-container:9200
    networks:
      - es-net
    depends_on:
      - elasticsearch
    ports:
      - 5601:5601
networks:
  es-net:
    driver: bridge
```
- Run the` docker-compose up` command 
Now you should have access to Elasticsearch via `http://localhost:9200/` and Kibana through `http://localhost:5601/`
 
####Step 2- Setup ASP.NET Core Web API project
This step is optional if you are not a .NET developer, just create an application store some logs into Elasticsearch.

- Open Visual Studio and create a Web API project
- Install [Serilog.AspNetCore](https://www.nuget.org/packages/Serilog.AspNetCore) and [Serilog.Sinks.Elasticsearch](https://www.nuget.org/packages/Serilog.Sinks.ElasticSearch/) nuget packages
- Open `Program.cs` file and modify `CreateHostBuilder` method and Add `UseSerilog` extension method
```csharp
public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        })
        .UseSerilog((hostingContext, loggerConfiguration) =>
            loggerConfiguration.ReadFrom.Configuration(hostingContext.Configuration));
```
- Open `appsettings.json` file and get rid of the logging section:
```json
"Logging": {
  "LogLevel": {
    "Default": "Information",
    "Microsoft": "Warning",
    "Microsoft.Hosting.Lifetime": "Information"
  }
}
```
- Add Serilog configuration section to `appsettings.json` file
```json
"Serilog": {
  "MinimumLevel": {
    "Default": "Debug",
    "Override": {
      "Microsoft": "Information",
      "System": "Information"
    }
  },
  "Enrich": [ "FromLogContext" ],
  "WriteTo": [
    {
      "Name": "Elasticsearch",
      "Args": {
        "nodeUris": "http://localhost:9200",
        "indexFormat": "demo-api-{0:yyyy.MM}",
        "autoRegisterTemplate": true,
        "autoRegisterTemplateVersion": "ESv7"
      }
    }
  ]
}
```
- Open `Startup` class and in the `Configure` method add the following code
```csharp
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    ....
    app.UseSerilogRequestLogging();
```

- Run the API project and make several API call through Swagger

####Step 3- Create index in Kibana
- In Kibana dashboard from the menu go to `Management -> Stack Management`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7kqex2f96ezfokw7t1g0.jpg)

- In `Stack Management` page and in Kibana section click on `Index Patterns`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wty97sbwkjx38v3qffe5.jpg)

- In Index Pattern click on `Create index pattern` button
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m0zc6y7uu4k9cryw13m5.jpg)

- In Create index pattern page you should see your index pattern `demo-api-2021.04` name (if your logs are saved successfully in Elasticsearch) and in `Index pattern name` input enter your index name `demo-api-*` and click `Next` button
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/id1xc71l9fybogki28be.jpg)

- In the next step in the `Time field` drop down select `@timestamp` and click on `Create index pattern` button
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8htqlgrcao4fcv6nqbl1.jpg)

- From the menu go to `Analytics -> Discover` page and select your index to view logs
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ph68l3chj5kiacb46627.jpg)
####Step 4- Create Visualizer
Before creating a dashboard you need to create several visualizers to add your log dashboard.

- From the menu click on `Analytics -> Visualize library` then click on `Create new visualization`
- From `New visualization` popup window click on `Aggregation based` (easiest way to create visualizer is using `Lens` but after creating visualizer aggregation based method, you can create other visualizers very quickly)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l2bc67mithnd5gak9j8h.jpg)

#####Total logs metric visualizer
- Click on `Metric` visualizer
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ama9zn30q5gpa6gm3xwn.jpg)
- After choosing `Metric` visualizer then click on your index `demo-api-*`
- By now I have 100 logs
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hpxjh7uhyknhd3fmkrxb.jpg)
- You just need to save the visualizer, click on the save button name it `Total logs`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jq18b69zo50m1f5rl5u0.jpg)

#####Log level metric visualizer
- Follow the above steps and create another metric visualizer
- In `Buckets` section click on `Add` then `Split group`
 ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dlxj3r05ltzfqmwmaw8r.jpg)
- From `Aggregation` dropdown select `Terms`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dsnf0sm8h6blxl9nok9h.jpg)
- After selecting `Terms` in `Field` dropdown select `level.raw` and then click the `Update` button at the bottom
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ofvnf7rqcbvvqagoj7l.jpg)
Now you have all log levels count and save the visualizer and name it `Log levels count`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aud62mt05e2749kyi0fa.jpg)

#####Log level pie visualizer
- Create a new visualizer and choose `Pie`
- In `Buckets` section click on `Add` then `Split slices`
- In `Aggregation` dropdown select `Terms`
- In `Field` dropdown select `level.raw`
- Click the `Update` button at the bottom
- Save the visualizer name it `Log levels percentage` 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/167mka6xkzop1ialgycn.jpg)
You can click on `Options` and change the appearance of the pie
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8yre896hspcsoligbseu.jpg)
You can also change the color of each level by clicking on the level label
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hpxadj88cdlyiycpyg35.jpg)

#####Vertical bar visualizer
- Create a new visualizer and choose `Vertical bar`
- In `Buckets` section click on `Add` then `X-axis`
- In Aggregation dropdown select `Date Histogram`
- Click the `Update` button at the bottom
- Save the visualizer name it `Total logs bar`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/of9540verq62iz6uo9f6.jpg)

#####Table visualizer
- Create a new visualizer and choose `Table`
- In `Buckets` section click on `Add` then `Split rows`
- In `Aggregation` dropdown select `Terms`
- In `Field` dropdown select `fields.SourceContext.raw`
- Click the `Update` button at the bottom
- Add another `Split rows`
- In `Aggregation` dropdown select `Terms`
- In `Field` dropdown select `level.raw`
- Save the visualizer name it `Log level sources`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iwi16szuii3i4whoe9vv.jpg)

#####Error log level search
- From the Analytic menu click on Discover
- In KQL input enter `level.raw:"Error`
- In Date filter choose `Today`
- Save the search name it `Today error level search`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aw9t81l036lz01ljqbw7.jpg)

It's time to create a dashboard. 
- From the Analytic menu click on Dashboard
- Click on `Create new dashboard` button
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jh16nrbsg7kjwyc7buea.jpg)
- From the right sidebar click all visualizer we have created previously
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mmexi9nxw9rfpjlmperd.jpg)
- From `Types` dropdown click on `Saved search` then click on `Today error level search`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q4krf7kccqqht9qgfh63.jpg)
- Click on `Save` button and save the dashboard 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iadw8ly01vn1zz28fpfc.jpg)

In the end, you can drag and drop or resize visualizers and arrange visualizers as you like.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9984kyqr84lc3f95u96d.jpg)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ny3eojji8kxi8ut4mw1p.gif)