.Net 6 WebAPI - Intro to ElasticSearch & Kibana  - Step by Step
Tags: dotnet, elasticsearch, docker, api
Published: 2021-12-20 09:42:04

In this article we will explore .NET 6 with ELK stack including ElasticSearch, Kibana and Serilog

### What are we covering in this article

- What is a Elastic Search
- What is Kibana
- Why use ElasticSearch
- How everything fits together
- Companies which use ELK
- Ingredients
- Code

You can watch the full video on Youtube
{% youtube 5exN6nQ7558 %}

And you can find the link to the source code on GitHub 
https://github.com/mohamadlawand087/Net6-ELK

### What is Elastic Search

Elasticsearch is a distributed, open-source search and analytics engine, Elasticsearch allows you to store, search, and analyze huge volumes of data quickly and in near real-time and give back answers in milliseconds.

ElasticSearch utilisation 

- Search your way
- Analyze at scale

### What is Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elasticsearch Clusters.

The core feature of Kibana is data querying & analysis. In addition, Kibana’s visualization features allow you to visualize data in alternate ways

With Kibana, it is easy to understand big data, and you can quickly build and share dynamic dashboards that frame-out changes to the Elasticsearch query in real-time.

### Why use ElasticSearch

- E**asy to get started**: Just fire up a docker compose file containing ElasticSearch and Kibana containers and you’re ready to start logging and searching.
- **Easy to Query:** ElasticSearch has a built-in full-text search engine that is based on Apache Lucene. Compared to other databases, Lucene is easy to query. Even non-technical people would be able to write common queries.
- Fast: Querying a large SQL database can easily take 10 or 20 seconds. It’s quite common for similar queries on a large ElasticSearch database to return results in under 10 milliseconds.
- **Fr**ee: Well almost. The basic features are free, well mostly. If you need security and alerting features in Kibana, you can buy the commercial X-pack subscription for Kibana, or you can install some open source alternatives.
- **Full Text Search Engine**
- **Analytical Engine**
- **RESTful API:** ElasticSearch has a RESTful API. Query results are returned in JSON which means results are easy to work with. Querying and inserting data via the RESTful API means it’s easy to use any programming language to work with ElasticSearch.
- S**calable:** It’s easy to scale. Combined with the fact that it's open source means it's easy on the wallet too.

### Companies which use ELK

- Netflix
- Ebay
- Walmart
- Adobe
- LinkedIn
- Stack Overflow

### Code Time

We create a docker file

```docker
version: '3.1'

services:
  elasticsearch:
    container_name: els
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.1
    ports:
      - 9200:9200
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    environment:
      - xpack.monitoring.enabled=true
      - xpack.watcher.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    networks:
      - elastic

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.16.1
    ports:
      - 5601:5601
    depends_on:
      - els
    environment:
      - ELASTICSEARCH_URL=http://localhost:9200
    networks:
      - elastic
  
networks:
  elastic:
    driver: bridge

volumes:
  elasticsearch-data:
```

Execute the command 

```docker
docker-compose up -d
```

To test they are running we visit the following URLs

- ELS: [http://localhost:9200](http://localhost:9200/)
- Kibana: [http://localhost:5601](http://localhost:5601/)

Now we create our web application

```bash
dotnet new webapi --no-https -n DotnetELK
```

Install the packages

```bash
dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Enrichers.Environment
dotnet add package Serilog.Sinks.Debug
dotnet add package Serilog.Sinks.Elasticsearch
dotnet add package Serilog.Exceptions
dotnet restore
```

Update appsettings.json

```json
{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Information",
        "System": "Warning"
      }
    }
  },
  "ElasticConfiguration": {
    "Uri": "http://localhost:9200"
  },
  "AllowedHosts": "*"
}
```

We need to update our program.cs

```csharp
using System.Reflection;
using Serilog;
using Serilog.Sinks.Elasticsearch;

var builder = WebApplication.CreateBuilder(args);

ConfigureLogging();
builder.Host.UseSerilog();

// Add services to the container.
builder.Services.AddControllersWithViews();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Home/Error");
}
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Home}/{action=Index}/{id?}");

app.Run();

void ConfigureLogging()
{
	var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
	var configuration = new ConfigurationBuilder()
		.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
		.AddJsonFile(
			$"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")}.json",
			optional: true)
		.Build();

	Log.Logger = new LoggerConfiguration()
		.Enrich.FromLogContext()
		.Enrich.WithExceptionDetails()
		.WriteTo.Debug()
		.WriteTo.Console()
		.WriteTo.Elasticsearch(ConfigureElasticSink(configuration, environment))
		.Enrich.WithProperty("Environment", environment)
		.ReadFrom.Configuration(configuration)
		.CreateLogger();
}

ElasticsearchSinkOptions ConfigureElasticSink(IConfigurationRoot configuration, string environment)
{
	return new ElasticsearchSinkOptions(new Uri(configuration["ElasticConfiguration:Uri"]))
	{
		AutoRegisterTemplate = true,
		IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}"
	};
}
```

Now we run our application 

```bash
dotnet run
```

Add integration

- click on menu
- Under Analytics click discover
- create index pattern
- in the Name type: dotnetelk-*
- in timestamp field choose @timestamp
- click on create index pattern
- Now go back to discover and we can see all of the logs so far there

Now let us create a custom message, inside our controller we update the following 

```csharp
[HttpGet(Name = "GetWeatherForecast")]
public IEnumerable<WeatherForecast> Get()
{
    _logger.LogInformation("WeatherForecastController Get - this is a nice message to test the logs", DateTime.UtcNow);
    return Enumerable.Range(1, 5).Select(index => new WeatherForecast
    {
        Date = DateTime.Now.AddDays(index),
        TemperatureC = Random.Shared.Next(-20, 55),
        Summary = Summaries[Random.Shared.Next(Summaries.Length)]
    })
    .ToArray();
}
```

Now we can search for it by typing this in the search box

```
message: "WeatherForecastController Get"
```

Now let us test errors, inside our controller

```csharp
try
{
    throw new Exception("Testing exceptions - ML");
}
catch (Exception ex)
{
    _logger.LogError(ex, "An unknown error occurred on the Index action of the HomeController");
}
```