Writing logs into Elastic with NLog , ELK and .Net 5.0 (part 1)
Tags: dotnet, elasticsearch, csharp, logstash
Published: 2021-08-04 10:25:58

If you are using Microservice-based architecture, one of the challenges is to integrate and monitor application logs from different services and ability to search on this data based on message string or sources, etc. You can also read [microservice roadmap](https://dev.to/majidqafouri/microservice-roadmap-4mci) here.

*This post assumes you already have a general idea of Elastic and Dotnet core and what it's used for, so if it's new to you, take a look at the Elastic official website.*

So, what is the ELK Stack?
“ELK” is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

On the other hand, NLog is a flexible and free logging platform for various .NET platforms, including .NET standard. NLog makes it easy to write to several targets. (database, file, console) and change the logging configuration on-the-fly.

If you combine these two powerful tools, you can get a good and easy way for processing and persisting your application’s logs
In this article, I’m going to use NLog to write my .Net 5.0 web application logs into Elastic. after that, I will show you how you can monitor and search your logs with different filters by Kibana.
First of all Open Visual Studio and select a new project (ASP.NET Core Web API)(If you want to know the best practice for Folder structure in your project, read my [article](https://medium.com/@m.qafouri/projects-folder-structures-best-practices-706e4136aaca))

![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kowygw86fo6sj1sanw3n.PNG)

On the next page set the target framework on **.Net 5.0(current)** and also check Enable **OpenAPI support** option for using swagger in the project.

![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bfjpzgoqajgh7fx1pbb6.PNG)

We need some Nuget packages, In order to install them, from the **Tools** menu select  **Nuget Package Manager** and then select **Manage Nuget packages for solutions**. 
In the opened panel search the below packages and install them one by one 

`NLog.Web.AspNetCore`
`NLog.Targets.ElasticSearch`


![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lr03os5v5jdtpvxruffv.png)

After you installed the packages, you need to add a config file for NLog. In order to add the file, right-click on the current project in the solution and select **add** => **new Item**, then add a `web configuration` file and name it `nlog.config`


![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x88x7oaeqz1m0i3shish.PNG)
Open the newly added file and paste the below codes

```csharp
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true">

	<extensions>
		<add assembly="NLog.Web.AspNetCore"/>
		<add assembly="NLog.Targets.ElasticSearch"/>
	</extensions>


	<targets async="true">

		<target name="elastic" xsi:type="ElasticSearch" index=" MyServiceName-${date:format=yyyy.MM.dd}"
				uri="http://localhost:9200"
				layout ="API:MyServiceName|${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}" >
		</target>

	</targets>

	<rules>
		<logger name="*" minlevel="Debug" writeTo="elastic" />
	</rules>
</nlog>
```

In the target tag, we define our configuration like Elastic service URI, layout pattern, Elastic index name and etc.
Let’s take a look at these tags and their attributes

**Index**: `MyServiceName -${date:format=yyyy.MM.dd}`: it means we will have different index for every day’s log. For example, `MyServiceName-2021.08.03` related to all the logs of the third of August  

**ElasticUri**:  Elastic URI address.

**Layout**: your log message pattern, you can determine which information you want to have in the message of your log.

In the **rules** tag, the `minimum level` for the writing log is defined.

You can read more detail about nlog tag config  [Here]
(https://github.com/NLog/NLog/wiki/Getting-started-with-ASP.NET-Core-5)
 
After we understand the structure and meaning of the config file, the next step is to set up NLog logger instead of **.Net build-in logger**. To achieve this, Open **program.cs** and change Main method like this

```csharp
var logger = NLogBuilder.ConfigureNLog("nlog.config").GetCurrentClassLogger();
            try
            {
                logger.Debug("init main");
                CreateHostBuilder(args).Build().Run();
            }
            catch (Exception exception)
            {
                logger.Error(exception, "Stopped program because of exception");
                throw;
            }
            finally
            {
                NLog.LogManager.Shutdown();
            }
```

Change the **CreateHostBuilder** in **program.cs** as well:

```csharp
public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                })
                .ConfigureLogging(logging =>
                {
                    logging.ClearProviders();
                    logging.SetMinimumLevel(LogLevel.Trace);
                })
                .UseNLog();  // NLog: Setup NLog for Dependency injection

```

In the next step, open **Startup.cs** and in the **ConfigureService** method add below code 

```csharp
services.AddLogging();
```

Now you are able to write your log into elastic with Injecting **ILogger<ObjectName>** in every class constructor.

```csharp
[ApiController]
    [Route("[controller]")]
    public class WeatherForecastController : ControllerBase
    {
        private ILogger<WeatherForecastController> _logger;

        public WeatherForecastController(ILogger<WeatherForecastController> logger)
        {
            _logger = logger;
        }

        [HttpGet]
        public IActionResult Get()
        {   
            _logger.LogDebug("Debug message");
            _logger.LogTrace("Trace message");
            _logger.LogError("Error message");
            _logger.LogWarning("Warning message");
            _logger.LogCritical("Critical message");
            _logger.LogInformation("Information message");

            return Ok();
        }
    }

 ```

From now on, If you run your application and send a request with Postman, all types of logs will be written into Elastic.

Well, We successfully configured our application to write the logs into Elastic, but how we can monitor and search out logs. For this purpose, Elastic has Kibana as a tool that lets users visualize data with charts and graphs in Elasticsearch.
After you install the Elastic, you can have access to [Kibana](http://localhost:5601/app/kibana/)


![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vwjhm515fy4fkyf8o1ja.png)

First of all, we should be able to create a pattern which our application's log will be fetched by that pattern. For doing this, select the `Management` option in the left panel

![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rarp5ql2qcvjbelqk7ml.png)

In the management panel select `Index Pattern` link in the left side

![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o6n8ogfb71lkaaoq1sp3.png)

In the showed panel click on `Create index pattern` button. You will see  `Create index pattern` panel, there is an `Index pattern` input box which you should input your `Index name`(in our example we set the index name in the **nlog.config** file like this `MyServiceName-${date:format=yyyy.MM.dd}`.)
If you want your pattern to include wild character instead of the date you should write down `myservicename-*` in the input(it means we want to create a pattern that is able to load all logs with a prefix of `myservicename-` ).

If there were any logs with this pattern, a Success message will be shown beneath the input box. You can select the pattern and click on Next step.

![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qds6rn0dlrany5l6215g.png)
In the next panel, you can add a `Time filter` to your pattern, which means you will be able to filter logs according to the insertion date. After you selected `@timestamp`, click on `Create index pattern` button to complete this process.

![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lsp8hbezchd2hngipz2h.png)
Now click on the `discover` link on the left panel to redirect to the logged dashboard.
![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w0lfdobxunpxvj38qaxj.png)

As you see, we have several logs with different types related to our application. If you do not see any log, be sure that you have selected your newly defined pattern on the left side of the dashboard
![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1i5htvc6dulcr0qvede4.png)
For filtering our logs according to the log message or insertion date you can use the filter bar located beneath the menu bar.

For example, in this picture, I have search logs that contain **critical** word in their log messages. 
After you set your filter click on the green `update` button on the right side.
![test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y4bn4mopn3e5741hh665.png)
 

That’s it. Now you can easily filter your application's logs with different patterns. 

For more information about Kibana features you can use this [link](https://www.elastic.co/guide/en/kibana/current/get-started.html)  

**Conclusion**
Aggregating application logs is one of the benefits of Elastic logstash and NLog, but with this approach, you also can easily export your application logs in any format and also could create aggregate logs which is very helpful for analyzing your logs according to the type of logs.
 
Read [Writing logs into Elastic with NLog , ELK and .Net 5.0 (Part 1)](https://dev.to/majidqafouri/writing-logs-into-elastic-with-nlog-elk-and-net-5-0-246c)
Follow me on [Medium](https://m-qafouri.medium.com/)

Maybe you like :
[Projects Folder Structures Best Practices](https://dev.to/majidqafouri/projects-folder-structures-best-practices-g9d)