Structured logging with Serilog and Seq and ElasticSearch under Docker
Tags: aspnetcore, docker, serilog, elasticsearch
Published: 2020-04-20 18:26:23

This blog post demonstrates Structured Logging with Serilog, Seq, ElasticSearch and kibana under Docker containers.
This post is a follow up on the beginner post I wrote on Serilog.

If you are a newbie on Serilog then I will recommend that you give a quick read to official Serilog page and this blog post.

## Example application
The complete below example shows serilog structured logging in a containerized web application with microservices style using docker, with events sent to the Seq and elastic search as well as a date-stamped rolling log file with the use of available below sinks:

* Serilog.Sinks.File
* Serilog.Sinks.Http
* Serilog.Sinks.Seq
* Serilog.Sinks.ElasticSearch

## Create a new empty solution
To start, we need create a new solution using Visual studio or your favorite IDE.
![Create a new empty solution](https://i1.wp.com/mselmi.net/wp-content/uploads/2020/04/create_new_proeject.png?w=1272&ssl=1)
## Create a new web Application project
Create new Web application project like below:
![Create a new web Application project](https://i0.wp.com/mselmi.net/wp-content/uploads/2020/04/create_application_with_docker_support.png?w=1278&ssl=1)
## Install the core Serilog package and the File, Seq and ElasticSearch sinks
In Visual Studio, open the Package Manager Console and type:

Install-Package Autofac
Install-Package Autofac.Extensions.DependencyInjection
Install-Package Microsoft.AspNetCore.Mvc.NewtonsoftJson
Install-Package Microsoft.Extensions.DependencyInjection
Install-Package Microsoft.Extensions.Hosting
Install-Package Microsoft.Extensions.Logging
Install-Package Microsoft.Extensions.Options
Install-Package Newtonsoft.Json
Install-Package Serilog
Install-Package Serilog.AspNetCore
Install-Package Serilog.Extensions.Hosting
Install-Package Serilog.Extensions.Logging
Install-Package Serilog.Settings.AppSettings
Install-Package Serilog.Settings.Configuration
Install-Package Serilog.Sinks.ElasticSearch
Install-Package Serilog.Sinks.File
Install-Package Serilog.Sinks.Http
Install-Package Serilog.Sinks.Seq
## Add the following code to Program.cs

### Create GetConfigurationmethod
```csharp
private static IConfiguration GetConfiguration()
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.Development.json", optional: true)
        .AddEnvironmentVariables();

    return builder.Build();
}
```
### Create CreateSerilogLoggermethod
```csharp
private static ILogger CreateSerilogLogger(IConfiguration configuration)
{
    var seqServerUrl = configuration["Serilog:SeqServerUrl"];
    return new LoggerConfiguration()
        .MinimumLevel.Verbose()
        .Enrich.WithProperty("ApplicationContext", AppName)
        .Enrich.FromLogContext()
        .WriteTo.File("catalog.api.log.txt", rollingInterval: RollingInterval.Day)
        .WriteTo.Elasticsearch().WriteTo.Elasticsearch(ConfigureElasticSink(configuration, "Development"))
        .WriteTo.Seq(string.IsNullOrWhiteSpace(seqServerUrl) ? "http://seq" : seqServerUrl)
        .CreateLogger();
}
```
### Create ConfigureElasticSinkmethod
```csharp
private static ElasticsearchSinkOptions ConfigureElasticSink(IConfiguration configuration, string environment)
{
    return new ElasticsearchSinkOptions(new Uri(configuration["Serilog:ElasticConfiguration"]))
    {
        BufferCleanPayload = (failingEvent, statuscode, exception) =>
        {
            dynamic e = JObject.Parse(failingEvent);
            return JsonConvert.SerializeObject(new Dictionary<string, object>()
                {
                    { "@timestamp",e["@timestamp"]},
                    { "level","Error"},
                    { "message","Error: "+e.message},
                    { "messageTemplate",e.messageTemplate},
                    { "failingStatusCode", statuscode},
                    { "failingException", exception}
                });
        },
        MinimumLogEventLevel = LogEventLevel.Verbose,
        AutoRegisterTemplate = true,
        AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
        CustomFormatter = new ExceptionAsObjectJsonFormatter(renderMessage: true),
        IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}",
        EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog | 
                               EmitEventFailureHandling.WriteToFailureSink |
                               EmitEventFailureHandling.RaiseCallback
    };
}
```
### Create GetDefinedPorthelper method
```csharp
private static int GetDefinedPort(IConfiguration config)
{
    var port = config.GetValue("PORT", 80);
    return port;
}
```
### Create CreateHostBuildermethod
```csharp
private static IWebHost CreateHostBuilder(IConfiguration configuration, string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .UseConfiguration(configuration)
        .CaptureStartupErrors(false)
        .ConfigureKestrel(options =>
        {
            var httpPort = GetDefinedPort(configuration);
            options.Listen(IPAddress.Any, httpPort, listenOptions =>
            {
                listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
            });
        })
        .UseStartup<Startup>()
        .UseContentRoot(Directory.GetCurrentDirectory())
        .UseSerilog()
        .Build();
```
### Create Main method
```csharp
public static readonly string Namespace = typeof(Program).Namespace;
public static readonly string AppName = Namespace.Substring(Namespace.LastIndexOf('.', Namespace.LastIndexOf('.') - 1) + 1);

public static int Main(string[] args)
{
    var configuration = GetConfiguration();
    Log.Logger = CreateSerilogLogger(configuration);
    try
    {
        Log.Information("Configuring web host ({ApplicationContext})", AppName);
        var host = CreateHostBuilder(configuration, args);
        Log.Information("Starting web host ({ApplicationContext})", AppName);
        host.Run();
        return 0;
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Program terminated unexpectedly ({ApplicationContext})!", AppName);
        return 1;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}
```
## Add the following code to Startup.cs

### Inject IConfigurationinto the constructure
```csharp
public Startup(IConfiguration configuration)
{
  	Configuration = configuration;
}
public IConfiguration Configuration { get; }
```
### Create ConfigureServices method
```csharp
public IServiceProvider ConfigureServices(IServiceCollection services)
{
    services.AddCustomMVC(Configuration);

    var container = new ContainerBuilder();
    container.Populate(services);

    return new AutofacServiceProvider(container.Build());
}
```
### Create Configuremethod
```csharp
public void Configure(IApplicationBuilder app, IWebHostEnvironment env, ILoggerFactory loggerFactory)
{
    var pathBase = Configuration["PATH_BASE"];

    if (!string.IsNullOrEmpty(pathBase))
    {
        loggerFactory.CreateLogger<Startup>().LogDebug("Using PATH BASE '{pathBase}'", pathBase);
        app.UsePathBase(pathBase);
    }

    app.UseCors("CorsPolicy");
    app.UseRouting();
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapDefaultControllerRoute();
        endpoints.MapControllers();
    });
}
```
### Create the CustomExtensionMethodsextension method
```csharp
public static class CustomExtensionMethods
{
    public static IServiceCollection AddCustomMVC(this IServiceCollection services, IConfiguration configuration)
    {
        services.AddControllers(options =>
        {

        }).AddNewtonsoftJson();

        services.AddCors(options =>
        {
            options.AddPolicy("CorsPolicy",
                builder => builder
                .SetIsOriginAllowed((host) => true)
                .AllowAnyMethod()
                .AllowAnyHeader()
                .AllowCredentials());
        });

        return services;
    }
}
```
## Add appsettings.json / appsettings.Development.json
It's very important to use the container name as ElasticSearchurl and not http://localhost:9200
```json
{
  "Serilog": {
    "SeqServerUrl": "http://seq",
    "LogstashgUrl": "http://locahost:8080",
    "ElasticConfiguration": "http://elasticsearch:9200",
    "MinimumLevel": {
          "Default": "Debug",
          "Override": {
            "Microsoft": "Debug",
            "CatalogAPI": "Debug",
            "MYSHOP": "Debug",
            "System": "Warning"
          }
        }
      }
  }
```
## Add Docker support to our solution

### Add Docker compose project to our solution
![Add Docker compose project to our solution](https://i1.wp.com/mselmi.net/wp-content/uploads/2020/04/2020-04-19-17_00_43-MyShop-Microsoft-Visual-Studio.png)
### Add Dockerfile to the project
```dockerfile
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/core/sdk:aspnet:3.1-buster AS build
WORKDIR /src

COPY "Services/Catalog/Catalog.API/CatalogAPI.csproj" "Services/Catalog/Catalog.API/CatalogAPI.csproj"
COPY "docker-compose.dcproj" "docker-compose.dcproj"

COPY "NuGet.config" "NuGet.config"

RUN dotnet restore "MyShop.sln"

COPY . .
WORKDIR /src/Services/Catalog/Catalog.API
RUN dotnet publish --no-restore -c Release -o /app

FROM build AS publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "CatalogAPI.dll"]
```
### Add docker-compose.yml to the docker project
```dockerfile
version: '3.4'

services:

  seq:
    image: datalust/seq:latest

  catalog.api:
    image: ${REGISTRY:-myshop}/catalogapi
    build:
      context: .
      dockerfile: Services/Catalog/CatalogAPI/Dockerfile

networks:
  elastic:
    driver: bridge

volumes:
  elasticsearchdata:
    external: true
```
### Add Docker docker-compose.overrides.yml to the docker project
```dockerfile
version: '3.4'

services:

  seq:
    environment:
      - ACCEPT_EULA=Y
    ports:
      - "5340:80"

  elasticsearch:
    build:
      context: elk/elasticsearch/
    volumes:
      - ./elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"

  logstash:
    build:
      context: elk/logstash/
    volumes:
      - ./elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./elk/logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "8080:8080"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: elk/kibana/
    volumes:
      - ./elk/kibana/config/:/usr/share/kibana/config:ro
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  catalog.api:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://0.0.0.0:80
      - PORT=80
      - PATH_BASE=/catalog-api
    ports:
      - "4201:80"
```
### Create Configuration for ElasticSearchand Kibana

#### Create the folder structure
Create "elk" folder in the root folder of the solution with following structure
![Create the folder structure](https://i2.wp.com/mselmi.net/wp-content/uploads/2020/04/2020-04-19-17_11_06-C__Work_Solutions_MyShop_elk.png)
#### Create elasticsearch configuration
Under elk/elasticsearch create new Dockerfileand new config folder like bellow:
![Create elasticsearch configuration](https://i2.wp.com/mselmi.net/wp-content/uploads/2020/04/2020-04-19-17_19_10-C__Work_Solutions_MyShop_elk_elasticsearch.png)
Add the following code to the Dockerfile
```dockerfile
# https://github.com/elastic/elasticsearch-docker
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2

# Add your elasticsearch plugins setup here
# Example: RUN elasticsearch-plugin install analysis-icu
```
Create new elasticsearch.yml under the config folder
```dockerfile
---
## Default Elasticsearch configuration from elasticsearch-docker.
## from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node
```
#### Create kibana configuration

Under elk/kibana create new Dockerfileand new config folder like bellow:
![](https://i2.wp.com/mselmi.net/wp-content/uploads/2020/04/2020-04-19-17_25_38-C__Work_Solutions_MyShop_elk_kibana.png)
Add the following code to the Dockerfile
```dockerfile
# https://github.com/elastic/kibana-docker
FROM docker.elastic.co/kibana/kibana-oss:7.6.2

# Add your kibana plugins setup here
# Example: RUN kibana-plugin install <name|url>
Create new kibana.yml under the config folder

## Default Kibana configuration from kibana-docker.
## from https://github.com/elastic/kibana-docker/blob/master/build/kibana/config/kibana.yml
#
server.name: kibana
server.host: "0"
elasticsearch.hosts: "http://elasticsearch:9200"
```
## Conclusion
We have seen how to configure .net core web application to log in Elasticsearch, Kibanaand Sequsing Serilog. This solution offers many advantages and I invite you to discover the range of possibilities on the Elastic and Seq websites.
The sources are here feel free to clone the solution and play with it.