Simplify observability .NET with OpenTelemetry Collector
Tags: observability, dotnet, opentelemetry, collector
Published: 2023-04-25 05:26:45

## Overview
The Observability concept become the standard of almost system in recently. It helps team to troubleshoot what's happening inside the system. There are 3 pillars of Observability - Traces, Metrics and Logs. 

- [OpenTelemetry](https://opentelemetry.io/) is the most straightforward way to collect Traces and Metrics. Then export to [Zipkin](https://zipkin.io/) or [Jaeger](https://www.jaegertracing.io/) for tracings; [Prometheus](https://prometheus.io/)

- [Serilog](https://serilog.net/) is also the most straightforward way to collect logs and then export to [Seq](https://datalust.co/seq) or [ElasticSearch](https://www.elastic.co/)

Because of the various exporting ways, we have to consider one of these options when implementing
- Support all type of exporting then toggle via settings. For example, only export to Zipkin if it's enabled
- Or, just only export to Zipkin or Jaeger

## Only one answer for the concerns
:question: Concerns
- Is there any way that just only one export for multiple consumers? Or,
- Is there any way that just only one export but change consumer without changing the code?

:star2: Only one answer
- And luckily, there is an elegant way to accomplish them - that is [OpenTelemetry Collector (OTEL Collector in short)](https://opentelemetry.io/docs/collector/)

> **_Objectives_**
> - **Usability:** Reasonable default configuration, supports popular protocols, runs and collects out of the box.
> - **Performance:** Highly stable and performant under varying loads and configurations.
> - **Observability:** An exemplar of an observable service.
> - **Extensibility:** Customizable without touching the core code.
> - **Unification:** Single codebase, deployable as an agent or collector with support for traces, metrics, and logs (future).

- An image more than thousand words


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/67r8o03g17fmi9b9ph2k.jpeg)


## :computer: Let our hand dirty

---

:point_right: The below steps are just the showcase of using **OTEL Collector** within .NET 7. The full of implementation can be found at - [.NET with OpenTelemetry Collector](https://github.com/kimcuhoang/practical-net-otelcollector)

:point_right: In which, we'll export the telemetry signals from application to **OTEL Collector** then they'll be exported to - [Zipkin](https://zipkin.io/) or [Jaeger](https://www.jaegertracing.io/) for tracings; [Prometheus](https://prometheus.io/); and [Loki](https://grafana.com/oss/loki/) for logs

---

### Nuget packages [Directory.Packages.props](https://github.com/kimcuhoang/practical-net-otelcollector/blob/main/Directory.Packages.props)

```xml

<!-- OpenTelemetry: Traces & Metrics -->
<ItemGroup>
	<PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.4.0" />
	<PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.4.0" />
	<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.14" />
</ItemGroup>

<!-- Serilog -->
<ItemGroup>
	<PackageVersion Include="Serilog.AspNetCore" Version="6.1.0" />
	<PackageVersion Include="Serilog.Enrichers.Context" Version="4.6.0" />
	<PackageVersion Include="Serilog.Sinks.OpenTelemetry" Version="1.0.0-dev-00129" />
	<PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol.Logs" Version="1.4.0-rc.4" />
</ItemGroup>

```

### Register OpenTelemetry, typically from `Program.cs`

```csharp

var builder = WebApplication.CreateBuilder(args);

builder.Host.AddSerilog();

builder.Services
       .AddOpenTelemetry()
       .AddTracing(observabilityOptions)
       .AddMetrics(observabilityOptions);

```

### Configure Tracings

```csharp
private static OpenTelemetryBuilder AddTracing(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
    builder.WithTracing(tracing =>
    {
       tracing
                .AddSource(observabilityOptions.ServiceName)
                .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(observabilityOptions.ServiceName))
                .SetErrorStatusOnException()
                .SetSampler(new AlwaysOnSampler())
                .AddAspNetCoreInstrumentation(options =>
                {
                    options.EnableGrpcAspNetCoreSupport = true;
                    options.RecordException = true;
                });

        /* Add more instrument here: MassTransit, NgSql ... */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */
        tracing
                .AddOtlpExporter(options =>
                {
                    options.Endpoint = observabilityOptions.CollectorUri;
                    options.ExportProcessorType = ExportProcessorType.Batch;
                    options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
                });
        });

        return builder;
}
```

### Configure for Metrics

```csharp
private static OpenTelemetryBuilder AddMetrics(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
        builder.WithMetrics(metrics =>
        {
            var meter = new Meter(observabilityOptions.ServiceName);

       metrics
                .AddMeter(meter.Name)
                .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(meter.Name))
                .AddAspNetCoreInstrumentation();

        /* Add more instrument here */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        metrics
                .AddOtlpExporter(options =>
                {
                    options.Endpoint = observabilityOptions.CollectorUri;
                    options.ExportProcessorType = ExportProcessorType.Batch;
                    options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
                });
        });

        return builder;
}
```

### Configure Logs

```csharp
public static IHostBuilder AddSerilog(this IHostBuilder hostBuilder)
{
     hostBuilder
            .UseSerilog((context, provider, options) =>
            {
                var environment = context.HostingEnvironment.EnvironmentName;
                var configuration = context.Configuration;

                ObservabilityOptions observabilityOptions = new();

                configuration
                    .GetSection(nameof(ObservabilityOptions))
                    .Bind(observabilityOptions);

                var serilogSection = $"{nameof(ObservabilityOptions)}:{nameof(ObservabilityOptions)}:Serilog";

                options
                    .ReadFrom.Configuration(context.Configuration, serilogSection)
                    .Enrich.FromLogContext()
                    .Enrich.WithEnvironment(environment)
                    .Enrich.WithProperty("ApplicationName", observabilityOptions.ServiceName);

                 /* ============== */
                 /* Only export to OpenTelemetry collector */
                 /* ============== */
                 options.WriteTo.OpenTelemetry(cfg =>
                 {
                     cfg.Endpoint = $"{observabilityOptions.CollectorUrl}/v1/logs";
                     cfg.IncludedData = IncludedData.TraceIdField | IncludedData.SpanIdField;
                     cfg.ResourceAttributes = new Dictionary<string, object>
                                                {
                                                    {"service.name", observabilityOptions.ServiceName},
                                                    {"index", 10},
                                                    {"flag", true},
                                                    {"value", 3.14}
                                                };
                  });
               
        });
        return hostBuilder;
}
```

### The interesting here

:one: - Refer to [docker-compose.observability.yaml](https://github.com/kimcuhoang/practical-net-otelcollector/blob/main/local/infra/docker-compose.observability.yaml)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bfg7objnvbcn0e9ch45o.png)

:two: - Refer to [otel-collector.yaml](https://github.com/kimcuhoang/practical-net-otelcollector/blob/main/local/infra/config-files/otel-collector.yaml) to configure OTEL Collector

```yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  logging:
    loglevel: info

  prometheus:
    endpoint: 0.0.0.0:8889

  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"
    format: proto

  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    format: json
    labels:
      resource:
        service.name: "service_name"
        service.instance.id: "service_instance_id"

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, jaeger, zipkin]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, prometheus]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, loki]
```


---

Let's :thumbsup: **OTEL Collector** and take fully implementation at - [.NET with OpenTelemetry Collector](https://github.com/kimcuhoang/practical-net-otelcollector)

Cheers!!! :beers: