Prevent .NET Application Insights telemetry loss
Tags: dotnet, azure, csharp, tutorial
Published: 2023-05-16 12:00:00

If you're using the .NET Application Insights SDK to instrument and monitor your [ASP.NET Core](https://learn.microsoft.com/en-us/azure/azure-monitor/app/asp-net-core) or [worker service applications](https://learn.microsoft.com/en-us/azure/azure-monitor/app/worker-service), this blog post is for you.
We'll explore **some potential causes for data loss in your telemetry**, and how to fix them:

1. Losing telemetry data on application shutdown
2. Not capturing application logs
3. Failing to capture end-user information in the ASP.NET Core authentication middleware
4. Not capturing personal identifiable information (for internal web applications only)
5. Capturing too much personal identifiable information (for external web applications only)
6. Data retention not long enough
7. Using the wrong Application Insights SDK NuGet package for your workload

## Losing telemetry data on application shutdown

The .NET Application Insights SDK typically [sends telemetry data every 30 seconds or whenever the internal buffer is full (500 items by default)](https://learn.microsoft.com/en-us/azure/azure-monitor/app/api-custom-events-metrics#flushing-data).
This periodic flush happens in the background and is automatic, so you don't have to do anything.
You can learn how it works by looking at the SDK source code: [TelemetryBuffer.cs](https://github.com/microsoft/ApplicationInsights-dotnet/blob/2.21.0/BASE/src/ServerTelemetryChannel/Implementation/TelemetryBuffer.cs#L55-L56).

However, when your application shuts down, either due to a graceful shutdown or an unexpected error, **the .NET Application Insights SDK might not send the buffered telemetry data**.
[TelemetryBuffer does try to listen for application shutdown](https://github.com/microsoft/ApplicationInsights-dotnet/blob/2.21.0/BASE/src/ServerTelemetryChannel/Implementation/TelemetryBuffer.cs#L47) in order to flush the remaining telemetry data, but the `IApplicationLifecycle` that is supposed to provide the `Stopping` event listener is `null` in modern .NET applications (ASP.NET Core 6+, .NET 6+ workers).

This [.NET DevBlogs post from 2020](https://devblogs.microsoft.com/premier-developer/application-insights-use-case-for-telemetryclient-flush-calls/) provides the required code to send the buffered telemetry data on shutdown (using `TelemetryClient.Flush()` and `Thread.Sleep()`), but it is outdated.

Since [April 22, 2021](https://github.com/microsoft/ApplicationInsights-dotnet/pull/2202), you can use the new `TelemetryClient.FlushAsync()` task-based API with a custom `IHostedService` to ensure that you don't lose any telemetry data:

```csharp
internal sealed class ApplicationInsightsShutdownFlushService : IHostedService
{
    private readonly TelemetryClient _telemetryClient;

    public ApplicationInsightsShutdownFlushService(TelemetryClient telemetryClient)
    {
        this._telemetryClient = telemetryClient;
    }

    public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask;

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        // Flush the remaining telemetry data when application shutdown is requested.
        // Using "CancellationToken.None" ensures that the application doesn't stop until the telemetry data is flushed.
        //
        // If you want to use the "cancellationToken" argument, make sure to configure "HostOptions.ShutdownTimeout" with a sufficiently large duration,
        // and silence the eventual "OperationCanceledException" exception. Otherwise, you will still be at risk of losing telemetry data.
        var successfullyFlushed = await this._telemetryClient.FlushAsync(CancellationToken.None);
        if (!successfullyFlushed)
        {
            // Here you can handle the case where transfer of telemetry data to server has failed with non-retriable HTTP status.
        }
    }
}
```

Don't forget to register this custom hosted service in your dependency injection services:

```csharp
services.AddHostedService<ApplicationInsightsShutdownFlushService>();
```

## Not capturing application logs

You've added Application Insights to your dependency injection services, and it's now capturing traces for incoming HTTP requests and external dependencies such as outgoing HTTP requests and database queries:

```csharp
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddApplicationInsightsTelemetry();

// [...]
```

However, you might not be capturing application logs. To do so, you need to register the Application Insights logging provider:

```csharp
builder.Logging.AddApplicationInsights();
```

Once you've added the logging provider, you can configure it through the dedicated `Logging:ApplicationInsights` configuration section. Here's an example using `appsettings.json`:

```json
{
  "ApplicationInsights": {
    "ConnectionString": "<YOUR_CONNECTION_STRING>"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    },
    "ApplicationInsights": {
      "LogLevel": {
        // Configure log levels specific to Application Insights to better
        // control the volume of logs being sent
        "Default": "Information"
      }
    },
    "Console": {
      // ...
    }
  }
}
```

## Failing to capture end-user information in the ASP.NET Core authentication middleware

Setting the `Context.User.AuthenticatedUserId` ([documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/app/data-model-complete#authenticated-user-id)) on `ITelemetry` items is essential for several reasons:

- Identifying and tracking individual user behavior across multiple sessions and devices.
- Pinpointing errors or problems specific to certain users.
- Recognizing differences in behavior between authenticated and anonymous users.
- Maintaining an audit trail of user activity.

Unfortunately, the Application Insights SDK for ASP.NET Core applications doesn't automatically handle this. To remedy this, you must register a custom `ITelemetryInitializer` that extracts the current authenticated user ID (typically from the current `ClaimsPrincipal`, but it can come from elsewhere) and assigns it to telemetry items. Here's an example that extracts the authenticated user's name:

```csharp
internal sealed class PrincipalTelemetryEnrichment : TelemetryInitializerBase
{
    public PrincipalTelemetryEnrichment(IHttpContextAccessor httpContextAccessor)
        : base(httpContextAccessor)
    {
    }

    protected override void OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
    {
        if (!string.IsNullOrEmpty(telemetry.Context.User.AuthenticatedUserId))
        {
            return;
        }

        if (platformContext.User.Identity is { IsAuthenticated: true, Name: { Length: > 0 } username })
        {
            telemetry.Context.User.AuthenticatedUserId = username;
        }
    }
}
```

You can register this `ITelemetryInitializer` as a singleton:

```csharp
services.TryAddEnumerable(ServiceDescriptor.Singleton<ITelemetryInitializer, PrincipalTelemetryEnrichment>());
```

The issue here is that `HttpContext.User`, which is a `ClaimsPrincipal`, **is only set at the end** of the ASP.NET Core authentication middleware. It's common to implement events like `OpenIdConnectEvents.OnTokenValidated` or `JwtBearerEvents.OnTokenValidated` to register custom behavior just before the end of the authentication process. However, if an error occurs at this stage, the associated telemetry will not contain the user ID being authenticated.

To fix this, you can store the ID of the user being authenticated in a location such as:
- The `HttpContext.Items` property,
- A custom scoped service,
- A custom `AsyncLocal`-based property.

Then, modify the `ITelemetryInitializer` to retrieve this stored user ID:

```csharp
services.AddAuthentication().AddOpenIdConnect(options =>
{
    options.Events.OnTokenValidated = context =>
    {
        // Store the ID of the user being authenticated for it to be read by the telemetry initializer
        if (context.HttpContext.User.Identity is { IsAuthenticated: true, Name: { Length: > 0 } username })
        {
            context.HttpContext.Items["AuthenticatedUserId"] = username;
        }
        
        // [...] Do something like register the user in a database, or whatever
        return Task.CompletedTask;
    };
});

// Change the telemetry initializer to read that stored authenticated user ID
internal sealed class PrincipalTelemetryEnrichment : TelemetryInitializerBase
{
    public PrincipalTelemetryEnrichment(IHttpContextAccessor httpContextAccessor)
        : base(httpContextAccessor)
    {
    }

    protected override void OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
    {
        if (!string.IsNullOrEmpty(telemetry.Context.User.AuthenticatedUserId))
        {
            return;
        }

        if (platformContext.User.Identity is { IsAuthenticated: true, Name: { Length: > 0 } identityName })
        {
            telemetry.Context.User.AuthenticatedUserId = identityName;
        }
        else if (platformContext.Items.TryGetValue("AuthenticatedUserId", out var usernameObj) && usernameObj is string authUserId)
        {
            // If the authentication process failed, at least we had a chance to retrieve the user ID
            telemetry.Context.User.AuthenticatedUserId = authUserId;
        }
    }
}
```

## Not capturing personal identifiable information (for internal web applications only)

> 🛑 This section only concerns internal enterprise web applications.

In an ASP.NET Core application, the .NET Application Insights SDK, by default, captures the user IP address. It then performs a geolocation lookup to populate the fields `client_City`, `client_StateOrProvince`, and `client_CountryOrRegion`. The IP address is subsequently discarded, and `0.0.0.0` is written to the `client_IP` field.

For internal enterprise web applications, discarding the IP address might be seen as a loss of valuable information. Fortunately, you can configure the Azure Application Insights resource to enable both IP collection and storage with the `DisableIpMasking` property.

[Refer to this documentation page](https://learn.microsoft.com/en-us/azure/azure-monitor/app/ip-collection#storage-of-ip-address-data) to learn how to change this property using infrastructure as code or directly in the Azure Portal.

## Capturing too much personal identifiable information (for external web applications only)

On the other hand, if you're building an application for external customers, you should respect their privacy and avoid collecting personal identifiable information (PII). The default city, state or province, and country or region fields populated by Application Insights could be considered as PII.

To disable the collection of these fields, you can register a custom `ITelemetryInitializer` that removes the collected IP address:

```csharp
internal sealed class RemoveGeolocationTelemetryInitializer : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        // Prevents Application Insights from extracting geolocation
        telemetry.Context.Location.Ip = "0.0.0.0";
    }
}
```

## Data retention not long enough

By default, Application Insights retains your telemetry data for 90 days. However, in some cases, you might need to store the data for longer periods, either for compliance reasons or deeper analysis. Make sure to check your data retention settings:

- Navigate to your Log Analytics workspace.
- In the left menu, click on "Usage and estimated costs".
- Click on on the "Data Retention" button.
- Move the slider to adjust the retention duration between 30 and 730 days, according to your needs.

## Using the wrong Application Insights SDK NuGet package for your workload

This final piece of advice is not specifically about the loss of telemetry data. However, not using the appropriate Application Insights SDK NuGet package could result in missing the opportunity to register the right built-in telemetry initializers for your workload.

Essentially, if you're building an ASP.NET Core web application, you'll want to install the [Microsoft.ApplicationInsights.AspNetCore](https://www.nuget.org/packages/Microsoft.ApplicationInsights.AspNetCore) NuGet package. It comes with numerous built-in web-specific telemetry initializers.

However, if you're building a console application, a background worker, or any non-web-based application, you should use the [Microsoft.ApplicationInsights.WorkerService](https://www.nuget.org/packages/Microsoft.ApplicationInsights.WorkerService) NuGet package, which doesn't include references to ASP.NET Core.