Extending the shutdown timeout setting to ensure graceful IHostedService shutdown
Tags: aspnetcore, generichost, hostbuilder
Published: 2020-05-31 17:11:44

---
title: Extending the shutdown timeout setting to ensure graceful IHostedService shutdown
published: true
date: 2020-05-12 10:00:00 UTC
tags: ASP.NET Core,Generic Host,HostBuilder
canonical_url: https://andrewlock.net/extending-the-shutdown-timeout-setting-to-ensure-graceful-ihostedservice-shutdown/
---

 ![Extending the shutdown timeout setting to ensure graceful IHostedService shutdown](https://andrewlock.net/content/images/2020/timeout_banner.jpg)

I was seeing an issue recently where our application wasn't running the `StopAsync` method in our `IHostedService` implementations when the app was shutting down. It turns out that this was due to some services taking too long to respond to the shutdown signal. In this post I show an example of the problem, discuss why it happens, and how to avoid it.

## Running background services with IHostedService

ASP.NET Core 2.0 introduced [the `IHostedService` interface for running background tasks](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-2.2&tabs=visual-studio). The [interface consists of two methods](https://github.com/dotnet/runtime/blob/f007634b02c4484f60ead75a893eb7f89f615ccf/src/libraries/Microsoft.Extensions.Hosting.Abstractions/src/IHostedService.cs):

```csharp
public interface IHostedService
{
    Task StartAsync(CancellationToken cancellationToken);
    Task StopAsync(CancellationToken cancellationToken);
}
```

`StartAsync` is called when the application is starting up. In ASP.NET Core 2.x this occurs [just _after_](https://github.com/aspnet/AspNetCore/blob/v2.1.12/src/Hosting/Hosting/src/Internal/WebHost.cs#L153) the application starts handling requests, while in ASP.NET Core 3.x the hosted services are started [just _before_](https://github.com/aspnet/AspNetCore/blob/v3.0.0-preview9.19424.4/src/Hosting/Hosting/src/Internal/WebHost.cs#L154) the application starts handling requests.

`StopAsync` is called when the application receives the shut down (`SIGTERM`) signal, for example when you push <kbd>CTRL+C</kbd> in the console window, or the app is stopped by the host system. This allows you to close any open connections, dispose resources, and generally clean up your class as required.

In practice, there are actually some subtleties to implementing this interface that means that you typically want to derive [from the helper class `BackgroundService`](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-3.1&tabs=visual-studio#backgroundservice-base-class).

> Steve Gordon has a course on Pluralsight "[Building ASP.NET Core Hosted Services and .NET Core Worker Services](https://pluralsight.pxf.io/vdn6j)" if you want to learn more.

## Problems shutting down an `IHostedService` implementation

The problem I saw recently was causing an `OperationCanceledException` to be thrown when the application was shutting down:

```
Unhandled exception. System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
```

I traced the source of this problem to one particular `IHostedService` implementation. We use `IHostedService`s as the host for each of our [Kafka consumers](https://github.com/confluentinc/confluent-kafka-dotnet). The specifics of this aren't important - the key is just that shutting down the `IHostedService` is relatively slow: it can take several seconds to cancel the subscription.

> Part of the problem is the way the Kafka library (and underlying `librdkafka` library) uses synchronous, blocking `Consume` calls, instead of async, cancellable calls. There's not a great way around that.

The easiest way to understand this issue is with an example.

## Demonstrating the problem

The easiest way to understand the problem is to create an application containing two `IHostedService` implementations:

- `NormalHostedService` logs when it starts up and shuts down, then returns immediately. 
- `SlowHostedService` logs when it starts and stops, but takes 10s to complete shutdown

The implementations for these two classes are shown below. The `NormalHostedService` is very simple:

```csharp
public class NormalHostedService : IHostedService
{
    readonly ILogger<NormalHostedService> _logger;

    public NormalHostedService(ILogger<NormalHostedService> logger)
    {
        _logger = logger;
    }

    public Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("NormalHostedService started");
        return Task.CompletedTask;
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("NormalHostedService stopped");
        return Task.CompletedTask;
    }
}
```

The `SlowHostedService` is almost identical, but it has a `Task.Delay` that takes 10s to simulate a slow shutdown

```csharp
public class SlowHostedService : IHostedService
{
    readonly ILogger<SlowHostedService> _logger;

    public SlowHostedService(ILogger<SlowHostedService> logger)
    {
        _logger = logger;
    }

    public Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("SlowHostedService started");
        return Task.CompletedTask;
    }

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("SlowHostedService stopping...");
        await Task.Delay(10_000);
        _logger.LogInformation("SlowHostedService stopped");
    }
}
```

> The `IHostedService`s I had in practice only took 1s to shutdown, but we had many of them, so the overall effect was the same as above!

The order the services are registered in `ConfigureServices` is important in this case - to demonstrate the issue, we need `SlowHostedService` to be shut down _first_. Services are shut down in reverse order, which means we need to register it _last_:

```csharp
public void ConfigureServices(IServiceCollection services)
{
    services.AddHostedService<NormalHostedService>();
    services.AddHostedService<SlowHostedService>();
}
```

When we run the application, you'll see the starting logs as usual:

```
info: ExampleApp.NormalHostedService[0]
      NormalHostedService started
info: ExampleApp.SlowHostedService[0]
      SlowHostedService started
...
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
```

However, if you press <kbd>CTRL+C</kbd> to shut down the application, there's a problem. The `SlowHostedService` completes shutting down, but then an `OperationCanceledException` is thrown:

```
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
info: ExampleApp.SlowHostedService[0]
      SlowHostedService stopping...
info: ExampleApp.SlowHostedService[0]
      SlowHostedService stopped

Unhandled exception. System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at ExampleApp.Program.Main(String[] args) in C:\repos\andrewlock\blog-examples\SlowShutdown\Program.cs:line 16
```

The `NormalHostedService.StopAsync()` method is never called. If the service needed to do some cleanup then you have a problem. For example, maybe you need to gracefully deregister the service from Consul, or unsubscribe from Kafka topics - that won't happen now.

So what's going on here? Where's that timeout coming from?

## The reason: HostOptions.ShutDownTimeout

You can find the problematic [code in the framework `Host` implementation](https://github.com/dotnet/runtime/blob/db2375097e40394ef270f9e0cb56baa9ab392ad2/src/libraries/Microsoft.Extensions.Hosting/src/Internal/Host.cs#L64-L107) that runs on application shutdown. A simplified version is shown below:

```csharp
internal class Host: IHost, IAsyncDisposable
{
    private readonly HostOptions _options;
    private IEnumerable<IHostedService> _hostedServices;

    public async Task StopAsync(CancellationToken cancellationToken = default)
    {
        // Create a cancellation token source that fires after ShutdownTimeout seconds
        using (var cts = new CancellationTokenSource(_options.ShutdownTimeout))
        using (var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(cts.Token, cancellationToken))
        {
            // Create a token, which is cancelled if the timer expires
            var token = linkedCts.Token;

            // Run StopAsync on each registered hosted service
            foreach (var hostedService in _hostedServices.Reverse())
            {
                // stop calling StopAsync if timer expires
                token.ThrowIfCancellationRequested();
                try
                {
                    await hostedService.StopAsync(token).ConfigureAwait(false);
                }
                catch (Exception ex)
                {
                    exceptions.Add(ex);
                }
            }
        }

        // .. other stopping code
    }
}
```

The key point here is the `CancellationTokenSource` that is configured to fire after `HostOptions.ShutdownTimeout`. [By default, this fires after 5 seconds](https://github.com/dotnet/runtime/blob/f007634b02c4484f60ead75a893eb7f89f615ccf/src/libraries/Microsoft.Extensions.Hosting/src/HostOptions.cs). That means hosted service shutdown is abandoned after 5s - shut down of all `IHostedService`s has to happen within this timeout.

```csharp
public class HostOptions
{
    public TimeSpan ShutdownTimeout { get; set; } = TimeSpan.FromSeconds(5);
}
```

On the first iteration of the `foreach` loop, the `SlowHostedService.Stopasync()` executes, which takes 10s to run. On the second iteration, the 5s timeout is exceeded, and so `token.ThrowIfCancellationRequested();` throws an `OperationConcelledException`. That exits the control flow, and `NormalHostedService.Stopasync()` is never executed.

There's a simple solution to this - increase the shutdown timeout!

## The solution: increase the shutdown timeout

`HostOptions` isn't explicitly configured anywhere by default, so you will need to configure it manually in your `ConfigureSerices` method. For example, the following config increases the timeout to 15s:

```csharp
public void ConfigureServices(IServiceCollection services)
{
    services.AddHostedService<NormalHostedService>();
    services.AddHostedService<SlowShutdownHostedService>();

    // Configure the shutdown to 15s
    services.Configure<HostOptions>(
        opts => opts.ShutdownTimeout = TimeSpan.FromSeconds(15));
}
```

Alternatively, you can also load the timeout from configuration. For example, if you add the following to _appsettings.json_:

```json
{
    "HostOptions": {
        "ShutdownTimeout": "00:00:15"
    }
    // other config
}
```

You can then bind the `HostOptions` configuration section to the `HostOptions` object:

```csharp
public class Startup
{
    public IConfiguration Configuration { get; }
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddHostedService<NormalHostedService>();
        services.AddHostedService<SlowShutdownHostedService>();

        // bind the config to host options
        services.Configure<HostOptions>(Configuration.GetSection("HostOptions"));
    }
}
```

This binds the serialised `TimeSpan` value `00:00:15` to the `HostOptions` value and sets the timeout to 15s. With that configuration, now when we stop the application, the services all shutdown correctly:

```
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
info: SlowShutdown.SlowShutdownHostedService[0]
      SlowShutdownHostedService stopping...
info: SlowShutdown.SlowShutdownHostedService[0]
      SlowShutdownHostedService stopped
info: SlowShutdown.NormalHostedService[0]
      NormalHostedService stopped
```

Your application will now wait up to 15s for all the hosted services to finish shutting down before exiting!

## Summary

In this post I discussed an issue recently where our application wasn't running the `StopAsync` method in our `IHostedService` implementations when the app was shutting down. This was due to some background services taking too long to respond to the shutdown signal, and exceeding the shutdown timeout. I demonstrated the problem with a single service taking 10s to shutdown, but in practice it happens whenever the _total_ shutdown time for all services exceeds the default 5s.

The solution to the problem was to extend the `HostOptions.ShutdownTimeout` configuration value to be longer than 5s, using the standard ASP.NET Core `IOptions<T>` configuration system.