What every ASP.NET Core Web API project needs - Part 5 - Polly
Tags: architecture, dotnet, aspnetcore, webapi
Published: 2021-03-25 13:02:11

The 4th part of this series was about [Error Message Reusability and Localization](https://dev.to/moesmp/what-every-asp-net-core-web-api-project-needs-part-4-error-message-reusability-and-localization-229i) and in this article, I'm going to show you how to add resilience and transient fault handling to `HttpClient` with Polly.

When you create an ASP.NET Core Web API project, there is a `WeatherForecastController` that returns a list of weather forecasts. I'm going to use external weather API to get real weather forecasts instead of returning static data.

###Step 1 - External API
- Go to the [weatherapi.com](https://www.weatherapi.com/) 
- Create an account
- After login, you find the API key at the dashboard
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ad0qqbxqhla1yqejcqfc.jpg)

###Step 2 - Typed HTTP Client
There are several ways to use [HttpClient](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-5.0#consumption-patterns) class in a project:
* Basic usage
* Named clients
* Typed clients
* Generated clients

Most of the time I Prefer to use typed client because of:
* Provide the same capabilities as named clients without the need to use strings as keys
* Provides IntelliSense and compiler help when consuming clients
* Provide a single location to configure and interact with a particular HttpClient. For example, a single typed client might be used:
  * For a single backend endpoint.
  * To encapsulate all logic dealing with the endpoint
* Work with DI and can be injected where required in the app

Let's create a typed client for weather external API:
- Create a new folder `HttpClients` inside `Infrastructure` folder
- Add new class `WeatherHttpClient.cs` to `HttpClients` folder
- Open `appsettings.json` file and add the following key/values:
```json
"WeatherSettings": {
  "ApiKey": "YOURKEY"
  "BaseUrl": "https://api.weatherapi.com",
  "NoDaysForecast": 5
} 
```
- Open `WeatherHttpClient.cs` file and create a class for API settings:
```csharp
public class WeatherSettings
{
    public string ApiKey { get; set; }

    public string BaseUrl { get; set; }

    public int NoDaysForecast { get; set; }
}
```
- Open `Startup.cs` class and inside `ConfigureServices` method bind and register weather settings:
```csharp
var weatherSettings = new WeatherSettings();
Configuration.GetSection("WeatherSettings").Bind(weatherSettings);
services.AddSingleton(weatherSettings);
```
- Inside `WeatherHttpClient.cs` file create an interface:
```csharp
public interface IWeatherHttpClient
{
    Task<IEnumerable<WeatherForecast>> GetForecastAsync(string cityName);
}
```
- Let's implement the above interface:
```csharp
public class WeatherHttpClient : IWeatherHttpClient
{
    private readonly HttpClient _client;
    private readonly WeatherSettings _settings;

    public WeatherHttpClient(HttpClient client, WeatherSettings settings)
    {
        _client = client;
        _settings = settings;
        _client.BaseAddress = new Uri(_settings.BaseUrl);
        _client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    }

    public async Task<IEnumerable<WeatherForecast>> GetForecastAsync(string cityName)
    {
        var url = $"v1/forecast.json?key={_settings.ApiKey}&q={cityName}&days={_settings.NoDaysForecast}";
        var response = await _client.GetAsync(url);
        response.EnsureSuccessStatusCode();
        var content = await response.Content.ReadAsStringAsync();
        ...
    }
}
```
The response of API is really verbose and I'm not going to create  an object for deserializing JSON instead I use an anonymous object to extract only data that I need:
```csharp
public async Task<IEnumerable<WeatherForecast>> GetForecastsAsync(string cityName)
{
    ...
    var days = JsonSerializerExtensions.DeserializeAnonymousType(content, new
    {
        forecast = new
        {
            forecastday = new[]
            {
               new
               {
                   date = DateTime.Now,
                   day = new { avgtemp_c = 0.0, condition = new { text = "" } }
               }
            }
        }
    }).forecast.forecastday;

    return days.Select(d => new WeatherForecast
    {
        Date = d.date,
        Summary = d.day.condition.text,
        TemperatureC = (int)d.day.avgtemp_c
    });

    // Other way to deserialize json without creating anonymous object
    // To get more information see https://docs.microsoft.com/en-us/dotnet/api/system.text.json.jsonelement?view=net-5.0
    //dynamic result = JsonSerializer.Deserialize<ExpandoObject>(content);
    //var days = result.forecast.GetProperty("forecastday").EnumerateArray();
    //foreach (var day in days)
    //{
    //    var date = day.GetProperty("date").GetDateTime();
    //    var temp = day.GetProperty("day").GetProperty("avgtemp_c").GetDouble();
    //    var condition = day.GetProperty("day").GetProperty("condition").GetProperty("text").GetString();
    //}
}
```
- Open `Startup.cs` class and register weather HTTP client service:
```csharp
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>();
```
- Open `WeatherForecastController` and inject `IWeatherHttpClient`:
```csharp
namespace CoolWebApi.Apis.V1.Controllers
{
    [ApiController]
    [ApiVersion("1.0")]
    [Route("api/v{version:apiVersion}/[controller]")]
    public class WeatherForecastController : ControllerBase
    {
        private readonly IWeatherHttpClient _weatherClient;

        public WeatherForecastController(IWeatherHttpClient weatherClient)
        {
            _weatherClient = weatherClient;
        }

        [HttpGet]
        [ProducesResponseType(StatusCodes.Status200OK)]
        [ProducesResponseType(StatusCodes.Status400BadRequest)]
        public async Task<IEnumerable<WeatherForecast>> Get(string city = "London")
        {
            return await _weatherClient.GetForecastsAsync(city);
        }
```
It's time to simulate an error when we call weather API. Hence we have no control over external API, I use [Fiddler Everywhere](https://www.telerik.com/fiddler) to capture and fiddle HTTP request and response.
 
###Step 3 - Using Fiddler
- [Download](https://www.telerik.com/download/fiddler-everywhere) and install Fiddler Everywhere
- Run Fiddler and call the `api/v1/weatherforecast` API
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qrst7yzv7dfnq6mzsd6o.jpg)
As you can see Fiddler captured HTTP traffics but we cannot see the request/response of HTTPS calls because they are encrypted, however, Fiddler has a mechanism to decrypt HTTPS traffics.
- On the menu click `View->Preferences` or hit `ctrl+,` to open `Settings`
- In the Settings window click on the `HTTPS` tab and click on `Trust root certificate` (You can remove the certificate anytime by clicking on `Remove root certificate`) 
- Call the API again and now you can see decrypted request/response
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nyki5fh1oqkjnx766gi3.jpg)
Let's filter captured traffic to get rid of other requests. 
- On the `URL` column click on vertical dots to open the filter window and enter `localhost:5001` and `api.weatherapi.com` URLs and change `And` to `Or`:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m6w71yt5yookq4a3vtpw.jpg)
Now it's time to change the response of `api.weatherapi.com` API.
- On the `Live Traffic` tab right-click on `api.weatherapi.com` row and from the menu click on `Add new rule`(1). In the `Auto Responder` tab click on the switch button to enable it(2) then click on the edit icon(3)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/itli2tjg1zfgjam0xt41.jpg)
- On the `Rule Editor` window clear `Raw` input and the following text then click on the Save button:
```
HTTP/1.1 503

Service is unavailable
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rs5m3fhpxi1bv8dgns66.jpg)
Let's test the rule. Again right-click on `api.weatherapi.com` row and from the menu click on `Reply->Reissue requests` and now API returns `503` instead of `200` status code
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t77c1vq07078rpymu08w.gif)
Try to call API with the Swagger and you will receive an `Internal Server Error`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1hw1igr6s9rbjap2rs3v.jpg)

###Step 4 - Installing Polly
- Install `Microsoft.Extensions.Http.Polly` nuget package
- Open `Startup.cs` class and in `ConfigureServices` method modify `WeatherHttpClient` registration to this:
```csharp
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.WaitAndRetryAsync(3, _ => TimeSpan.FromSeconds(4)));
```
Now `WeatherHttpClient` retries 3 times and waits 4 seconds between each retry. Let's try again and see how many API calls will be made by `WeatherHttpClient`:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2mmlkashzgpz89ts1hh7.gif)
Let's try one more time and during API call I will turn off Fiddler Auto Responder:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y1r6mdz8qeo6u45j6k3y.gif)
This time after turning off Auto Responder API returns the status code 200 and forecast result.
If you hover your mouse on the `AddTransientHttpErrorPolicy` method, you get noticed that `PolicyBuilder` handles the following error categories:
* Network failures (as System.Net.Http.HttpRequestException)
* HTTP 5XX status codes (server errors)
* HTTP 408 status code (request timeout)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tjp7rer0qex9pq5f5sjf.jpg)
For example, no retry will be made on HTTP error code 400. 

Polly offers multiple resilience policies:
- Retry
- Circuit-breaker
- Timeout
- Bulkhead Isolation
- Cache
- Fallback
- PolicyWrap

Failing fast is better than making users/callers wait. If the external service is down or seriously struggling, it's better to give that system a break. In this case we can chain multiple policies in Polly to give a break. Let's chain retry policy with circuit breaker policy.
```csharp
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => 
        policy.WaitAndRetryAsync(2, _ => TimeSpan.FromSeconds(2)))
    .AddTransientHttpErrorPolicy(policy => 
        policy.CircuitBreakerAsync(2, TimeSpan.FromSeconds(5)));
```
We want to retry 2 times and wait 2 seconds between each retry and after 4 failed retries, stop for 5 seconds.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rxn6ovjvvwjw8m4gijsg.gif)

#### More examples:
- Different retry times:
```csharp
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.WaitAndRetryAsync(new[]
        {
            TimeSpan.FromSeconds(2),
            TimeSpan.FromSeconds(6),
            TimeSpan.FromSeconds(10)
        }));
```
This policy will delay 2 seconds before the first retry, 6 seconds before the second retry, and 10 seconds before the third retry.
- Handle other status codes
```csharp
var policy = HttpPolicyExtensions
  .HandleTransientHttpError()
  .OrResult(response => (int)response.StatusCode == 417) // RetryAfter
  .WaitAndRetryAsync(...);
```
As I mentioned earlier by default `HandleTransientHttpError` handles `HttpRequestException`, `5XX` and `408` errors. The above policy can handle the 429 status code too.
- Custom error handling logic:
```csharp
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.RetryAsync(3, onRetry: (exception, retryCount) =>
    {
        //Add logic to be executed before each retry
    }));
```
- Retry forever until it succeeds
```csharp
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.RetryForeverAsync());
``` 
- Advanced Circuit Breaker
```csharp
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.AdvancedCircuitBreakerAsync(
        failureThreshold: 0.5,
        samplingDuration: TimeSpan.FromSeconds(10),
        minimumThroughput: 8,
        durationOfBreak: TimeSpan.FromSeconds(30)
    ));
```
Breaks the circuit for 30 seconds if 50% of above of incoming requests fails or a minimum of 8 faults occur within a 10 second period. The circuit is reset/closed after 30 seconds.
- Fallback
```csharp
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => 
        policy.FallbackAsync(new HttpResponseMessage(HttpStatusCode.RequestTimeout)));
```
The fallback technique helps to return a fallback value instead of an exception re-throw when faults keep occurring. This helps the system to ensure that it gracefully tries to keep the system stable when it detects a fallback value coming by.
- Timeout
```csharp
var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(10);
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddPolicyHandler(timeoutPolicy);
```
Timeout policy allows you to specify how long a request should take to respond and if it doesn’t respond in the time period you specify, the request will be canceled.
- Dynamic selection strategy
```csharp
var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(10);
var noOpPolicy = Policy.NoOpAsync().AsAsyncPolicy<HttpResponseMessage>();
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddPolicyHandler(request => request.Method == HttpMethod.Get 
        ? retryPolicy 
        : noOpPolicy
);
```
You may want to define a strategy that will only apply to `GET` requests but not other HTTP verbs. In the above example, the timeout policy will only be used for `GET` requests.
- Policy Registry
```csharp
var registry = services.AddPolicyRegistry();
registry.Add("DefaultRetryStrategy", HttpPolicyExtensions.HandleTransientHttpError().WaitAndRetryAsync(...));
registry.Add("DefaultCircuitBreaker", HttpPolicyExtensions.HandleTransientHttpError().CircuitBreakerAsync(...));

// Or

var registry = new PolicyRegistry()
{
    { "DefaultRetryStrategy", HttpPolicyExtensions.HandleTransientHttpError().WaitAndRetryAsync(...) },
    { "DefaultCircuitBreaker", HttpPolicyExtensions.HandleTransientHttpError().CircuitBreakerAsync(...) }
};
services.AddSingleton<IReadOnlyPolicyRegistry<string>>(registry);
```
```csharp
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddPolicyHandlerFromRegistry("DefaultCircuitBreaker")
    .AddPolicyHandlerFromRegistry("DefaultCircuitBreaker");
```
Polly provides a policy registry, it is equivalent to the strategy storage center, the registered strategy allows you to reuse it in multiple locations in the application.

You can find the source code for this walkthrough on [Github](https://github.com/mo-esmp/cool-webapi).