Fault Resilience in Web Calls Using Polly
Tags: csharp
Published: 2019-12-25 09:31:13

Some time ago, I was working on a project using WCF RPC calls.  We discovered that of the calls that we made, around 10 - 20% were failing.  Once we discovered this, I implemented a retry policy to try and cope with the flaky network.  I didn't realise at the time (or maybe it didn't exist at the time) that there's a library out there that does that for you: <a href="https://github.com/App-vNext/Polly">Polly</a>.

To test this out, let's create a bog standard API Web Project; from Powershell create a directory; e.g.:

```
mkdir PollyDemo
```

Then create a Web API project:

```
dotnet new webapi -n UnreliableApi
```

<a href="https://www.pmichaels.net/wp-content/uploads/2019/12/polly-0.png"><img src="https://www.pmichaels.net/wp-content/uploads/2019/12/polly-0.png" alt="" width="339" height="272" class="alignnone size-full wp-image-4039" /></a>

Let's add a client project console app:

```csharp
    static async Task Main(string[] args)
    {

        HttpClient httpClient = new HttpClient()
        {
            BaseAddress = new Uri("https://localhost:44368")
        };

        var result = await httpClient.GetAsync("weatherforecast");
        if (result.IsSuccessStatusCode)
        {
            var output = await result.Content.ReadAsStringAsync();
            Console.WriteLine(output);
        }
    }
```


This should call the API fine (using the default API project you get a weather forecaster).  Let's now introduce a level of entropy into the system.  In the server, let's add the following code:

```csharp
        static Random _rnd = new Random();

        [HttpGet]
        public IEnumerable<WeatherForecast> Get()
        {
            // 1 in 3 chance of an error
            if (_rnd.Next(3) == 1)
            {
                throw new Exception("Bad stuff happened here");
            }

            var rng = new Random();
            return Enumerable.Range(1, 5).Select(index => new WeatherForecast
            {
                Date = DateTime.Now.AddDays(index),
                TemperatureC = rng.Next(-20, 55),
                Summary = Summaries[rng.Next(Summaries.Length)]
            })
            .ToArray();
        }
```

Now, if you run the client a few times, you'll see that it's occasionally failing (actually, from this screenshot, all you can really see if that it didn't succeed):

<a href="https://www.pmichaels.net/wp-content/uploads/2019/12/polly-1.png"><img src="https://www.pmichaels.net/wp-content/uploads/2019/12/polly-1.png" alt="" width="979" height="512" class="alignnone size-full wp-image-4036" /></a>

We know that this error is transient - just needs an F5, so we can put some code into the client to handle this:

```csharp
for (int i = 1; i <= 3; i++)
{
    var result = await httpClient.GetAsync("weatherforecast");
    if (result.IsSuccessStatusCode)
    {
        var output = await result.Content.ReadAsStringAsync();
        Console.WriteLine(output);
        break;
    }
    else
    {
        Console.WriteLine("Bad things happened... ");
    }
}

Console.WriteLine("If we get here, we either succeeded or gave up ");

```

And we're back in business:

<a href="https://www.pmichaels.net/wp-content/uploads/2019/12/polly-2.png"><img src="https://www.pmichaels.net/wp-content/uploads/2019/12/polly-2.png" alt="" width="979" height="512" class="alignnone size-full wp-image-4035" /></a>

The only problem here is that our code is suddenly a little cumbersome; we have a foreach loop around the code, and we're risking each person implementing this doesn't introduce a bug by not breaking when successful or whatever.

Before we introduce Polly as the solution, it's probably worth mentioning that, whilst some errors can legitimately be fixed by simply trying again, in other cases this can be incredibly harmful.  You should ensure that your code is <a href="https://en.wikipedia.org/wiki/Idempotence">idempotent</a>.  Imagine you're creating a sales order on this call, and it fails after writing something to the database - you risk creating multiple sales orders!

<h1>Polly</h1>

Polly is basically a library built around this concept.  It's a NuGet package, so import it from <a href="https://www.nuget.org/packages/Polly/">here</a> into your client.  Let's start by refactoring our code to call the service:

```csharp
        private static async Task<bool> GetTheWeather(HttpClient httpClient)
        {
            var result = await httpClient.GetAsync("weatherforecast");
            if (result.IsSuccessStatusCode)
            {
                var output = await result.Content.ReadAsStringAsync();
                Console.WriteLine(output);
                return true;
            }
            else
            {
                Console.WriteLine("Bad things happened... ");
                return false;
            }
        }
```

Now let's call that using Polly:

```csharp
        static async Task Main(string[] args)
        {

            //https://localhost:44368/weatherforecast

            HttpClient httpClient = new HttpClient()
            {
                BaseAddress = new Uri("https://localhost:44368")
            };

            var retryPolicy = Policy
                .HandleResult<bool>(false)               
                .RetryAsync(3, (response, retryCount) =>
                  {
                      Console.WriteLine($"Received a response of {response.Result}, retrying {retryCount}.");
                  });

            var result = await retryPolicy.ExecuteAsync(() => GetTheWeather(httpClient));            

            Console.WriteLine("If we get here, we either succeeded or gave up ");
        }
```

All we're doing here is telling Polly that we want a policy that acts on a return value of False (in theory, I imagine you could set this to something like .HandleResult<string>("Aardvark") and have it retry while the method returned a value of Aardvark).  RetryAsync sounds pretty obvious, but the Async part is very important, otherwise, you won't be able to use ExecuteAsync... and you'll spend a while wondering why (so I hear)!

ExecuteAsync is awaitable, so it will wrap the retry logic in this single line.

The advantage here is that you can define a retry policy for your application, or several retry policies for your application.






<h1>References</h1>

<a href="https://github.com/App-vNext/Polly">https://github.com/App-vNext/Polly</a>

<a href="https://github.com/bryanjhogan/trydotnet-polly">https://github.com/bryanjhogan/trydotnet-polly</a>