Learn about the Retry Pattern in 5 minutes
Tags: architecture, pattern, beginners
Published: 2019-08-20 00:00:00

---
title: Learn about the Retry Pattern in 5 minutes
date: "2019-08-20"
published: true
cover_image: https://thepracticaldev.s3.amazonaws.com/i/kmlnmes1ddm3jyayq4l2.jpg
tags: 
- architecture
- pattern
- beginners
---

Last week we checked out the [circuit-breaker](https://dev.to/azure/learn-about-the-circuit-breaker-pattern-in-5-minutes-1l45). This week, we'll examine the retry pattern. The basis for the retry pattern is to handle transient failures to a service or network resource. Requests temporarily fail for many reasons. Examples of these failures include the network having a faulty connection, a site is reloading after a deployment, or data hasn't propagated to all instances.

In the following examples, I'm going to base failures off of this intentionally simplified client/service request model. A successful request:

![Successful Request](https://thepracticaldev.s3.amazonaws.com/i/os1tbkj2k3ecp5j340fa.png)

The retry pattern encapsulates more than the idea of just retrying but different ways to categorize issues and make choices around retrying. 

If the problem is a rare error, retry the request immediately:

![Retry immediately if the error is rare](https://thepracticaldev.s3.amazonaws.com/i/6b2irl709igwgx94utaq.png)


If the problem is a more common error, retry the request after some amount of a delay:

![Increasing Interval Delay](https://thepracticaldev.s3.amazonaws.com/i/ct55r8girn90rdosjw51.png)


If the issue indicates that the failure isn't transient, for example, invalid credentials that are unlikely to succeed on subsequent requests, cancel the request:

![Forbidden issue implies a permanent issue.](https://thepracticaldev.s3.amazonaws.com/i/nmaaswzwq2t5lj7ng2b8.png)

Sometimes the client may think that there is an error, but the operation has successfully completed. If the request isn't an idempotent operation, this can cause problems if a state change occurs and the request is duplicated:

![Network error masking successful request.](https://thepracticaldev.s3.amazonaws.com/i/6346p0cgi54oarwblwmf.png)


Regardless of the type of the issue, some tracking should be done to ensure that endless retries aren't occurring. Implementing a [circuit-breaker](https://dev.to/azure/learn-about-the-circuit-breaker-pattern-in-5-minutes-1l45) in this situation can be helpful to limit the impact of a retry storm on a failed or recovering service. 

When operating a service, it's important to track metrics around retries. Metrics can tell you a lot about your customers' experiences with your service. It might be a case that a service needs to be scaled out or up. Additionally, even if the problem is on the client's end, if they are repeatedly experiencing degraded service, the problem is going to reflect on your service. 

I managed a service that used Apache Traffic Server at one point. Apache Traffic server did not handle _Expect:_ request headers as specified by the HTTP/1.1 spec. 1KB and larger requests from applications built with the cURL library had 1-second delays unless customers turned off the _Expect:_ header explicitly. Compounding the issue, retries due to not receiving a response within the expected 50ms response time led to retry storms. When I saw this pattern in my logs, I wrote up documentation and set up tools that would catch this issue so that I could reach out to customers proactively.

When using a third-party service or external tool, make sure that you're not layering retry logic. Layering retries at different levels or cascading retries can lead to increased latency where failing fast would have been the preferred operation.

Many SDKs include retry configurations. Generally, there will be a maximum number of retries and a delay factor that may be an exponential, incremental, or randomized interval of the amount of time to delay subsequent retries. 

Separate the transient issues in logging so that they are warnings rather than errors. While they should be monitored, paging out on something that self-resolves is a recipe for the increased loss of attention, especially when it pages folks on off-hours. There is nothing so frustrating as losing sleep over something that fixed itself.