Hertz client's rich retry functionality
Tags: go, tutorial, webdev, beginners
Published: 2022-11-09 04:08:32

# Hertz client's rich retry functionality

[Hertz](https://github.com/cloudwego/hertz) is an ultra-large-scale enterprise-level microservice HTTP framework, featuring high ease of use, easy expansion, and low latency etc.

Hertz uses the self-developed high-performance network library [Netpoll](https://github.com/cloudwego/hertz/tree/develop/pkg/network/netpoll) by default. In some special scenarios, Hertz has certain advantages in QPS and latency compared to go net.


[Hertz](https://github.com/cloudwego/hertz) client provides users with customized retry logic,Let's take a look at how to use Client Retry(This section mainly configures the times of retry and the delay policy )

## Retry times and delay policy configuration
First create a Client, and use the configuration item `WithRetryConfig()` to configure the Retry related logic

```go
package main

import (
	"github.com/cloudwego/hertz/pkg/app/client"
	"github.com/cloudwego/hertz/pkg/app/client/retry"
)

func main() {
	cli, err := client.NewClient(
		client.WithRetryConfig(
			retry.WithXxx(), // the method of setting retry config
		),
	)
}
```

| Configuration Name  | Type                                                         | Description                                                  |
| :------------------ | :----------------------------------------------------------- | :----------------------------------------------------------- |
| WithMaxAttemptTimes | uint                                                         | Set the maximum attempts times. Default：1 times (That is, only request once without retry) |
| WithInitDelay       | time.Duration                                                | Set initial delay time. Default: 1ms                         |
| WithMaxDelay        | time.Duration                                                | Set the maximum delay time. Default: 100ms                   |
| WithMaxJitter       | time.Duration                                                | Set the maximum jitter time, which needs to be used in conjunction with `RandomDelayPolicy`, and will generate a random time not exceeding the maximum jitter time. Default: 20ms |
| WithDelayPolicy     | type DelayPolicyFunc func(attempts uint, err error, retryConfig *Config) time.Duration | Set the delay policy, you can use any combination of the following four policies, `FixedDelayPolicy`,  `BackOffDelayPolicy`, `RandomDelayPolicy`, `DefaultDelayPolicy`  ( [See the next section **Delay Policy** for details](#delay-policy) ) . Default: DefaultDelayPolicy  (That is, the retry delay is 0) |

### Delay Policy

`retry.WithDelayPolicy()`  usage

```go
cli, err := client.NewClient(
		client.WithRetryConfig(
			...
			retry.WithDelayPolicy(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy, retry.RandomDelayPolicy)),
    		...
		),
	)
```



| Function Name      | Description                                                  |
| :----------------- | :----------------------------------------------------------- |
| CombineDelay       | It is used to combine any of the following four policies and sum the values calculated by the selected policy. When you only need one of the following four policies, you can choose to use `CombineDelay` or directly pass any policy into `WithDelayPolicy` as a parameter |
| FixedDelayPolicy   | Set the fixed delay time and use the value set by `WithInitDelay` to generate an equivalent delay time |
| BackOffDelayPolicy | Set the exponential delay time. Use the value set by `WithInitDelay`. The exponential delay time is generated according to the number of retries currently |
| RandomDelayPolicy  | Set the random delay time. Use the value set by `WithMaxJitte`r to generate a random delay time that does not exceed this value |
| DefaultDelayPolicy | Set the default delay time (That is, 0) . Generally, it is used alone and has no effect when combined with other policies |

### Complete example

```Go
package main

import (
	"github.com/cloudwego/hertz/pkg/app/client"
	"github.com/cloudwego/hertz/pkg/app/client/retry"
)
func main() {

	cli, err := client.NewClient(
		client.WithRetryConfig(
			retry.WithMaxAttemptTimes(3), // Maximum number of attempts, including initial calls
			retry.WithInitDelay(1*time.Millisecond), // Initial delay
			retry.WithMaxDelay(6*time.Millisecond), // Maximum delay.No matter how many retries and what the policy is, the delay will not exceed this delay
			retry.WithMaxJitter(2*time.Millisecond), // Maximum jitter delay, which will have effect only when combined with RandomDelayPolicy
			/*
			   To configure the delay policy, you can select any combination of the following four, and the final result is the sum of each delay policy.
			   FixedDelayPolicy uses the value set by retry.WithInitDelay,
			   BackOffDelayPolicy increases exponentially with the number of retries based on the value set by retry.WithInitDelay,
			   RandomDelayPolicy generates a random value of [0, 2*time.Millisecond). 2*time.Millisecond is the value set by retry.WithMaxJitter,
			   DefaultDelayPolicy generates a value of 0. If it is used alone, retry again immediately,
			   retry.CombineDelay() sums the values generated by the set delay policy, and the final result is the delay time of the current retry,
			   The first call failed -> Retry delay：1 + 1<<1 + rand[0,2)ms -> The second call failed -> Retry delay：min(1 + 1<<2 + rand[0,2) , 6)ms -> The third call succeeded/failed
			*/
			retry.WithDelayPolicy(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy, retry.RandomDelayPolicy)),
		),
	)
}
```



## Retry condition configuration

If you want to customize the conditions under which retries occur, you can use ` client SetRetryIfFunc() ` configuration. The parameter of this function is a function, and the signature is:

```go
func(req *protocol.Request, resp *protocol.Response, err error) bool
```

Relevant parameters include the `req`, `resp` and `err` fields in the Hertz request. You can use these parameters to determine whether the request should be retried. In the following example, when the status code returned by the request is not 200 or `err!=nil` during the call, we return true, that is, we retry.

```Go
cli.SetRetryIfFunc(func(req *protocol.Request, resp *protocol.Response, err error) bool {
   return resp.StatusCode() != 200 || err != nil
})
```

Note that if you do not set `client SetRetryIfFunc()` . We will judge according to Hertz's default retry conditions, that is, whether the request meets the following `DefaultRetryIf()` function and whether the call is idempotent. ( Idempotent call: when `canIdempotentRetry` is [true](#table1) in [pkg/protocol/http1/client.go::Do()](https://github.com/cloudwego/hertz/blob/develop/pkg/protocol/http1/client.go#L328 ) and [pkg/protocol/http1/client.go::doNonNilReqResp()](https://github.com/cloudwego/hertz/blob/develop/pkg/protocol/http1/client.go#L411) )

```Go
// DefaultRetryIf Default retry condition, mainly used for idempotent requests.
// If this cannot be satisfied, you can implement your own retry condition.
func DefaultRetryIf(req *protocol.Request, resp *protocol.Response, err error) bool {
   // cannot retry if the request body is not rewindable
   if req.IsBodyStream() {
      return false
   }

   if isIdempotent(req, resp, err) {
      return true
   }
   // Retry non-idempotent requests if the server closes
   // the connection before sending the response.
   //
   // This case is possible if the server closes the idle
   // keep-alive connection on timeout.
   //
   // Apache and nginx usually do this.
   if err == io.EOF {
      return true
   }

   return false
}
func isIdempotent(req *protocol.Request, resp *protocol.Response, err error) bool {
   return req.Header.IsGet() ||
      req.Header.IsHead() ||
      req.Header.IsPut() ||
      req.Header.IsDelete() ||
      req.Header.IsOptions() ||
      req.Header.IsTrace()
}
```



When `canIdempotentRetry` in Hertz source code [doNonNilReqResp()](https://github.com/cloudwego/hertz/blob/develop/pkg/protocol/http1/client.go#L411)  is true.

| doNonNilReqResp() return true                                |
| ------------------------------------------------------------ |
| err = conn.SetWriteDeadline(currentTime.Add(c.WriteTimeout)) |
| err = reqI.Write(req, zw)                                    |
| err = reqI.ProxyWrite(req, zw)                               |
| err = zw.Flush()                                             |
| err = conn.SetReadTimeout(c.ReadTimeout)                     |
| ( err = respI.ReadHeaderAndLimitBody() \|\| err = respI.ReadBodyStream() ) && (err != errs.ErrBodyTooLarge) |


## References

- https://github.com/cloudwego/hertz
- https://github.com/cloudwego/hertz/pull/239
- https://www.cloudwego.io/docs/hertz/tutorials/basic-feature/retry/