FetchApp: UrlFetchApp with Retries
Tags: gas, googleappsscript, opensource
Published: 2024-01-04 22:56:14

Google Apps Script is often used to pull data from various services via HTTP requests.  However, these requests sometimes fail due to network or service issues. The default behavior of `UrlFetchApp` is to throw an exception, which you have to catch. Otherwise, the script execution will be interrupted.

We often need more: send the request again instead of failing. There is no built-in way to do retries in Apps Script.

## Solution

To solve this problem and not copy-and-paste code snippets from project to project, I created `FetchApp` ([GitHub](https://github.com/dataful-tech/fetch-app)) -- an open-source Google Apps Script library. 

Its main features are:

1. **Optional Retries:** Depending on the response code received.
2. **Delay Strategies:** Choose between linear or exponential delay between retries.
3. **Custom Callbacks:** Implement callbacks on failed attempts for tailored actions and logic.
4. **Enhanced Type Hints:** Improved hints for UrlFetchApp's `params` argument.
5. **Automatic Logging:** Logs failed attempts automatically.

I use this library in many projects in production, especially where I have to communicate with unreliable third-party APIs, or there is a chain of related requests that I do not want to repeat if one of them randomly fails.

_**If you find this library useful, please give the [repository](https://github.com/dataful-tech/fetch-app) a star and share the link with others.**_

## Features, Use Cases, Examples

There are multiple ways and use cases to use `FetchApp`.

### Drop-in Replacement for UrlFetchApp

`FetchApp` is designed to work as a drop-in replacement for `UrlFetchApp`. Caveat: `FetchApp` sets `muteHttpExceptions: true` in `params` unless explicitly specified otherwise.

```js
// `url` and `params` are defined elsewhere

// regular UrlFetchApp
const response1 = UrlFetchApp.fetch(url, params);

// FetchApp without configuration is a pass-through to UrlFetchApp
const response2 = FetchApp.fetch(url, params);

// FetchApp with retries and delay enabled
const config = {
  maxRetries: 5,
  successCodes: [200],
  delay: 500,
};
const response3 = FetchApp.fetch(url, params, config);

// If there are no `params`, pass an empty object
const response4 = FetchApp.fetch(url, {}, config);
```

### Configurable Client

If you need to use FetchApp multiple times, you can initiate a client to reuse the configuration:

```js
// FetchApp with retries and delay enabled
const config = {
  maxRetries: 5,
  retryCodes: [500, 502, 503, 504],
  delay: 500,
};

const client = FetchApp.getClient(config);

// All client's fetch calls will use this config
const response1 = client.fetch(url, params);

// Partially modify the config for a specific request
const response2 = client.fetch(url, params, { successCodes: [200] });
```

### Success or Retry Response Codes

`FetchApp` retries requests depending on the response code received in two different modes: 
- `successCodes`: deem responses with these codes successful and return the response. If provided, `retryCodes` are ignored.
- `retryCodes`: requests with these codes are not successful; retry.

Examples:

```js
const response1 = FetchApp.fetch(
  url,
  params,
  {
    successCodes: [200],  // Everything else leads to retries
    maxRetries: 3,
  },
)

const response2 = FetchApp.fetch(
  url,
  params,
  {
    retryCodes: [500, 502, 503],  // Everything else is deemed successful
    maxRetries: 3,
  },
)

const response3 = FetchApp.fetch(
  url,
  params,
  {
    successCodes: [200],  // Takes priority over retryCodes
    retryCodes: [500, 502, 503],  // Ignored
    maxRetries: 3,
  },
)
```

### Delay Between Requests

`FetchApp` supports constant or exponential delay between retries:

```js
const response1 = FetchApp.fetch(
  url,
  params,
  {
    successCodes: [200],  // Everything else leads to retries
    maxRetries: 3,
    delay: 300,  // Constant delay of 300ms after each request
  },
)

const response2 = FetchApp.fetch(
  url,
  params,
  {
    successCodes: [200],  // Everything else is deemed successful
    maxRetries: 3,
    // Exponential delay of 1, 2, 4, 8, etc. seconds
    delay: 1000,
    delayFactor: 2,
  },
)

const response2 = FetchApp.fetch(
  url,
  params,
  {
    successCodes: [200],  // Everything else is deemed successful
    maxRetries: 10,
    // Exponential delay of 1, 2, 4, 8, 10 seconds.
    delay: 1000,
    delayFactor: 2,
    maxDelay: 10000,  // Limit delay to maximum 10 seconds
  },
)
```

### Throw Exceptions via Callbacks

If you need to throw an exception if all attempts fail, you can do it via a `onAllRequestsFailure` callback:

```js
// Throw an exception
const throwException = ({ retries, url }) => {
  throw new Error(`All ${retries + 1} requests to ${url} failed`);
};

const config = {
  successCodes: [200],
  maxRetries: 5,
  delay: 500,
  onAllRequestsFailure: throwException,
};

const response = FetchApp.fetch(url, params, config);
```

### Send Notifications via Callbacks

One of the difficulties with Apps Script is that it functions as a black box unless you add logging and notifications to the script. With the `onRequestFailure` and `onAllRequestsFailure` callbacks, you can send notifications about failed requests.

```js
// Define the `sendNotification` function elsewhere

// Send notification if access is denied, maybe because the credentials expired
const accessDenied = ({ url, response }) => {
  const responseCode = response.getResponseCode();
  if ([401, 403].includes(responseCode)) {
    sendNotification(`Received ${responseCode} when accessing ${url}`);
  }
};

// Send a notification if all attempts failed
const allAttemptsFailed = ({ retries, url }) => {
  throw new Error(`All ${retries + 1} requests to ${url} failed`);
};

const config = {
  successCodes: [200],
  maxRetries: 5,
  delay: 500,
  onRequestFailure: accessDenied,
  onAllRequestsFailure: allAttemptsFailed,
};

const response = FetchApp.fetch(url, params, config);
```

### Autocomplete and Type Hints

`FetchApp` comes with JSDoc declarations that enable type hints and autocomplete in Google Apps Script IDE, both for `UrlFetchApp`'s `params` and for `FetchApp`'s `config`:

![FetchApp: Autocomplete params and config](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ftijuieucxhwpz8guxl.png)

Unfortunately, due to the IDE limitations, rich autocomplete doesn't work when you attach `FetchApp` as a library (as opposed to copying the code).  Nevertheless, you still have the description of possible options in the type hints:

![FetchApp: Type hints for arguments](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kjbff81g8qym4xdmvqqb.png)



---

Contributions are welcome. Feel free to submit pull requests or issues on [GitHub](https://github.com/dataful-tech/fetch-app).