Combining async requests in the background
Tags: javascript, promise, async
Published: 2023-05-09 23:55:42

When I was in Shopee, there was an interesting case about a React component triggering some API requests and a long list of this component often appears in a page. As a result, client-side send many requests in a short time and not only slowed down the interface but also occupied server resource significantly.

To improve this, my smart teammate created a "batching" solution that I loved so much and used the idea in my interview later. 

How did we achieve this?

Let me recap the situation with some detailed setups first:

1. There is a `<ListItem/>` component sends a API `GET /item` under certain event (e.g, mouse enter) with compulsory parameters like item ID, like this:
    
```jsx
import React, { useCallback, useState } from 'react';

// Proper error handling is ignored here...
const fetchItem = id => fetch(`/item?id=${id}`).then(res => res.json());

export function ListItem({ id }) {
  const [item, setItem] = useState({});
  const onMouseEnter = useCallback(() => {
    fetchItem(id).then(setItem);
  }, [id]);
  return <div onMouseEnter={onMouseEnter} >{item.name}</div>;
}
```
    
2. The `<ListItem/>` component is used to display a long list, like this:
    
```jsx
import React from 'react';
import { ListItem } from './ListItem';
export function List({ ids }) {
  return (
    <div>
      {ids.map(id => 
        <ListItem id={id} />
      )}
    </div>
  );
}
```
    

Now you may question that isn't the solution just sending a list request like `GET /items?ids=1,2,3` in `<List/>` component instead? Well, it is not the case here as **the request is only sent when user triggers a specific event** on the item. But you are getting closer to the point - instead of `GET /item?id=xxx` repeatedly with different sets of parameters, it is better to find a way to send `GET /items?ids=1,2,3` instead.

As these requests are sent at different times, here is an important decision to make: **under what condition shall we combine the requests**?

Here are 2 ideas:

1. Combine the requests triggered within a certain period;
2. Combine the requests when the request queue reaches a certain length.

Timing is the first factor to consider as if the interface is waiting for the response data from the request, we shall not keep user waiting for too long. 

Queuing request length however, depends more on the server side. How many items can they process at one time (considering each item may require extra information to process)? Is there any request URL (for GET requests) or body size limit (load balancer like Nginx has a default setting, and sometimes BE framework sets limit as well)?

## Combining by time

Speaking of time and requests, there are 2 well-known existing strategies: [throttle and debounce](https://www.syncfusion.com/blogs/post/javascript-debounce-vs-throttle.aspx).

Debounce delays the function invocation by a defined period of time to avoid unnecessary invocations. It works well with button clicks and key events.

Throttle invokes the callback function at regular intervals as long as the event trigger is active. It suits more in the continuous events.

Here I will demonstrate with a strategy inspired from throttle.

First of all,  let's add a `fetchItems()` helper:

```javascript
const fetchItems = ids => fetch(`/item?ids=${ids.join(',')}`).then(res => res.json());
```

Then tweak the original `fetchItem()` to store the single set of parameters into a queue:

```javascript
const queue = [];
const fetchItem = id => {
	queque.push(id);
	// TODO: flush the queue
};
```

And handover the parameters to the `fetchItems()` helper with a timer delaying `200ms`.

```javascript
const queue = [];
const fetchItem = id => {
  queue.push(id);
	if (queue.length === 1) {
	  setTimeout(() => {
	    fetchItems(quque);
	    queue.length = 0;
	  }, 200);
	}
}
```

You may notice that I only created a timer when the parameters start to queue up, and clear the queue once the batch request `fetchItems()` is sent. 

Now it should work for the requests not looking forward to responses (e.g. recording user behaviours). If we would like to process with the responses, we may create a `Promise` and cache it for later like this:

```javascript
const queue = [];
let task;
const fetchItem = id => {
  queue.push(id);
	if (queue.length === 1) {
	  task = new Promise((resolve) => {
			setTimeout(() => {
		    fetchItems(quque);
		    queue.length = 0;
		  }, 200);
		});
	}
	return task;
}
```

The promise is wrapping the timer, and each single `fetchItem()` "request" is returning this promise. As a result, you may get all the items in the same batch once this promise resolves:

```javascript
fetchItem(1).then(items => items[0]);
fetchItem(2).then(items => items[1]);
fetchItem(3).then(items => items[2]);
```

Furthermore, we may abstract this logic for future usage:

```javascript
function batchRequests(request, paramMerger, period = 200) {
  const queue = [];
  let task;

  return (params) => {
    queue.push(params);
    if (queue.length === 1) {
      task = new Promise((resolve) => {
        setTimeout(() => {
          const params = paramMerger(queue.slice());
          queue.length = 0;
          resolve(request(params));
        }, period);
      });
    }
    return task;
  };
}
const fetchItem = id => batchRequests(
  fetchItems,
  ids => ids
);
```

Here is the CodeSandbox to try: 

{% embed https://codesandbox.io/embed/combining-async-requests-by-time-8nkfhn?autoresize=1&expanddevtools=1&fontsize=14&hidenavigation=1&theme=dark %}

## Combining by count

Combining requests by count alone is quite straightforward. Let's reuse the `fetchItems()` created above:

```javascript
const MAX_QUEUE = 5;
const queue = [];
const fetchItem = id => {
  queue.push(id);
	if (queue.length === MAX_QUEUE) {
    fetchItems(quque);
    queue.length = 0;
	}
}
```

However, considering count only is a bad idea as `fetchItem()` may never send the request till it collects enough parameters. Therefore, we shall consider with time together, like this:

```javascript
const MAX_QUEUE = 5;
const queue = [];
let timer;
const flushQueue = () => {
	clearTimeout(timer);
	fetchItems(quque);
  queue.length = 0;
};
const fetchItem = id => {
  queue.push(id);
	if (queue.length === 1) {
	  timer = setTimeout(flushQueue, 200);
	} else if (queue.length === MAX_QUEUE) {
    flushQueue();
	}
}
```

And oops, it gets tricky if we expect the response data from the function:

```javascript
const MAX_QUEUE = 5;
const queue = [];
let timer;
let flushQueue = () => null;
const fetchItem = id => {
  queue.push(id);
  if (queue.length === 1) {
    const timer = setTimeout(() => flushQueue(), period);
    task = new Promise((resolve) => {
      flushQueue = () => {
        const params = paramMerger(queue.slice());
        queue.length = 0;
        clearTimeout(timer);
        resolve(request(params));
      };
    });
  } else if (queue.length === MAX_QUEUE) {
    flushQueue();
  }
  return task;
};
```

This is because it is trying to resolve the promise for both 2 conditions, but we only create the timer and promise once for each batch. It looks quite messy and risky. We will discuss alternatives later.

Similar as above, we may extract the logic for future usage:

```javascript
function batchRequests(request, paramMerger, period = 200, maxQueue = 5) {
  const queue = [];
  let task;
  let flush = () => null;
  if (maxQueue < 2) {
    throw new Error("Max queuable requests must be more than 1!");
  }

  return (params) => {
    queue.push(params);
    if (queue.length === 1) {
      const timer = setTimeout(() => flush(), period);
      task = new Promise((resolve) => {
        flush = () => {
          const params = paramMerger(queue.slice());
          queue.length = 0;
					clearTimeout(timer);
          resolve(request(params));
        };
      });
    } else if (queue.length === maxQueue) {
      flush();
    }
    return task;
  };
}
```

Here is the CodeSandbox to try: 

{% embed https://codesandbox.io/embed/combining-async-requests-by-count-we7vy9?autoresize=1&expanddevtools=1&fontsize=14&hidenavigation=1&theme=dark %}

## The alternative

**When a logic gets tricky, it is highly likely we are doing things wrongly or simple alternative exists.** 

Here for getting the `fetchItems()` response I believe state manager such as Redux is a better alternative. With the state manager, we just need to update the items state in our modified `fetchItem()`. Then the "reactive" framework will notify the change and render the latest data, and resolving promise is no longer expected from the `fetchItem()`.

## The risk

Everything looks pretty nice before user close the tab without sending the request and your important data (e.g. data relates to advertisement charges) get lost! It is necessary to consider moving this part of the logic into `ServiceWorker` to avoid losses. 

## Words at last

To wrap up, handling async requests is an essential Front-end skill nowadays, and sometimes it is more than just applying `aysnc`/`await` and `Promise` APIs.