Promise wait for multiple functions and retry some failed function
Tags: javascript, typescript, node
Published: 2020-09-24 03:20:54

So, I found myself in a situation where I need to create an asynchronous function that fetches data from multiple third party's APIs, daily. My concern is we can't control the third party's up and downtime. And its (function) being automatically executed is making me more confused. (well, I just want a happy and quiet weekend (: )

Multiple asynchronous requests from 1 URL, could overload its server. When it happens, the server will be restarted and offline for a few seconds (maybe minutes or more), and all the requests will result failed.

In some cases, you need to crawl many websites, because not all websites offer an API. Crawling many big websites, simultaneously, at one time, could take quite a long time. The more time it needs to connect to the website, the more chance for request timeout to happen. Especially, if your application has limited bandwidth. This could lead to a request timeout.

It's more complicated if your application needs to do all the above automatically, without the dev manually restarting request or so.

Here's my solution, we only need 3 functions (4th one is optional):
1. Main function, where we call Promise All function with an array of URLs as a parameter.
2. Promise All function, where we put `Promise.all()` and what we want after all promise has been executed.
3. Axios function, where we put `axios.get()` function, to get request from third party, and return either JSON if the request is success or URL (string) if the request is failed.
4. Delay (request again) function, where we put `setTimeout()` to call Promise All function after *n* seconds delay. (this function is optional if you want to delay the next request).

## Axios Function
For example, we will use this simple [api](http://echo.jsontest.com/hello/world/test/one). The expected result as JSON:
```json
{
  "test": "one",
  "hello": "world"
}
```
Assume that this API results in an error (internal server), instead of throw, we will return the URL.

```typescript
const axiosFunc: any = (async (url: string) => {
  return await axios.get(url)
    .then(response => response.data)
    .catch(e => url);
});
```

## Promise All Function
`Promise.all()` accepts an Array of Promises. Instead of writing like this:
```typescript
Promise.all([axiosFunc(url1),axiosFunc(url2),...]);
```
We could write it like this:
```typescript
const urls = [url1,url2,...];
Promise.all(urls.map(url => this.axiosFunc(url));
```
The expected result is an Array of results from Axios Function, either json object (result from api) or string (url). With simple `if..else` and `.some()`, we will check whether the result have some string or not. If there are some string in it, we will split the result by its type with `.filter()`.
```typescript
if(data.some(val => typeof val == 'string')) {
  /** filter failed one by checking its type and execute again */
  const failedReq = data.filter(val => typeof val == 'string');
  this.requestAgain(failedReq);

  /** filter success one by checking its type and save its result to db */
  const successReq = data.filter(val => typeof val !== 'string');
  this.saveResult(successReq);
} else {
  this.saveResult(data);
}
```
The string one (failed one), goes to `this.requestAgain()`. We will get to this function later in this post. And the not string one (success one), goes to `this.saveResult()`. In this function, data from successful results will be saved in database or cache or other data saving methods.

Below is the full code of this function:
```typescript
const promiseAllFunc: void = ((urls: string[]) => {
  return await Promise.all(urls.map(url => this.axiosFunc(url))
    .then(data => {
      if(data.some(val => typeof val == 'string')) {
        /** filter failed one by checking its type and execute again */
        const failedReq = data.filter(val => typeof val == 'string');
        this.requestAgain(failedReq);

        /** filter success one by checking its type and save its result to db */
        const successReq = data.filter(val => typeof val !== 'string');
        this.saveResult(successReq);
      } else {
        this.saveResult(data);
      }
    })
});
```
## Main Function
Call Promise All function from above with array of urls (third party's api) as parameter.
```typescript
const main: any = (() => {
  let dummyUrls = [
    'http://echo.jsontest.com/hello/world/test/one',
    'http://echo.jsontest.com/hello/world/test/two',
    'http://echo.jsontest.com/hello/world/test/three'
  ];
  return this.promiseAllFunc(dummyUrls);
});
```

## Delay Function
This function is optional. If you want to delay the next request, you could write additional function with `setTimeout()`. Below is example with 10 seconds delay for the next request.
```typescript
const requestAgain: any = ((urls: string[]) => {
  setTimeout(() => {
    this.promiseAllFunc(urls);
  }, 10000);
});
```

This method solves all of my problems. It's automatically executed (with Cronjob). Retrying failed requests (you could modify Delay Function by using Cronjob too). 

Well, that's it, Thank you :)
Feel free to add some feedback in comment section. This is my first post here. Maybe some grammar mistakes here and there (or more :D ), but I think it's still understandable.