String.prototype.replace asynchronously?
Tags: javascript, vuepress, vue
Published: 2018-04-28 10:07:48

---
title: String.prototype.replace asynchronously?
published: true
description: This article describes how we could write asyncStringReplace.
tags: javascript, vuepress, vue
---
*Original post: https://www.ycmjason.com/blog/2018/04/28.html*
> this article assumes basic knowledge of RegExp.

## Background

I was working with [vuepress](https://github.com/vuejs/vuepress) last week and I realise I want to be able to break my very long markdown into partials. So I raised this [issue](https://github.com/vuejs/vuepress/issues/222). And the legend, Evan You, suggested to use `<!-- include ./sectionA.md -->`. Then I picked up his advise and started digging into the code of vuepress. 


### [String.prototype.replace](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace)

Before I explain how I solved the problem, I would like to make sure we are all on the same page. My solution is based on `String.prototype.replace` function which I will very briefly explain how this function works. This function takes in two arguments:

1. What to replace (RegExp | String)
2. What to replace with (String | Function)

#### String.prototype.replace(*String*, *String*)

```js
const str = 'I am very happy, happy, happy.';
str.replace('happy', 'sad'); // I am very sad, happy, happy.
```

The above example shows how we could replace a word in a string. Notice that only the first occurrence of `happy` is replaced by `sad`. This behaviour is similar to when you pass in a RegExp without global flag.

#### String.prototype.replace(*String*, *Function*)

```js
const str = 'I am very happy, happy, happy.';
str.replace('happy', word => 'not ' + word);
// ^ I am very not happy, happy, happy.
```

You could retrieve the matched word by passing in a replacer function. The value returned from the replacer function would be used to replace the `word`.

This use case is rare and probably not very useful as you already know the targeting word. You could simply do `str.replace('happy', 'not happy')` to have the same effect.

#### String.prototype.replace(*RegExp*, *String*)

```js
const str = 'I am very happyyyyy, happy, happy.';
str.replace(/happ(y+)/, 'sleep$1'); // I am very sleepyyyyy, happy, happy.
str.replace(/happ(y+)/g, 'sleep$1'); // I am very sleepyyyyy, sleepy, sleepy.
```

Should be fairly straight forward. Two things to note:

1. `/happ(y+)/` matches "happy" and all the "y"s that come after it.
2. `$1` will be replaced by whatever is matched in the groups `()` of the RegExp. You can have more than one groups and simply use `$2`, `$3`, `$4` as their placeholders.

#### String.prototype.replace(*RegExp*, *Function*)

```js
const str = 'I am very happyyyyy, happy, happyy.';

str.replace(/happ(y+)/, (match, ys) => {
	// match: 'happyyyyy'; ys: 'yyyyy'
	return 'sleep' + ys;
}); // I am very sleepyyyyy, happy, happyy.

str.replace(/happ(y+)/g, (match, ys) => {
	// This function is called 3 times:
	//     1. match: 'happyyyyy'; ys: 'yyyyy'
	// 	   2. match: 'happy'; ys: 'y'
	//     3. match: 'happyy'; ys: 'yy'
	return 'sleep' + ys;
}); // I am very sleepyyyyy, sleepy, sleepyy.
```

The comments should be quite self-explanatory.

### The synchronous way

Back to the problem we have, to replace `<!-- include ./sectionA.md -->` with the content of `./sectionA.md`.

Any decent regex-er could come up with a regex to match that placeholder, and we came up with something like:

```js
const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g
```
Note: `\s` matches any space/tab etc. See [here](https://www.w3schools.com/jsref/jsref_regexp_whitespace.asp) for more information.

This RegExp will match the placeholder as a whole and group the filename after the `include`.

So I basically use the `String.prototype.replace` to do the job:
```js
const { readFileSync, existsSync } = require('fs');

const replaceIncludePlaceholdersWithFileContents = str => {
	const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g;
    return str.replace(placeholderRe, (placeholder, filename) => {
	    if (!existsSync(filename)) return placeholder;
        return readFileSync(filename, 'utf8');
    });
};
```

This works, we just need to handle one more case, i.e. when the partial being included also contain `<!-- include file.md -->`. Obviously this become a recursive problem. The way to deal with this is simply doing the *Leap of faith*. 

Simply by applying `replaceIncludePlaceholdersWithFileContents` recursively on the content of each file included by the current file would do the job!

So we have something like:

```js
const { readFileSync, existsSync } = require('fs');

const replaceIncludePlaceholdersWithFileContents = str => {
	const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g;
    return str.replace(placeholderRe, (placeholder, filename) => {
	    if (!existsSync(filename)) return placeholder;
        return replaceIncludePlaceholdersWithFileContents(
	        readFileSync(filename, 'utf8')
	    );
    });
};
```

This time our base case is when the included file do not contain the placeholder, then the function should terminate as the replacer function would not be called.

### The asynchronous way

So I submitted the pull request, and some feedback has been given to me suggesting the use of `fs.readFile`, the async version of `fs.readFileSync`. 

Immediately I realise, if I have a function called `asyncStringReplace(str, search, replacer)` which does what `String.prototype.replace` does but allow `replacer` to return a `Promise`, then I could just change my code to the following and it would work.

```js
const { readFile, existsSync } = require('fs-extra');

const replaceIncludePlaceholdersWithFileContents = async str => {
	const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g;
    return await asyncStringReplace(str, placeholderRe, async (placeholder, filename) => {
	    if (!existsSync(filename)) return placeholder;
        return await replaceIncludePlaceholdersWithFileContents(
	        await readFile(filename, 'utf8')
	    );
    });
};
```

Spent so much time on thinking about the replacement of the placeholder, I would love to retain the already existing logic as much as possible.

So now what I need to write is just the `asyncStringReplace` method.

## asyncStringReplace

The `asyncStringReplace` method should take in three arguments:

1. `str` - the original string
2. `regex` - the RegExp that represents the substring of `str` to be replaced
3. `aReplacer` - an asynchronous function that takes in each match, should return `Promise`.

I basically copied from [mdn](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/exec) the "while-loop" that loops through the matches using `RegExp.prototype.exec`. By using `RegExp.prototype.exec` we could track the `RegExp.lastIndex` and `match.index` of each match, which I couldn't think of a way to achieve this with `String.prototype.match`. 

```js
const asyncStringReplace = async (str, regex, aReplacer) => {
    const substrs = [];
    let match;
    let i = 0;
    while ((match = regex.exec(str)) !== null) {
        // put non matching string
        substrs.push(str.slice(i, match.index));
        // call the async replacer function with the matched array spreaded
        substrs.push(aReplacer(...match));
        i = regex.lastIndex;
    }
    // put the rest of str
    substrs.push(str.slice(i));
    // wait for aReplacer calls to finish and join them back into string
    return (await Promise.all(substrs)).join('');
};
```

My approach basically split the given `str` with the given `regex` into substrings and put them into `substrs`. 

`substrs` therefore contains:
```js
[
	/* first loop in while */
	NON_MATCHING_STRING,
	aReplacer(MATCHING_STRING),

	/* second loop in while */  
	NON_MATCHING_STRING,
	aReplacer(MATCHING_STRING),

	/* ... */,

	/* n-th loop in while */  
	NON_MATCHING_STRING,
	aReplacer(MATCHING_STRING),

	/* substrs.push(restStr) */
	REST_NON_MATCHING_STRING
]
```

E.g.
If we call the following
```js
asyncStringReplace('i am happyy, happy === happyyy very!', /happ(y+)/g, someAsyncReplacer);
```

The corresponding `substrs` would be:
```js
[
    /* first loop in while */
	'i am ',
	someAsyncReplacer('happyy', 'yy'),
	
    /* second loop in while */
	', ',
	someAsyncReplacer('happy', 'y'),
	
    /* third loop in while */
	' === ',
	someAsyncReplacer('happyyy', 'yyy'),

	/* substrs.push(restStr) */
	' very!'
]
```



Notice since `aReplacer` is an asynchronous function, `aReplacer(MATCHING_STRING)` would therefore be a `Promise`. `Promise.all` could be used here to construct a `Promise` which resolves when all promises are resolved in this list. 

The last line
```js
	return (await Promise.all(substrs)).join('')
```

`await Promise.all(substrs)` would yield to an array of string and `.join('')` would join all of them back together.

An example of how this could be applied:
```js
const { readFile, existsSync } = require('fs-extra');

const replaceIncludePlaceholdersWithFileContents = async str => {
	const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g;
    return await asyncStringReplace(str, placeholderRe, async (placeholder, filename) => {
	    if (!existsSync(filename)) return placeholder;
        return await replaceIncludePlaceholdersWithFileContents(
	        await readFile(filename, 'utf8')
	    );
    });
};
```