From Callback Hell to Callback Heaven
Tags: javascript, typescript, node, async
Published: 2020-12-04 17:09:16

Remember the time when a lot of JavaScript code looked like this?

```js
router.put('/some-url', (req, res) => {
  fs.readFile(filePath(req), (err, file) => {
    if (err) res.status(500).send()
    else {
      parse(file, (err, parsed) => {
        if (err) res.status(500).send()
        else db.insert(parsed, err => {
          if (err) res.status(500).send()
          else res.status(200).send()
        })
      })
    }
  })
})
```

For those fortunate enough to not remember those days, this was called a _callback hell_, for obvious reasons. Fortunately, we have moved on, and these days equivalent code most likely looks something like this:
```js
router.put('/some-url', async (req, res) => {
  try {
    const file = await fs.readFile(filePath(req));
    const value = await parse(file);
    await db.insert(value);
    response.status(200).send();
  } catch {
    response.status(500).send();
  }
})
```

Of course now that we have `async`/`await` and `Promise`s, it is easy to blame the _callback hell_ era on lack of syntactic features of JS at the time and move on. But I do think there is value in reflecting back, analyzing the core issue, how it was solved, and what's to learn from it all.

---

# The Issue

Lets look back at the overall structure of the _hellish_ example above:

```js
doX(args, (err, res) => {
  if (err) { ... }
  else {
    doY(args, (err, res) => {
      if (err) { ... }
      ...
    })
  }
})
```
A glaring issue here is that most of whats on the screen is dedicated to _not-really-important_ stuff:
```js
doX(args /*, (err, res) => {
  if (err) { ... }
  else {*/
    doY(args /*, (err, res) => {
      if (err) { ... } */
      ...
    /*}*/)
  /*}*/
/*}*/)
```
For comparison, this would be the equivalent structure for the modern not-hellish version:
```js
/* try { */
  /*await*/ doX(args)
  /*await*/ doY(args)
  ...
/*} catch { }*/
```
The commented bits in both versions are indicative of the same things: `doX()` and `doY()` are async functions, and also there might be some errors. In the _hellish_ version though, you need to spend much more space for these side notes, which results in a far less readable code.


👉 Note that we could trim the boilerplate and restructure the code into a more readable format without additional syntax too. Historically speaking, that _IS_ what happened, in the form of `Promise` libraries (which then got standardized and a bit of more love with some syntactic support):

```js
doX(args)
.then(() => doY(args))
.then(() => ...)
.catch(() => { ... })
```
```js
doX(args)
/*.then(() =>*/doY(args)/*)*/
/*.then(() =>*/.../*)*/
/*.catch(() => { ... })*/
```
An important difference between this code and the _hellish_ code is that in the _hellish_ code, important stuff and boilerplate stuff are extremely intertwined, while with promise libraries, they are neatly separated, making the code easier to read even in cases where amount of boilerplate is almost the same:

```js
// without promises:
doX(args/*, (err, res) => { ... }*/)
``` 
```js
// with promises:
doX(args)/*.then(() => { ... })*/
```
```js
// with async/await:
/*await*/ doX(args)
```
Promises also provide other important features that help with async programming ergonomics, most importantly:
- Promises are automatically _flattened_ when chained.
- Promises are _shared_.

However, I think these properties, although beneficial, aren't as important as the aforementioned separation. To illustrate that, lets create an experimental _promise_ library that just does the separation and nothing else, and see how it fares.

---

# The Experiment

So initially, we started with functions looking like this:

```ts
doX(args, (err, res) => {...})
```
The callback here is the main boilerplate (and namesake of our hell), so easiest separation is to take it out of argument list of `doX()`, and put it in a deferred function instead:
```js
doX(args)((err, res) => {...})
```

☝️ This is basically a change in how `doX` is implemented, from this:
```js
function doX(args, callback) {
  // do stuff
  // maybe do more
  callback(undefined, 42)
}
```
To this:
```js
function doX(args) {
  // do stuff
  return callback => {
    // maybe do more
    callback(undefined, 42)
  }
}
```
In other words, we just changed the convention from:

> accept a callback as last argument

to:

> return a function that accepts a callback as argument

Our _separation convention_ seems to not have helped much on its own, as we still have the same amount of boilerplate. However, it did open the door for simplistic utilities that help us take away the boilerplate. To see that, let me first introduce the [`pipe()`](https://stackblitz.com/edit/callback-heaven?file=lib%2Fpipe.ts) utility:

```js
function pipe(...cbs) {
  let res = cbs[0];
  for (let i = 1; i < cbs.length; i++) res = cbs[i](res);
  return res;
}
```

Simply put, this:
```js
pipe(a, b, c, d)
```
equals to this:
```js
let x = a
x = b(x)
x = c(x)
x = d(x)
```

In the not so distant future, `pipe()` [might even get incorporated into JavaScript itself](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Pipeline_operator), which would look like this:

```js
a |> b |> c |> d
```

Anyways, the `pipe()` operator allows us to neatly _transform_ the function returned by the (new convention) `doX()` (which remember, is a function accepting a standard callback), without having to write callbacks manually. For example, I could create a [`then()`](https://stackblitz.com/edit/callback-heaven?file=lib%2Foperators%2Fthen.ts) utility as follows:

```js
export function then(f) {
  return src => {
    src((err, res) => {
      if (!err) f(res)
    })

    return src
  }
}
```

and with these utilities, my async code would transform from this:

```js
doX(args)((err, res) => { ... })
```

to this:

```js
pipe(
  doX(args),
  then(() => { ... })
)
```

or better yet (with pipeline operator incorporated):

```js
doX(args) |> then(() => { ... })
```

which looks a lot like a standard promise library:

```js
doX(args).then(() => { ... })
```

I could also create a simple [`catch()`](https://stackblitz.com/edit/callback-heaven?file=lib%2Foperators%2Fcatch.ts) utility:

```js
function catch(f) {
  return src => {
    src((err) => {
      if (err) f(err)
    })

    return src
  }
}
```

Which would give me async codes like this:
```js
doX(args)
|> then(() => doY(args))
|> then(() => ...)
|> catch(() => { ... })
```
```js
doX(args)
/*|> then(() =>*/ doY(args) /*)*/
/*|> then(() =>*/ ... /*)*/
/*|> catch(() => { ... })*/
```

Which is as succinct as promise libraries, with almost no effort. Better yet, this method gives us extensibility as well, as we are not bound to a set `Promise` object and can create / use a much wider range of utility functions:

```js
function map(f) {
  return src => cb => src((err, res) => {
    if (err) cb(err, undefined)
    else cb(undefined, f(res))
  })
}
```
```js
function delay(n) {
  return src => cb => src((err, res) => {
    if (err) cb(err, undefined)
    else setTimeout(() => cb(undefined, res), n)
  })
}
```

and start getting a bit wild:

```js
doX(args)
|> then(() => doY(args))
|> map(yRes => yRes * 2)
|> delay(200)
|> then(console.log)
```

---

# Real-life Examples

Ok so it seems like a simple change of convention allowed us to create utilities and libraries that provide the same convenience provided by promise libraries (and almost similar to `async`/`await` syntax). To get a better perspective, let's look at real-life examples. For this purpose (and mostly out of curiosity), I have created [an online playground with an implementation of our experimental lib](https://stackblitz.com/edit/callback-heaven?file=index.ts).

First off, let's take a look at our original example, which looked like this in its most _hellish_ version:

```js
router.put('/some-url', (req, res) => {
  fs.readFile(filePath(req), (err, file) => {
    if (err) res.status(500).send()
    else {
      parse(file, (err, parsed) => {
        if (err) res.status(500).send()
        else db.insert(parsed, err => {
          if (err) res.status(500).send()
          else res.status(200).send()
        })
      })
    }
  })
})
```
This is how the modern JavaScript version looks like:
```js
router.put('/some-url', async (req, res) => {
  try {
    const file = await fs.readFile(filePath(req));
    const value = await parse(file);
    await db.insert(value);
    response.status(200).send();
  } catch {
    response.status(500).send();
  }
})
```
And this is how our _new callback convention_ code looks like:
```js
router.put('/some-url', (req, res) => {
  fs.readFile(filePath(req))
  |> map(parse)
  |> flatten
  |> map(db.insert)
  |> flatten
  |> then(() => res.status(200).send())
  |> catch(() => res.status(500).send())
})
```
The convention gets us pretty close to the convenience of `async`/`await`. There is a little nuance though: see the `flatten` utility used twice in the middle? That is because unlike promises, our callbacks are not flattened while chaining. We assumed that `parse()` is also async, i.e. it also returns a _promise-ish_. `map(parse)` then maps the result of `readFile()` to a new _promise-ish_, which should be flattened to resolved values before being passed to `db.insert()`. In the `async`/`await` code, this is done by the `await` keyword before `parse()`, and here we need to do it with the `flatten` utility.

P.s., the [`flatten()` utility](https://stackblitz.com/edit/callback-heaven?file=lib%2Foperators%2Fflatten.ts) is also pretty simplistic in nature:
```js
function flatten(src) {
  return cb => src((err, res) => {
    if (err) cb(err, undefined)
    else res((err, res) => {
      if (err) cb(err, undefined)
      else cb(undefined, res)
    })
  })
}
```

Let's also take a look at another example: here, we want to fetch some Pokémon info from [PokéAPI](https://pokeapi.co) and log its abilities:

```js
fetch('https://pokeapi.co/api/v2/pokemon/ditto')
|> map(res => res.json())
|> flatten
|> then(res => console.log(res.abilities))
```
```js
async(() => {
  let res = await fetch('https://pokeapi.co/api/v2/pokemon/ditto')
  res = await res.json()
  console.log(res.abilities)
})()
```

---

# Conclusion

So to recap, these seem to have been the main issues resulting in _callback hells_:

- Lots of boilerplate code
- Boilerplate code severely intertwined with important code

As per our little experiment, addressing the second issue in the simplest manner (just separating boilerplate code and important code with no other change) was pretty key: it allowed us to bundle boilerplate code into small utility functions and reduce the ratio of boilerplate code and important code, making it (almost) as convenience as a heavy-handed solution such as adding new syntax to the language itself.

This notion is particularly important: you might have ugly implementation details and boilerplate that you cannot get rid of, but you can always bundle it together and separate it from actual important code, and doing this even in the simplest manner opens the door for turning a _hellish_ situation into a _heavenly_ one.

It is also notable that the same methodology is applicable to other, similar problems we are facing today. While we have (mostly) solved the issues of _asynchronous functions_, newer constructs such as _asynchronous streams_ (which are like async functions but with many, possibly infinite outputs instead of one) keep creeping into our toolbox and demand similar problem solving.

_P.s. the name callback heaven actually comes from the [callbag specification](https://github.com/callbag/callbag), which is like our new callback convention but for streams instead of promises. If you enjoyed this post, be sure to check it out as well._