Testing in parallel with Mocha v8.0.0
Tags: node, javascript, testing
Published: 2020-06-10 21:03:01

---
title: 'Testing in parallel with Mocha v8.0.0'
published: true
description: 'Use parallel mode to achieve significant speedups for large test suites'
tags: node, javascript, testing
canonical_url: https://developer.ibm.com/technologies/node-js/articles/parallel-tests-mocha-v8/
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/cei0slwuopm0xg4glyut.png
---

With the [release of Mocha v8.0.0](https://github.com/mochajs/mocha/releases/tag/v8.0.0), Mocha now supports running in _parallel mode_ under Node.js. Running tests in parallel mode allows Mocha to take advantage of multi-core CPUs, resulting in significant speedups for large test suites.

>_Read about parallel testing in [Mocha’s documentation](https://mochajs.org/#parallel-tests)._

Before v8.0.0, Mocha only ran tests in _serial_: one test must finish before moving on to the next. While this strategy is not without benefits &mdash; it's deterministic and snappy on smaller test suites &mdash; it can become a bottleneck when running a large number of tests.

Let's take a look at how to take advantage of parallel mode in Mocha by enabling it on a real-world project: Mocha itself!

## Installation

>_Node.js v8.0.0 has reached End-of-Life; Mocha v8.0.0 requires Node.js v10, v12, or v14._

Mocha doesn’t need to install itself, but you might. You need Mocha v8.0.0 or newer, so:

```bash
npm i mocha@8 --save-dev
```

Moving right along...

## Use the `--parallel` flag

In many cases, all you need to do to enable parallel mode is supply `--parallel` to the `mocha` executable. For example:

```bash
mocha --parallel test/*.spec.js
```

Alternatively, you can specify any command-line flag by using a Mocha [configuration file](https://mochajs.org/#configuring-mocha-nodejs). Mocha keeps its default configuration in a YAML file, `.mocharc.yml`. It looks something like this (trimmed for brevity):

```yaml
# .mocharc.yml
require: 'test/setup'
ui: 'bdd'
timeout: 300
```

To enable parallel mode, I'm going to add `parallel: true` to this file:

```yaml
# .mocharc.yml w/ parallel mode enabled
require: 'test/setup'
ui: 'bdd'
timeout: 300
parallel: true
```

>_Note: Examples below use `--parallel` and `--no-parallel` for the sake of clarity._

Let's run `npm test` and see what happens!

## Spoiler: It didn't work the first time

Oops, I got a bunch of "timeout" exceptions in the unit tests, which use the default timeout value (300ms, as shown above). Look:

```
  2) Mocha
       "before each" hook for "should return the Mocha instance":
     Error: Timeout of 300ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/boneskull/projects/mochajs/mocha/test/node-unit/mocha.spec.js)
      at Hook.Runnable._timeoutError (lib/runnable.js:425:10)
      at done (lib/runnable.js:299:18)
      at callFn (lib/runnable.js:380:7)
      at Hook.Runnable.run (lib/runnable.js:345:5)
      at next (lib/runner.js:475:10)
      at Immediate._onImmediate (lib/runner.js:520:5)
      at processImmediate (internal/timers.js:456:21)
```

That’s weird. I run the tests a second time, and _different_ tests throw "timeout" exceptions. Why?

Because of _many_ variables &mdash; from Mocha to Node.js to the OS to the CPU itself &mdash; parallel mode exhibits a much wider range of timings for any given test. These timeout exceptions don't indicate a newfound performance issue; rather, they're a symptom of a naturally higher system load and nondeterministic execution order.

To resolve this, I'll increase Mocha's default test timeout from 300ms (0.3s) to 1000ms (1s):

```yaml
# .mocharc.yml
# ...
timeout: 1000
```

Mocha's "timeout" functionality is _not_ to be used as a benchmark; its intent is to catch code that takes an unexpectedly long time to execute. Since we now _expect_ tests to potentially take longer, we can safely increase the `timeout` value.

Now that the tests pass, I'm going to try to make them _pass more_.

## Optimizing parallel mode

By default, Mocha's maximum job count is _n - 1_, where _n_ is the number of CPU cores on the machine. This default value _will not be optimal for all projects_. The job count also does _not_ imply that "Mocha gets to use _n - 1_ CPU cores," because that's up to the operating system. It is, however, a default, and it does what defaults do.

> When I say "_maximum_ job count" I mean that Mocha _could_ spawn this many worker processes if needed. It depends on the count and execution time of the test files.

To compare performance, I use the friendly benchmarking tool, [hyperfine](https://github.com/sharkdp/hyperfine); I'll use this to get an idea of how various configurations will perform.

> Regarding `hyperfine` usage:

> In the below examples, I’m passing two options to `hyperfine`: `-r 5` for "runs," which runs the command five (5) times. The default is ten (10), but this is slow, and I'm impatient. The second option I pass is `--warmup 1`, which performs a single "warmup" run. The result of this run is discarded. Warmup runs reduce the chance that the first _k_ runs will be significantly slower than the subsequent runs, which may skew the final result. If this happens, `hyperfine` will even warn you about it, which is why I'm using it!  
> If you try this yourself, you need to replace `bin/mocha` with `node_modules/.bin/mocha` or `mocha`, depending on your environment; `bin/mocha` is the path to the `mocha` executable relative to the working copy root.

Mocha's integration tests (about 260 tests over 55 files) typically make assertions about the output of the `mocha` executable itself. They also need a longer `timeout` value than the unit tests; below, we use a timeout of ten (10) seconds.

I run the integration tests in serial. Nobody ever claimed they ran at _ludicrous speed_:

```bash
$ hyperfine -r 5 --warmup 1 "bin/mocha --no-parallel --timeout \
10s test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --no-parallel --timeout 10s test/integration/**/*.spec.js
  Time (mean ± σ):     141.873 s ±  0.315 s    [User: 72.444 s, System: 14.836 s]
  Range (min … max):   141.447 s … 142.296 s    5 runs
```

That's over two (2) minutes. Let’s try it again in parallel mode. In my case, I have an eight-core CPU (`n = 8`), so by default, Mocha uses seven (7) worker processes:

```bash
$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel --timeout 10s \
test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --parallel --timeout 10s test/integration/**/*.spec.js
  Time (mean ± σ):     65.235 s ±  0.191 s    [User: 78.302 s, System: 16.523 s]
  Range (min … max):   65.002 s … 65.450 s    5 runs
```

Using parallel mode shaves 76 seconds off of the run, down to just over a minute! That's almost a 53% speedup. But, can we do better?

I can use the `--jobs/-j` option to specify _exactly_ how many worker processes Mocha will potentially use. Let's see what happens if I _reduce_ this number to four (4):

```bash
$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel --jobs 4 --timeout 10s \
test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --parallel --jobs 4 --timeout 10s \
test/integration/**/*.spec.js
  Time (mean ± σ):     69.764 s ±  0.512 s    [User: 79.176 s, System: 16.774 s]
  Range (min … max):   69.290 s … 70.597 s    5 runs
```

Unfortunately, that's slower. What if I _increased_ the number of jobs, instead?

```bash
$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel --jobs 12 --timeout 10s \
test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --parallel --jobs 12 --timeout 10s test/integration/**/*.spec.js
  Time (mean ± σ):     64.175 s ±  0.248 s    [User: 80.611 s, System: 17.109 s]
  Range (min … max):   63.809 s … 64.400 s    5 runs
```

Twelve (12) is ever-so-slightly faster than the default of seven (7). Remember, my CPU has eight (8) cores. Why does spawning _more_ processes increase performance?

I speculate it's because these tests aren't CPU-bound. They mostly perform asynchronous I/O, so the CPU has some spare cycles waiting for tasks to complete. I could spend more time trying to squeeze another 500ms out of these tests, but for my purposes, it's not worth the bother. Perfect is the enemy of good, right?  The point is to illustrate how you can apply this strategy to your own projects and arrive at a configuration you're happy with.

## When to avoid parallel mode

Would you be shocked if I told you that running tests in parallel is _not always appropriate?_ No, you would not be shocked.

It's important to understand two things:

1. **Mocha does not run individual tests in parallel**. Mocha runs _test files_ in parallel.
1. Spawning worker processes is not free.

That means if you hand Mocha _a single, lonely test file_, it will spawn _a single worker process_, and that worker process will run the file. _If you only have one test file, you'll be penalized for using parallel mode._ Don't do that.

Other than the "lonely file" non-use-case, the unique characteristics of your tests and sources will impact the result. There's an inflection point below which running tests in parallel will be _slower_ than running in serial.

In fact, Mocha's own unit tests (about 740 tests across 35 files) are a great example. Like good unit tests, they try to run quickly, in isolation, without I/O. I'll run Mocha's unit tests in serial, for the baseline:

```bash
$ hyperfine -r 5 --warmup 1 "bin/mocha --no-parallel test/*unit/**/*.spec.js"
Benchmark #1: bin/mocha --no-parallel test/*unit/**/*.spec.js
  Time (mean ± σ):      1.262 s ±  0.026 s    [User: 1.286 s, System: 0.145 s]
  Range (min … max):    1.239 s …  1.297 s    5 runs
```

Now I'll try running them in parallel. Despite my hopes, this is the result:

```bash
$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel test/*unit/**/*.spec.js"
Benchmark #1: bin/mocha --parallel test/*unit/**/*.spec.js
  Time (mean ± σ):      1.718 s ±  0.023 s    [User: 3.443 s, System: 0.619 s]
  Range (min … max):    1.686 s …  1.747 s    5 runs
```

Objectively, running Mocha's unit tests in parallel _slows them down_ by about half a second. This is the overhead of spawning worker processes (and the requisite serialization for inter-process communication). 

I'll go out on a limb and predict that _many projects_ having very _fast unit tests_ will _see no benefit_ from running those tests in Mocha's parallel mode.

Remember my `.mocharc.yml`?  I yanked that `parallel: true` out of there; instead, Mocha will use it only when running its integration tests.

In addition to being generally unsuitable for these types of tests, parallel mode has some other limitations; I'll discuss these next.

## Caveats, disclaimers and gotchas, Oh my

Due to technical limitations (i.e., "reasons"), a handful of features aren't compatible with parallel mode. If you try, Mocha will throw an exception.

>_[Check out the docs](https://mochajs.org/#parallel-tests) for more information and workarounds (if any)._

### Unsupported reporters

If you're using the `markdown`, `progress`, or `json-stream` reporters, you're out of luck for now. These reporters need to know how many tests we intend to execute _up front_, and parallel mode does not have that information.

>_This could change in the future, but would introduce breaking changes to these reporters._

### Exclusive tests

[Exclusive tests](https://mochajs.org/#exclusive-tests) (`.only()`) do not work. If you try, Mocha runs tests (as if `.only()` was not used) until it encounters usage of `.only()`, at which point it aborts and fails.

Given that exclusive tests are typically used in a single file, parallel mode is _also_ unsuitable for this situation. 

### Unsupported options

Incompatible options include `--sort`, `--delay`, and importantly, `--file`. In short, it's because we cannot run tests in any specific order.

Of these, `--file` likely impacts the greatest number of projects. Before Mocha v8.0.0, `--file` was recommended to define "root hooks." Root hooks are hooks (such as `beforeEach()`, `after()`, `setup()`, etc.) which all other test files will inherit. The idea is that you would define root hooks in, for example, `hooks.js`, and run Mocha like so:

```bash
mocha --file hooks.js "test/**/*.spec.js"
```

All `--file` parameters are considered test files and will be run _in order_ and _before any other test files_ (in this case, `test/**/*.spec.js`). Because of these guarantees, Mocha "bootstraps" with the hooks defined in `hooks.js`, and this affects all subsequent test files.

This still works in Mocha v8.0.0, but _only_ in serial mode. But wait!  Its use is now _strongly discouraged_ (and will eventually be fully deprecated). In its place, Mocha has introduced Root Hook Plugins.

## Root Hook Plugins

Root Hook Plugins are modules (CJS or ESM) which have a named export, `mochaHooks`, in which the user can freely define hooks. Root Hook Plugin modules are loaded via Mocha's `--require` option.

>_[Read the docs on using Root Hook Plugins](https://mochajs.org/#root-hook-plugins)_

The documentation (linked above) contains a thorough explanation and more examples, but here’s a straightforward one.

Say you have a project with root hooks loaded via `--file hooks.js`:

```js
// hooks.js
beforeEach(function() {
  // do something before every test
  this.timeout(5000); // trivial example
});
```

To convert this to a Root Hook Plugin, change `hooks.js` to be:

```js
// hooks.js
exports.mochaHooks = {
  beforeEach() {
    this.timeout(5000);
  }
};
```

>_Tip: This can be an ESM module, for example, `hooks.mjs`; use a named export of `mochaHooks`._

When calling the `mocha` executable, replace `--file hooks.js` with `--require hooks.js`. Nifty!

## Troubleshooting parallel mode

While parallel mode should _just work_ for many projects, if you're still having trouble, refer to this checklist to prepare your tests:

- ✅ Ensure you are using a [supported reporter](https://mochajs.org#reporter-limitations).
- ✅ Ensure you are not using [other unsupported flags](https://mochajs.org#file-order-is-non-deterministic).
- ✅ Double-check your [config file](https://mochajs.org/#configuring-mocha-nodejs); options set in config files will be merged with any command-line option.
- ✅ Look for _root hooks_ (they look like [this](https://mochajs.org/#root-hooks-are-not-global)) in your tests. Move them into a [root hook plugin](https://mochajs.org/#root-hook-plugins).
- ✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be [migrated](https://mochajs.org/#migrating-a-library-to-use-root-hook-plugins) for compatibility with parallel mode.
- ✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via [`--timeout`](https://mochajs.org/#-timeout-ms-t-ms))
- ✅ Ensure your tests do not depend on being run in a specific order.
- ✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files.

## What's next

Parallel mode is _new_ and _not perfect_; there's room for improvement. But to do so, Mocha needs your help. Send the Mocha team [your feedback](https://github.com/mochajs/mocha/issues)! Please give [Mocha v8.0.0](https://github.com/mochajs/mocha/releases/tag/v8.0.0) a try, enable parallel mode, use Root Hook Plugins, and share your thoughts.