Beware silently skipped tests in Mocha
Tags: javascript, webdev, testing, mocha
Published: 2021-03-03 17:01:28

I'm a big believer in test-driven development, though I admit I'm a recent convert. My framework of choice is [Mocha](https://mochajs.org/).

There are a couple of gotchas when using Mocha that can cause you to accidentally *skip tests*. The scary version of this results in Mocha telling you that all your tests have passed, when in fact some of your tests *weren't run at all*!

Here are the two causes of this that have hit me particularly hard, and that are both easy to fix:

1. Async errors thrown outside of test definitions are silently swallowed by Mocha.js.
2. If you frequently use the `.only` flag to limit test runs to a subset of tests, you might forget to remove that flag.

### Silent, swallowed errors.

A while ago there was a CLI tool I was building that had been merrily passing all of its tests for *weeks*. When I finally made it available for the team, it was completely broken. Weirdly, it was broken in ways that I *knew* I had test coverage for. How could this be?

It turns out that errors thrown in async contexts can cause Mocha to exit early without registering all tests, all while swallowing the error that that caused it to happen! Observe:

```js
describe('My test suite', async function(){

  // throw new Error("Bwahahaha! Tricked you!");

  it('can pass this test', async function(){
    // This will "pass", even without any code,
    // since Mocha tests pass unless an error is thrown.
  });

  it('cannot pass this test', async function(){
    throw new Error("OH NOOOOOO!");
  });
});
```

This test runs as expected, informing us that one test passed and one failed:

![Screenshot of the expected output of a failed Mocha test.](https://i.imgur.com/2bScqem.png)

But what happens if we uncomment out that extra thrown error?  Despite there now being *two* explicitly thrown errors in that little Mocha snippet, we get this:

![Screenshot of Mocha output falsely showing that a test didn't fail.](https://i.imgur.com/iEdXgXe.png)

Yeah, sure it says *zero* passed, which sounds like a failure. But that's in green, because Mocha saw *zero tests*! This is a *success state*, because Mocha doesn't care about things that pass (or that nothing passed), only things that *fail*. And nothing failed, according to Mocha.

When something fails, Mocha exits with a non-zero status. That non-zero exit would be used to inform downstream tools that something has gone wrong, preventing your automated pipelines from continuing when tests fail. But here we got a `0` status despite obvious test failures.

Even without the automation problem, this same bug can be hard to spot when doing things manually. Sure, in this case "0 tests passed" is quite obviously wrong. But this problem can cause a *subset* of tests to get skipped, so you might see "321 tests passed" when there should have been "351". If you hadn't memorized how many tests you had there'd be no way to realize you were skipping tests.

As a workaround, you can tell the Node process to catch such errors and force a non-zero exit status:

```js
function onUncaught(err){
  console.log(err);
  process.exit(1);
}

process.on('unhandledRejection', onUncaught);

describe('My test suite', async function(){

  throw new Error("Bwahahaha! Tricked you!");
  // ...
```

And now we get:

![Screenshot showing the error no longer being silently swallowed.](https://i.imgur.com/HZNqDWV.png)

***NOTE:*** *While you can, technically, use `async` callbacks in your `describe()`s, it very likely will not behave the way that you expect! If you remove the `async` in the example's `describe`, the thrown error is no longer swallowed. The protection mechanism shown here should be treated as a backup for accidentally making a `describe()` async!*

### Forgotten `.only()`

When you're actively working on a new feature, or debugging an existing one, the test-driven approach is to first write the tests, ensure they're failing where they should be, and then code until all tests are passing.

If you're doing this in the context of a large project, you probably don't want to be running *all* tests just to see if the current thing is working. Mocha provides a few mechanisms for dealing with that, the easiest one being to use `.only` to indicate that *only that test* (and any others similarly flagged) should be run:

```js
describe('My test', function(){
  it.only('will run this test', function(){});
  it('will not run this test', function(){});
});
```

But what happens when you inevitably forget to remove that `.only` to ensure your other tests run again? You'll be bypassing tests that might be failing! I've done this countless times myself.

Mocha has a great solution for this: the `--forbid-only` flag.

When you add this flag to your CLI call, Mocha treats the mere *existence* of `.only` in *any* part of your test code as a test failure, and exits with a non-zero status.

For my build pipelines I *always* use this flag. It pairs nicely with `--bail`, which aborts as soon as a single test fails so that you don't waste time running other tests on a bad build.


*This article was adapted from the [DevChat newsletter](https://www.bscotch.net/post/devchat-6).*