Killing mutants to improve your tests
Tags: webdev, mutationtesting, testing, javascript
Published: 2022-11-07 10:10:24

---
title: "Killing mutants to improve your tests"
published: true
description: 
tags: 
- webdev
- mutationtesting 
- testing
- javascript
canonical_url: https://codesai.com/posts/2019/05/killing-mutants-to-improve-tests
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lrfhrhgefivjc26e5reo.jpg
# Use a ratio of 100:42 for best results.
# published_at: 2022-11-05 23:13 +0000
---

At my current client we're working on having a frontend architecture for writing SPAs in JavaScript similar to [re-frame](https://github.com/Day8/re-frame)'s one: an event-driven bus with effects and coeffects for state management<a href="#nota1"><sup>[1]</sup></a> (commands) and subscriptions using [reselect](https://github.com/reduxjs/reselect)'s selectors (queries).  

One of the pieces we have developed to achieved that goal is [reffects-store](https://github.com/trovit/reffects/tree/master/packages/reffects-store). Using this store, [React](https://reactjs.org/) components can be subscribed to given [reselect](https://github.com/reduxjs/reselect)'s selectors, so that they only render when the values in the application state tracked by the selectors change.

After we finished writing the code for the store, we decided to use [mutation testing](https://en.wikipedia.org/wiki/Mutation_testing) to evaluate the quality of our tests. *Mutation testing* is a technique in which, you introduce bugs, (*mutations*), into your production code, and then run your tests for each mutation. If your tests fail, it's ok, the mutation was "killed", that means that they were able to defend you against the regression caused by the mutation. If they don't, it means your tests are not defending you against that regression. The higher the percentage of mutations killed, the more effective your tests are.

There are tools that do this automatically, [stryker](https://stryker-mutator.io/)<a href="#nota2"><sup>[2]</sup></a> is one of them. When you run *stryker*, it will create many mutant versions of your production code, and run your tests for each mutant (that's how mutations are called in *stryker*'s' documentation) version of the code. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. Let's have a look at the the result of runnning *stryker* against [reffects-store](https://github.com/trovit/reffects/tree/master/packages/reffects-store)'s code:

```shell
$ npm run test:mutants

> reffects-store@0.1.0 test:mutants /home/trikitrok/todos-spike/reffects-store
> stryker run

21:38:12 (4489) INFO ConfigReader Using stryker.conf.js in the current working directory.
21:38:12 (4489) INFO InputFileResolver Found 3 of 13 file(s) to be mutated.
21:38:12 (4489) INFO InitialTestExecutor Starting initial test run. This may take a while.
21:38:15 (4489) INFO InitialTestExecutor Initial test run succeeded. Ran 23 tests in 2 seconds (net 125 ms, overhead 2331 ms).
21:38:15 (4489) INFO MutatorFacade 41 Mutant(s) generated
21:38:15 (4489) INFO SandboxPool Creating 8 test runners (based on CPU count)
Mutation testing  [==================================================] 100% (ETC n/a) 41/41 tested (6 survived)

15. [Survived] Block
/home/trikitrok/todos-spike/reffects-store/src/store.js:38:63
-     listeners.forEach(function iterateListeners(currentListener) {
-       if (currentListener === listener) {
-         listener = null;
-       } else {
-         out.push(currentListener);
-       }
-     });
+     listeners.forEach(function iterateListeners(currentListener) {});

Ran all tests for this mutant.
18. [Survived] IfStatement
/home/trikitrok/todos-spike/reffects-store/src/store.js:39:8
-       if (currentListener === listener) {
+       if (true) {

Ran all tests for this mutant.
19. [Survived] BinaryExpression
/home/trikitrok/todos-spike/reffects-store/src/store.js:39:8
-       if (currentListener === listener) {
+       if (currentListener !== listener) {

Ran all tests for this mutant.
20. [Survived] Block
/home/trikitrok/todos-spike/reffects-store/src/store.js:39:38
-       if (currentListener === listener) {
-         listener = null;
-       } else {
+       if (currentListener === listener) {} else {

Ran all tests for this mutant.
21. [Survived] Block
/home/trikitrok/todos-spike/reffects-store/src/store.js:41:11
-       } else {
-         out.push(currentListener);
-       }
+       } else {}

Ran all tests for this mutant.
22. [Survived] Block
/home/trikitrok/todos-spike/reffects-store/src/store.js:53:33
-   export function unsubscribeAll() {
-     listeners = [];
-   }
+   export function unsubscribeAll() {}

Ran all tests for this mutant.
Ran 15.27 tests per mutant on average.
------------------|---------|----------|-----------|------------|----------|---------|
File              | % score | # killed | # timeout | # survived | # no cov | # error |
------------------|---------|----------|-----------|------------|----------|---------|
All files         |   85.37 |       35 |         0 |          6 |        0 |       0 |
 store.js         |   75.00 |       18 |         0 |          6 |        0 |       0 |
 storeUtils.js    |  100.00 |        4 |         0 |          0 |        0 |       0 |
 subscriptions.js |  100.00 |       13 |         0 |          0 |        0 |       0 |
------------------|---------|----------|-----------|------------|----------|---------|
21:38:27 (4489) INFO HtmlReporter Your report can be found at: file:///home/trikitrok/todos-spike/reffects-store/reports/mutation/html/index.html
21:38:27 (4489) INFO Stryker Done in 14 seconds.
```

Notice how *stryker* shows the details of every mutation that survived our tests, and look at the summary the it produces at the end of the process.

All the surviving mutants were produced by mutations to the `store.js` file. Having a closer look to the mutations in *stryker*'s output we found that the functions with mutant code were `unsubscribeAllListeners` and `unsubscribeListener`.
After a quick check of their tests, it was esay to find out why `unsubscribeAllListeners` was having surviving mutants. Since it was a function we used only in tests for cleaning the state after each test case was run, we had forgotten to test it. 

However, finding out why `unsubscribeListener` mutants were surviving took us a bit more time and thinking.
Let's have a look at the tests that were exercising the code used to subscribe and unsubscribe listeners of state changes:

```javascript
describe('subscriptions to store changes', () => {
  test('subscribing to store changes', () => {
    const newValue = 'lolo';
    const path = ['koko'];
    store.initialize({ koko: 'loko' });

    store.subscribeListener(function(newState) {
      expect(newState).toEqual({ koko: newValue });
    });

    store.setState({ path, newValue });
  });

  test('unsubscribing from store changes', () => {
    const newValue = 'lolo';
    const path = ['koko'];
    store.initialize({ koko: 'loko' });
    function functionToUnsubscribe() {
      throw new Error('check is still suscribed!!');
    }
    store.subscribeListener(function(newState) {
      expect(newState).toEqual({ koko: newValue });
    });
    store.subscribeListener(functionToUnsubscribe);

    store.unsubscribeListener(functionToUnsubscribe);

    store.setState({ path, newValue });
  });
});
```

If we examine the mutations and the tests, we can see that the tests for `unsubscribeListener` are not good enough. They are throwing an exception from the subscribed function we unsubscribe, so that if the `unsubscribeListener` function doesn't work and that function is called the test fails. Unfortunately, the test passes also if that function is never called for any reason. In fact, most of the surviving mutants that *stryker* found above have are variations on that idea.

A better way to test `unsubscribeListener` is using spies to verify that subscribed functions are called and unsubscribed functions are not (this version of the tests includes also a test for `unsubscribeAllListeners`):

```javascript
describe('subscriptions to store changes', () => {
  test('subscribing a listener to store changes', () => {
    const newValue = 'lolo';
    const path = ['koko'];
    store.initialize({ koko: 'loko' });
    const fn = jest.fn(newState =>
      expect(newState).toEqual({ koko: newValue })
    );

    store.subscribeListener(fn);
    store.setState({ path, newValue });

    expect(fn).toHaveBeenCalledTimes(1);
  });

  test('unsubscribing a listener from store changes', () => {
    const newValue = 'lolo';
    const path = ['koko'];
    store.initialize({ koko: 'loko' });
    const spyToBeUnsubscribed = jest.fn();
    const spyNotUnsubscribed = jest.fn();

    store.subscribeListener(spyNotUnsubscribed);
    store.subscribeListener(spyToBeUnsubscribed);
    store.unsubscribeListener(spyToBeUnsubscribed);
    store.setState({ path, newValue });

    expect(spyToBeUnsubscribed).toHaveBeenCalledTimes(0);
    expect(spyNotUnsubscribed).toHaveBeenCalledTimes(1);
  });

  test('unsubscribing all listeners from store changes', () => {
    const newValue = 'lolo';
    const path = ['koko'];
    store.initialize({ koko: 'loko' });
    const spyToBeUnsubscribed1 = jest.fn();
    const spyToBeUnsubscribed2 = jest.fn();
    
    store.subscribeListener(spyToBeUnsubscribed1);
    store.subscribeListener(spyToBeUnsubscribed2);
    store.unsubscribeAllListeners();
    store.setState({ path, newValue });

    expect(spyToBeUnsubscribed1).toHaveBeenCalledTimes(0);
    expect(spyToBeUnsubscribed2).toHaveBeenCalledTimes(0);
  });
});
```

After this change, when we run *stryker* we got the following output:

```shell
$ npm run test:mutants

> reffects-store@0.1.4 test:mutants /home/trikitrok/todos-spike/reffects-store
> stryker run

21:49:18 (4981) INFO ConfigReader Using stryker.conf.js in the current working directory.
21:49:18 (4981) INFO InputFileResolver Found 4 of 14 file(s) to be mutated.
21:49:18 (4981) INFO InitialTestExecutor Starting initial test run. This may take a while.
21:49:21 (4981) INFO InitialTestExecutor Initial test run succeeded. Ran 24 tests in 2 seconds (net 122 ms, overhead 2313 ms).
21:49:21 (4981) INFO MutatorFacade 39 Mutant(s) generated
21:49:21 (4981) INFO SandboxPool Creating 8 test runners (based on CPU count)
Mutation testing  [==================================================] 100% (ETC n/a) 39/39 tested (0 survived)

Ran 15.74 tests per mutant on average.
--------------|---------|----------|-----------|------------|----------|---------|
File          | % score | # killed | # timeout | # survived | # no cov | # error |
--------------|---------|----------|-----------|------------|----------|---------|
All files     |  100.00 |       39 |         0 |          0 |        0 |       0 |
 subscription |  100.00 |       13 |         0 |          0 |        0 |       0 |
  index.js    |  100.00 |       13 |         0 |          0 |        0 |       0 |
 store.js     |  100.00 |       22 |         0 |          0 |        0 |       0 |
 utils.js     |  100.00 |        4 |         0 |          0 |        0 |       0 |
--------------|---------|----------|-----------|------------|----------|---------|
21:49:32 (4981) INFO HtmlReporter Your report can be found at: file:///home/trikitrok/todos-spike/reffects-store/reports/mutation/html/index.html
21:49:32 (4981) INFO Stryker Done in 13 seconds.
```

No mutants survived!! This means this new version of the tests is more reliable and will protect us better from regressions than the initial version. 

Mutation testing is a great tool to know if you can trust your tests. This is event more true when working with legacy code.

**Acknowledgements.**

Many thanks to <a href="https://twitter.com/MrMSanchez">Mario Sánchez</a> and <a href="https://twitter.com/alexhoma_">Alex Casajuana Martín</a> for all the great time coding together, and thanks to Porapak Apichodilok for the [photo used in this post](https://www.pexels.com/photo/boy-child-clouds-kid-346796/) and to [Pexels](https://www.pexels.com/).

**Footnotes**:

<div class="foot-note">
  <a name="nota1"></a> [1] See also <a href="https://github.com/trovit/reffects">reffects</a> which is the synchronous event bus with effects and coeffects we wrote to manage the application state.
</div>
<div class="foot-note">
  <a name="nota2"></a> [2] The name of this tool comes from a fictional Marvel comics supervillain <a href="https://en.wikipedia.org/wiki/William_Stryker">Willian Stryker</a> who was obsessed with the eradication of all mutants.
</div>