Getting started with Mutation Testing
Tags: testing, php, laravel, tutorial
Published: 2021-02-12 10:35:53

---
title: Getting started with Mutation Testing
description: Mutation testing is a great tool to analyze and increase the reliability of your test suite. This post highlights the concepts of Mutation Testing, how to get started using Infection PHP, and how to use mutation tests in your CI workflow.
published: true
date: 2021-02-10 00:00:00 UTC
tags: testing, php, laravel, tutorial
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/l2df8wxdosq3t5v7n2ip.png
canonical_url: https://johnbraun.blog/posts/mutation-testing
---

## Introduction
How do you determine if the current test suite adds enough value? Do the tests cover all edge cases? If you solely rely on **code coverage**, you might be missing out. 

In a recent talk at the [Laracon EU Online](https://laracon.eu/online/discover/conference/?21) conference, my colleague [Jeroen Groenendijk](https://jeroeng.dev/) highlighted the significance of  *Mutation Testing* in achieving **greater confidence** in the **test suite** of your application.

In this blog I'd like to highlight the concepts of [Mutation Testing](https://en.wikipedia.org/wiki/Mutation_testing), explain how to get started using [Infection PHP](https://infection.github.io/) by [Maks Rafalko](https://twitter.com/maks_rafalko), show some practical examples and finally explain how to use Mutation Testing in a CI setup.

## What is Mutation Testing
A mutation testing tool will manipulate (*mutate*) pieces of your source code and run the test suite against this piece of *mutant* source code. The mutated code **should** trigger a failing test, or the mutant **escapes**. Escaped mutants are a sign of weakly tested code.

As an illustration, take a look at the following example.

![Mutation Testing diagram](https://dev-to-uploads.s3.amazonaws.com/i/zt7ti4456xzdk0xbkbjb.png)

The `add()` method is mutated in three different ways:

>- The method returns `null`
- The `+` operator changed to the `-` operator
- Method visibility changed from `public` to `protected`

In the diagram above, we see that the tests fail in case `null` is returned and when the addition is replaced with a subtraction. The method's visibility however is likely too broad and *mutating* it to `protected` didn't yield a failing test. Therefore, it should be changed to `protected` or even `private`, adhering to the best practice of keeping the public API of a class to the minimum.

## Getting started with Infection PHP
If you just want to play around, check the [Infection Playground](https://infection-php.dev/), where you can write the code, tests, and run Infection right from within your browser. Read on to learn more about setting up Infection PHP in your project.

### Setting up Infection PHP
Infection requires PHP version 7.2 (or higher) and an enabled debugger of choice: [Xdebug](https://xdebug.org/), [phpdbg](https://infection.github.io/guide/usage.html#Running-with-phpdbg) or [pcov](https://github.com/krakjoe/pcov/blob/develop/INSTALL.md). *Tip: make sure to check out [LearnXDebug.com](https://learnxdebug.com/) when you're setting up Xdebug in a Laravel oriented development environment.*

In the steps below, I assume having xDebug installed.

While you have [multiple options](https://infection.github.io/guide/installation.html) to install Infection, I would recommend installing it as a `dev` dependency using composer within your project.

#### Step 1. Install Infection PHP

```bash
composer require --dev infection/infection
```

#### Step 2. Run Infection
```bash
vendor/bin/infection
```

On the first run, Infection will ask for input:

> - Directories to include:
    - For **Laravel projects** this means your `app` directory.
    - For **PHP packages** this means the `src` directory.
- Excluding directories from source directories:
    - Leave this blank, unless you have PHP code living in the source directory that shouldn't be mutated.
- Where to store the text log file:
    - I would recommend saving the mutations to *e.g.* `infection.log`. All escaped mutants are saved here for later review.
    - Alternatively, you may use the `--show-mutations` option to log the mutations to the terminal output.

You'll now find `infection.json.dist` in the root of your project reflecting your input:

```json
{
    "source": {
        "directories": [
            "src"
        ]
    },
    "logs": {
        "text": "infection.log"
    },
    "mutators": {
        "@default": true
    }
}
```

You can specify if you want to enable (or disable) a specific mutator under the "mutators" key. Make sure to check the [complete list of available mutators](https://infection.github.io/guide/mutators.html).

Tip: allow the mutation tests to run in parallel by providing the `--threads` option and setting it greater than 1, for example by running `vendor/bin/infection --threads=4`. This will greatly speed up running the mutation tests. An overview of all command-line options can be found [here](https://infection.github.io/guide/command-line-options.html).

## Code Examples

### Example 1: Calculating shipping cost
To demonstrate Mutation Testing in practice, I'll borrow the example from Jeroen's talk.

Imagine having a `ShippingCalculator` service class to determine if a passed in `Product` qualifies for free shipping. For simplicity’s sake, let's say the `Product` class accepts a `$price` integer through its constructor and provides public access to a `$shipsForFree` property.

The `ShippingCalculator` class determines that a `Product` receives free shipping when:
- The price is equal to or greater than the threshold (set to an arbitrary value)
- The product's `$shipsForFree` property is `true` (or truthy)

```php
class ShippingCalculator
{
    const FREE_SHIPPING_THRESHOLD = 20;
    
    public static function hasFreeShipping(Product $product): bool
    {        
        if ($product->price >= self::FREE_SHIPPING_THRESHOLD) {
            return true;
        }
        
        if ($product->shipsForFree) {
            return true;
        }
        
        return false;
    }
}
```

#### Testing the code with PHPUnit
To make sure the `ShippingCalculator::hasFreeShipping()` method works properly, we can think of adding the following unit tests to ensure proper behavior:

- When a product's `$price` exceeds the threshold, it should ship for free
- When a product's `$price` does not exceed the threshold, it should not ship for free
- When a product's `$shipsforFree` property set to `true`, it should ship for free

```php
class ShippingCalculatorTest extends TestCase
{
    /** @test */
    function product_ships_for_free_when_price_is_above_treshold()
    {
        $product = new Product($price = ShippingCalculator::FREE_SHIPPING_THRESHOLD + 1);

        $this->assertTrue(ShippingCalculator::hasFreeShipping($product));
    }

    /** @test */
    function product_does_not_ship_for_free_when_price_is_below_treshold()
    {
        $product = new Product($price = ShippingCalculator::FREE_SHIPPING_THRESHOLD - 1);

        $this->assertFalse(ShippingCalculator::hasFreeShipping($product));
    }

    /** @test */
    function product_ships_for_free_when_ships_for_free_property_is_true()
    {
        $product = new Product(ShippingCalculator::FREE_SHIPPING_THRESHOLD - 1);
        $product->shipsForFree = true;

        $this->assertTrue(ShippingCalculator::hasFreeShipping($product));
    }
}
```

With these three tests, the code coverage report from PHPUnit (which you can generate using `vendor/bin/phpunit --coverage-text`) reveals a code coverage of 100% for both classes.

```bash
 Summary:
  Classes: 100.00% (2/2)
  Methods: 100.00% (2/2)
  Lines:   100.00% (7/7)
```

#### Running Infection PHP
Now, we'll run Infection PHP using `vendor/bin/infection` and see if any mutants escape.

```bash
.M....                                               (6 / 6)

6 mutations were generated:
       5 mutants were killed
       0 mutants were not covered by tests
       1 covered mutants were not detected
       ...

Metrics:
         Mutation Score Indicator (MSI): 83%
         Mutation Code Coverage: 100%
         Covered Code MSI: 83%
```

Oh no, a mutant has escaped!

Furthermore, our **MSI** is **83%**, while the generated mutations covered 100% of the code. This means 5 out of 6 mutants were killed.

When we check our log file, we see the `[M] GreaterThanOrEqualTo` mutant escaped:

```bash
Escaped mutants:
================
1) ../src/ShippingCalculator.php:15    [M] GreaterThanOrEqualTo

--- Original
+++ New
@@ @@
-        if ($product->price >= self::FREE_SHIPPING_THRESHOLD) {
+        if ($product->price > self::FREE_SHIPPING_THRESHOLD) {
```

#### The missing test
Now it becomes obvious that we are **missing a crucial test**: we did not assert what happens when the `$price` is **equal** to the free shipping threshold.

Infection PHP expected at least one of our tests to fail when mutating the conditional from a `great-than-or-equals` to a `greater-than` comparison. Since our test suite didn't fail, this mutation got away unnoticed.

Let's fix that by adding in the "forgotten" boundary test:

```php
/** @test */
function product_ships_for_free_when_price_equals_threshold()
{
    $product = new Product(ShippingCalculator::FREE_SHIPPING_THRESHOLD);

    $this->assertTrue(ShippingCalculator::hasFreeShipping($product));
}
```

When we run Infection again, we get an MSI of 100%. No escaped mutants this time!

### Example 2: Redirection
After using mutation testing in my Laravel projects, I learned that mutation testing drives out tests I would otherwise not have written.

When redirecting users to a specific page with a "date" parameter from the `store()` controller action.

```php
class AppointmentController
{
    public function store()
    {
        // controller logic
        
        return redirect(route('appointments.index', ['date' => $date]));
    }
}
```

In my test for this controller action, I initially did not include a check that the user was redirected to a specific page. However, when running the Infection PHP mutation test suite I got the following **escaped mutant**:

```bash
3) .../app/Http/Controllers/AppointmentController.php:163    [M] ArrayItemRemoval

--- Original
+++ New
@@ @@
-        return redirect(route('appointments.index', ['date' => $date]));
+        return redirect(route('appointments.index', []));
     }
 }
```

While this `date` parameter in the redirect is **crucial** for users of the application to land on the appropriate page, **I didn't have a test** for this behavior. Until now, not having a test for the specific redirect could go unnoticed. Thanks to mutation testing I've now discovered this gap and it has **forced** me to add a test covering this scenario.

## Using Infection in CI
It is possible to run mutation tests in Continuous Integration (CI) against newly added code or before builds. 

You can set a `--min-msi` score option to force a certain percentage of mutants to be killed and gradually increase this number to improve the project's test suite. If the MSI score is lower than required, the build will fail. 

### Small projects
For small projects, the simplest option is to first run the PHPUnit tests while saving the test coverage files in a `build` directory and feeding those directly to Infection PHP.

```bash
vendor/bin/phpunit --coverage-xml=build/coverage-xml --log-junit=build/phpunit.junit.xml
vendor/bin/infection --threads=2 --coverage=build --min-msi=70
```

### GitHub Actions
Although Infection can be integrated with any CI setup, I like to highlight the newly added integration with GitHub Actions (in version 0.20), where it is possible to log escaped mutants in committed code directly within a PR.

![Infection GitHub Annotations](https://dev-to-uploads.s3.amazonaws.com/i/j798ogcgp6jaewaag157.png)

Basically, you can add the following action to your GitHub Action workflow:

```yml
- name: Run Infection for added (A) and modified (M) files
  run: |
    git fetch --depth=1 origin $GITHUB_BASE_REF
    php vendor/bin/infection --threads=2 --git-diff-base=origin/$GITHUB_BASE_REF --git-diff-filter=AM --logger-github
```

Check out [the release documentation](https://infection.github.io/2020/11/01/whats-new-in-0.20.0/#Github-Annotations-logger-%F0%9F%9A%80) for more details.

### Larger projects
If you want to use Mutation Testing in CI for larger projects, make sure to check out [this article](https://alejandrocelaya.blog/2018/02/17/mutation-testing-with-infection-in-big-php-projects/) by Alejandro Celaya. 

In his blog post, he advises using `phpdbg` to run Infection which leads to a clear gain in performance.

## Conclusion
I hope this blog post provides some insight and direction to get started with Mutation Testing. In summary, Mutation Testing creates mutants from your source code which run against your existing test suite. If none of these tests fail, it is a sign that this piece of the source code (*the mutant*) is weakly tested. This methodology helps to identify gaps in your current test suite as well as dead or unnecessary code (or *e.g.* unnecessary broad method visibility).

Although this blog post merely covers some examples, there is a lot more to explore in Mutation Testing: make sure to check out [all available mutators](https://infection.github.io/guide/mutators.html) and the [how-to guide](https://infection.github.io/guide/how-to.html).


### Drawbacks
While Mutation Testing has a lot of advantages, think of the following possible drawbacks and things to be aware of.

>- Running mutation tests is quite slow. You can speed up the process by allowing testing in parallel (set `--threads` > 1).
- Not all mutants have to be killed. Trying to kill all mutants might lead to **tight coupling** between your code and your tests. Or lead to code that is *too strict* and no longer **flexible**.

### More resources
If you want to learn more about Mutation Testing, make sure to check out these great resources on Mutation Testing (using Infection):

Articles:
>- [Infection: Mutation Testing Framework - Maks Rafalko (Infection PHP creator)](https://medium.com/@maks_rafalko/infection-mutation-testing-framework-c9ccf02eefd1)
- [Mutation testing in PHP - Moxio (	Arnout Boks)](https://www.moxio.com/blog/39/mutation-testing-in-php)
- [Mutation Testing With Infection - Matthew Daly](https://matthewdaly.co.uk/blog/2018/09/13/mutation-testing-with-infection/)
- [Mutation testing with infection in big PHP projects - Alejandro Celaya](https://alejandrocelaya.blog/2018/02/17/mutation-testing-with-infection-in-big-php-projects/)

Talks:
>- [Kill All Mutants! - Dave Aronson](https://talksatconfs.com/talks/ddfa9479-2ceb-4634-90e7-ca53aa26c24c/kill-all-mutants-intro-to-mutation-testing)
- [Better Code By Making Bugs with Mutation Testing - Theo Fidry](https://skillsmatter.com/skillscasts/11237-better-code-by-making-bugs-with-mutation-testing)