When Was a Bug Introduced? Bisecting with Pytest
Tags: python, git, pytest, debugging
Published: 2019-06-24 21:59:39

Turns out that a very old test was broken for a while and wasn't running in the automatic build, whoops.

It would be very helpful to find the exact commit that broke the test. Well, [git-bisect](https://git-scm.com/docs/git-bisect) to the rescue: git bisect will binary search the commit history to find the bad commit. And we can use [pytest](https://docs.pytest.org/) to tell git if the test is passing or not for any possible commit.

## Set Up
Here is a toy repository with a simple function and a unit test:
```bash
$ ls
my_script.py  README
```

And this is `my_script.py`:
```python
from functools import reduce
from operator import mul

def product(factors):
    return reduce(lambda x , y: x - y, factors)


def test_product():
    assert product([1, 2, 3,]) == 6
```

Pretty simple so far. `test_product` broke for some reason.

Let's look at the commit history:
```
$ git log --oneline
7e26872 Unrelated commit # 21
f67f19e Unrelated commit # 20
...
2b37410 Unrelated commit # 15
80956a1 Unrelated commit # 14
d2a3327 Some innocent change that certainly did not break anything
8cc3f8b Unrelated commit # 13
1b09ced Unrelated commit # 12
...
3e73dfa Unrelated commit # 2
790ba87 Unrelated commit # 1
0c75c27 unrelated commit
d7af196 Added product function with a unit-test
```

## Bisecting
There are several ways to use `bisect`. We are going to tell it when the test was passing, and how to run the test, and it will take care of the rest.

We begin with `bisect start`:
```bash 
$ bisect_start
```

Tell it that our current commit is broken:
```bash
$ git bisect bad HEAD
```

Let's find a commit where we know the test was passing. Say, the commit that introduced the test:
```bash
$ git blame my_script.py | grep test_product
d7af196a (oren 2019-06-24 12:48:35 -0700 8) def test_product():
```

Tell `bisect` this commit is good:
```bash
$ git bisect good d7af196a
Bisecting: 11 revisions left to test after this (roughly 4 steps)
[97c3a24f28f72bec1885445eb3947e446d0804fe] Unrelated commit # 10
```

Now we start bisect off with a command to run the test, bisect uses the exit code of the command to tell whether each commit is good or bad, that works because `pytest` has a non-zero exit code if tests fail (And, `pytest` can run [a specific function](https://docs.pytest.org/en/latest/usage.html)):

```bash
git bisect run pytest -qqq my_script.py::test_product
....
....
....
d2a3327df55cb9bf9b43780f9adb223e93ba093d is the first bad commit
commit d2a3327df55cb9bf9b43780f9adb223e93ba093d
Author: oren <----@----.--->
Date:   Mon Jun 24 12:50:58 2019 -0700

    Some innocent change that certainly did not brake anything

:100644 100644 0be2cd4c2d0e450026507ee86e058785de56cadc 98c1b14d74c4f46b538df5cc1d32b9bb42afc791 M	my_script.py
bisect run success
```

Cool!

And indeed, this is the bad commit:
```bash
$ git show d2a3327df55cb9bf9b43780f9adb223e93ba093d
commit d2a3327df55cb9bf9b43780f9adb223e93ba093d
Author: oren <----@----.--->
Date:   Mon Jun 24 12:50:58 2019 -0700

    Some innocent change that certainly did not brake anything
.......
.......
 
 def product(factors):
-    return reduce(mul, factors)
+    return reduce(lambda x , y: x - y, factors)
```

Note: if you are using a script as a command to `git bisect run`. Make sure this script is not tracked by git, otherwise it might change when git changes revisions.


Cheers!