Improving Python code incrementally
Tags: python, pytest, black
Published: 2020-10-11 15:18:41

---
title: Improving Python code incrementally
published: true
description: How to use Darker to reformat only modified regions in Python code 
tags: python, pytest, black
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/lfsmf8dxft7h1cka4y9w.jpg
---
_Photo: [Matěj Baťha][photographer] / [CC-BY-SA 2.5][cc-by-sa]_

---

Many of us have adopted tools to improve code quality, like
- code formatters,
- linters, and
- test coverage measurement.

For Python, such tools include for example:
- [Black][black]     (code formatting)
- [isort][isort]     (import sorting & formatting)
- [Pylint][pylint]   (code quality)
- [Flake8][f8]       (code style)
- [Mypy][mypy]       (static typing)
- [coverage.py][cov] (test coverage)

For new and small projects, it's simple to enforce such automatic code modifications and quality checks for the whole code base. It can be done locally with Git hooks, as well as in a CI pipeline.

But if you have a large existing code base, or are contributing to an Open Source project, it may not be feasible to turn all checks green in one go.

One good reason to not reformat a code base in one go is that authorship information is lost for all reformatted lines, which may make collaboration on bugs and features trickier from then on. While [git --ignore-rev][ignore-rev] offers one solution for this problem, this blog post presents another alternative.

# Introducing [Darker][drkr] – apply [Black][black] formatting gradually

The idea behind [Darker][drkr] is to reformat code using [Black][black] (and optionally [isort][isort]), but only apply new formatting to regions which have been modified by the developer.

By applying only changes suggested by Darker, the codebase will gradually converge towards a point where all code is formatted according to [Black][black] rules.

[Darker][drkr] can also run quality checks. The violations are filtered so you only see ones touching newly modified regions.

Finally, the [pytest-darker][pt-drkr] plugin integrates all this with [Pytest][pytest] so reformatting and checks are run along with unit tests.

# I'm sold, how to use [Darker][drkr]?

In the following example, I'll show first what [Darker][drkr] would have done in a real-world situation for a popular Open Source project when used manually on the command line.

I'll also show how to integrate it with the test suite using the [pytest-darker][pt-drkr] plugin.

# Case study: dateutil

Let's first clone the [dateutil][dateutil] package:
```bash
$ git clone https://github.com/dateutil/dateutil.git
$ cd dateutil
```

In January 2020, there was a tiny feature branch which we can use as a guinea pig for [Darker][drkr] (see [dateutil/dateutil#993][dateutil#993]):
```bash
$ export GIT_PAGER=cat

$ git log --graph --oneline a332879..6edfecd
* 6edfecd Update documentation links
*   110a09b Merge pull request #993 from pganssle/parsererror_repr
|\  
| * ea7f441 Fix custom repr for ParserError
|/  
* 21fe6e9 Merge pull request #987 from eastface/issue_981
```

We can now recreate the situation before the [`pganssle/parsererror_repr`][parsererror_repr] branch was merged by resetting `master` back to before the branch...
```bash
$ git reset --hard 21fe6e9
HEAD is now at 21fe6e9 Merge pull request #987 from eastface/issue_981
```
...and recreating the [`pganssle/parsererror_repr`][parsererror_repr] branch at the point before the merge was done:
```bash
$ git checkout -b pganssle/parsererror_repr ea7f441
Switched to a new branch 'pganssle/parsererror_repr'
```

Our repository is now in a similar state as before the branch was merged:
```bash
$ git log --graph --oneline master^..
* ea7f441 (HEAD -> pganssle/parsererror_repr) Fix custom repr for ParserError
* 21fe6e9 (master) Merge pull request #987 from eastface/issue_981
* a332879 Added Mark Bailey to AUTHORS file
* 79d2e48 Fix TypeError in parser wrapper logic
```

The changes introduced by the [`pganssle/parsererror_repr`][parsererror_repr] branch are:
```diff
$ git diff -U0 master
diff --git a/changelog.d/991.bugfix.rst b/changelog.d/991.bugfix.rst
new file mode 100644
index 0000000..473082e
--- /dev/null
+++ b/changelog.d/991.bugfix.rst
@@ -0,0 +1 @@
+Fixed the custom ``repr`` for ``dateutil.parser.ParserError``, which was not defined due to an indentation error. (gh issue #991, gh pr #993)
diff --git a/dateutil/parser/_parser.py b/dateutil/parser/_parser.py
index 7fcfa54..8d67584 100644
--- a/dateutil/parser/_parser.py
+++ b/dateutil/parser/_parser.py
@@ -1603,2 +1603,3 @@ class ParserError(ValueError):
-        def __repr__(self):
-            return "%s(%s)" % (self.__class__.__name__, str(self))
+    def __repr__(self):
+        args = ", ".join("'%s'" % arg for arg in self.args)
+        return "%s(%s)" % (self.__class__.__name__, args)
diff --git a/dateutil/test/test_parser.py b/dateutil/test/test_parser.py
index 605705e..cfa4bbb 100644
--- a/dateutil/test/test_parser.py
+++ b/dateutil/test/test_parser.py
@@ -945,0 +946,9 @@ def test_decimal_error(value):
+
+def test_parsererror_repr():
+    # GH 991 — the __repr__ was not properly indented and so was never defined.
+    # This tests the current behavior of the ParserError __repr__, but the
+    # precise format is not guaranteed to be stable and may change even in
+    # minor versions. This test exists to avoid regressions.
+    s = repr(ParserError("Problem with string: %s", "2019-01-01"))
+
+    assert s == "ParserError('Problem with string: %s', '2019-01-01')"
```

On the surface, all the changes look [Black][black] compliant. If the whole code base at `master` was already [Black][black] formatted, we could simply run [Black][black] to see if we missed anything in the [`pganssle/parsererror_repr`][parsererror_repr] branch. But alas:
```bash
$ pip install -q black

$ black --diff .
### ... LOTS of output redacted ...
All done! ✨ 🍰 ✨
35 files would be reformatted, 3 files would be left unchanged.
```

Ok thanks [Black][black], but we're only interested in whether the changes in *this* branch are well formatted.

Unfortunately [Black][black] doesn't offer this functionality out of the box. Instead we'll use [Darker][drkr]:
```diff
$ pip install -q darker

$ darker --diff --revision master... .
--- dateutil/test/test_parser.py
+++ dateutil/test/test_parser.py
@@ -944,6 +944,7 @@
     with pytest.raises(ParserError):
         parse(value)
 
+
 def test_parsererror_repr():
     # GH 991 — the __repr__ was not properly indented and so was never defined.
     # This tests the current behavior of the ParserError __repr__, but the
```

Ha! So we did miss one newline before a function definition!

We can now just run [Darker][drkr] without the `--diff` flag so it actually writes that change to the source code:
```diff
$ darker --revision master... . 

$ git diff -U1
diff --git a/dateutil/test/test_parser.py b/dateutil/test/test_parser.py
index cfa4bbb..53a02de 100644
--- a/dateutil/test/test_parser.py
+++ b/dateutil/test/test_parser.py
@@ -946,2 +946,3 @@ def test_decimal_error(value):
 
+
 def test_parsererror_repr():
```

# Running [Darker][drkr] as part of the test suite

We can now install [pytest-darker][pt-drkr]:
```bash
$ pip install -q pytest-darker
```

We'll configure [Darker][drkr] so it always checks but doesn't modify source files. This is done in the `pyproject.toml` file:
```toml
[tool.darker]
src = [
    "."
]
check = true
diff = true
revision = "master..."
```

Now we can run the test suite and have it check the formatting of the feature branch as well:
```bash
$ pip install --quiet --requirement requirement-dev.txt
$ pytest --darker
====================== test session starts =======================
# ... lots of output ...
dateutil/test/test_parser.py F............................ [ 38%]
.......................................................... [ 40%]
.......................................................... [ 43%]
.......................................................... [ 46%]
...................xxxxxxxxxxxxx..........                 [ 48%]
# ... more output ...
============================ FAILURES ============================
______________________ Darker format check _______________________
Edited lines are not Black formatted
---------------------- Captured stdout call ----------------------
--- dateutil/test/test_parser.py
+++ dateutil/test/test_parser.py
@@ -944,6 +944,7 @@
     with pytest.raises(ParserError):
         parse(value)
 
+
 def test_parsererror_repr():
     # GH 991 — the __repr__ was not properly indented and so was never defined.
     # This tests the current behavior of the ParserError __repr__, but the
==================== short test summary info =====================
FAILED dateutil/test/test_parser.py::DARKER
===== 1 failed, 2012 passed, 84 skipped, 19 xfailed in 7.12s =====
```

# More goodies

I've shown how to run [Darker][drkr] on the command line as well as integrated in the test suite. In future blog posts, I plan to
- guide you to integrate [Darker][drkr] with an IDE,
- show how to have [Travis CI][travis] on GitHub run [Darker][drkr], and
- explain the mysterious `--revision master...` command line option.
- show how to sort imports and do code linting through [Darker][drkr]

# Contribute!

We're a small happy bunch of developers contributing to [Darker][drkr] on [its GitHub repository](https://github.com/akaihola/darker) – drop by, say hello and lend your hand reviewing and submitting pull requests and making bug reports! We also sometimes have [virtual meetups][meetups] spanning continents and time zones, and we'd happy to see you there as well!

[black]: https://pypi.org/project/black
[isort]: https://pypi.org/project/isort
[pylint]: https://pypi.org/project/pylint
[f8]: https://pypi.org/project/flake8
[mypy]: http://www.mypy-lang.org/
[cov]: https://pypi.org/project/coverage
[ignore-rev]: https://akrabat.com/ignoring-revisions-with-git-blame/
[drkr]: https://pypi.org/project/darker
[pt-drkr]: https://pypi.org/project/pytest-darker
[dateutil]: https://pypi.org/project/python-dateutil/
[dateutil#993]: https://github.com/dateutil/dateutil/pull/993
[parsererror_repr]: https://github.com/pganssle/dateutil/tree/parsererror_repr
[pytest]: https://docs.pytest.org/
[travis]: https://travis-ci.com/
[meetups]: https://github.com/akaihola/darker/issues/60![Rubik's_Snake_9](https://dev-to-uploads.s3.amazonaws.com/i/3z1tz0pb75jezeh7lvht.jpg)
[photographer]: https://commons.wikimedia.org/wiki/User:Jedudedek
[cc-by-sa]: https://creativecommons.org/licenses/by-sa/2.5/deed.en