Writing hooks for "pre-commit" framework
Tags: python, git, precommit
Published: 2019-06-18 04:55:26

There are already hundreds of great [hooks](https://pre-commit.com/hooks.html) around. But what if we want one more? Well, reading [official docs](https://pre-commit.com/#new-hooks) is the great place to start (and reference in the future).

What I want is to:
- show by example how particular idea can be developed
- describe pitfalls which surprised me the most (and how to overcome them)
- bring attention to such a wonderful tool that `pre-commit` is

The theme I chose for illustration is assignment expressions (PEP572). And down the line I will create three different hooks for that.


## Preparations

For experiments with hooks already existing in the wild you would need only one git repository. And then you would configure it to install/use hooks from somewhere else.

But to write your own, there should be two separate repositories.

Let's begin.

- If you didn't install *pre-commit* into system/environment - see [documentation](https://pre-commit.com/#install) on how to do that.

- Start with basic structure like this (two folders with git init)
```
[~/code/temp/try-pre-commit]
$ tree -a -L 2
.
├── code
│   └── .git
└── hooks
    └── .git
```

- Create dummy *hooks/.pre-commit-hooks.yaml*
```
$ touch hooks/.pre-commit-hooks.yaml
```

- Create dummy *code/experiments.txt*
```
$ touch code/experiments.txt
```

- cd into *code* folder and initialize *pre-commit* (don't let this second install confuse you, it is different from installation into system/environment. It's scope is repository-wide, one git hook file, actually)
```
$ cd code
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
```

Here is how it should look now
```
[~/code/temp/try-pre-commit]
$ tree -a -L 2
.
├── code
│   ├── experiments.txt
│   └── .git
└── hooks
    ├── .git
    └── .pre-commit-hooks.yaml
```


## Example 1: grep

The first hook will be very basic, leveraging existing possibilities of the framework. I will call it "walrus-grep".

Update *hooks/.pre-commit-hooks.yaml*
```yaml
- id: walrus-grep
  name: walrus-grep
  description: Warn about assignment expressions
  entry: ":="
  language: pygrep
```

*pygrep* is a cross-platform version of *grep*. 

*entry* takes regexp of something that you don't want to commit. In my case I used ":="-pattern without actual regexp magic or escaping symbols.

Update *code/experiments.txt*
```
text file
a := 1
```

The most convenient way to experiment is with `try-repo` command. It allows to test hooks without commiting *.pre-commit-hooks.yaml* every time, and without pinning to revision later on in *.pre-commit-config.yaml*.

We can either stage files each time before running checks or use `try-repo` command with `--all-files` flag.
```
[master][~/code/temp/try-pre-commit/code]
$ pre-commit try-repo ../hooks walrus-grep --verbose --all-files
```


![image1-no-head](https://thepracticaldev.s3.amazonaws.com/i/eki9gykp0uxgi4o5cn9h.png)

This happened because hooks repo doesn't have a single commit - there is nothing HEAD could  point to. Commit *.pre-commit-hooks.yaml* and repeat.

![image2-no-tracked-files](https://thepracticaldev.s3.amazonaws.com/i/dretzgniixfy34cnlbz6.png)

See "*(no files to check)*"? Often this would be true, checks could have pre-defined limits e.g. file extension, path, ignore options. But we have our *experiments.txt* file and do not expect any of those!

**Gotcha:** `--all-files` flag is more like "all **tracked**, not only staged files". So either way you should add files to git before running checks.


![image3-first-failed](https://thepracticaldev.s3.amazonaws.com/i/ak0vgvvmi7p30wbeuk2m.png)

At last! "Failed" is what we needed to prove that grep hook catches ":=" inside some random staged file. So here failure is a good thing.

But to think about it situation is far from ideal:
- first, we don't actually want to catch ":=" inside text files, only in python code
- second, what if ":=" happens inside comments, docstrings, strings?

For "python code only" part there is `types: [python]` option which I will add to *.pre-commit-hooks.yaml*
```yaml
- id: walrus-grep
  ...
  types: [python]
```

![image4-skip-text-files](https://thepracticaldev.s3.amazonaws.com/i/88lgfr687d0nv21chrpz.png)

We see "*(no files to check)Skipped*" again, but now it is expected.

To double-check
```
$ mv experiments.txt experiments.py
$ git add experiments.py
$ pre-commit try-repo ../hooks walrus-grep --verbose --all-files
```

and we have our beloved failure back.

As for the second concern, ":=" inside comments or docstrings, we would require something different.


## Example 2: AST

The second hook will be analyzing syntax tree to find exact node type. I will call it "walrus-ast".

**Disclaimer:** This one will work only in Python 3.8 environment (beta1 is currently available for testing).

Update *hooks/.pre-commit-hooks.yaml*
```yaml
...
- id: walrus-ast
  name: walrus-ast
  description: Warn about assignment expressions
  entry: walrus-ast
  language: python
  types: [python]
```

and *experiments.py*
```python
# .py file with `:=` (Python3.8 syntax)
(x := 123)
```

The hook itself will be at *hooks/walrus_ast.py*
```python
import argparse
import ast


def check(filename):
    with open(filename) as f:
        contents = f.read()

    try:
        tree = ast.parse(contents)
    except SyntaxError:
        print('SyntaxError, continue')
        return

    for node in ast.walk(tree):
        if isinstance(node, ast.NamedExpr):
            return node.lineno


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('filenames', nargs='*')
    args = parser.parse_args()

    for filename in args.filenames:
        lineno = check(filename)
        if lineno:
            print(f"{filename}:{lineno}")
            return 1

    return 0


if __name__ == '__main__':
    exit(main())
```

What we can or can't do here? The rules per documentation are simple

> The hook must exit nonzero on failure or modify files in the working directory

Things to note:
- `exit()` with some code ("0" is when everything OK) -  will decide whether hook fails or not
- checks like `if __name__ == '__main__':` and names like `main()` or `check()` are not required (i.e. you can have useless file with `exit(1)` as its only contents)
- parsing filenames is needed if you want to be able to limit by filename. Not required but nice to have

Now run `try-repo` with new hook name
```
$ pre-commit try-repo ../hooks walrus-ast --verbose --all-files
```
![image5-directory-not-installable](https://thepracticaldev.s3.amazonaws.com/i/xkm10jkhp9qkdhv2wk2v.png)

*"Directory '.' is not installable".* What happened?

It turns out that (in case of python scripts) we should describe how to install it as usual, with *setup.py* or *pyproject.toml*.

Create *hooks/setup.py*
```python
from setuptools import setup
setup(
    name="hooks",
    py_modules=["walrus_ast",],
    entry_points={
        'console_scripts': [
            "walrus-ast=walrus_ast:main",
        ],
    },
)
```

**Note:** instead of `py_modules` you can use something like `packages=find_packages(".")` or anything else you are used to do in *setup.py*

Track both new files in git
```
$ git add setup.py
$ git add walrus_ast.py
```

Now working directory should look like this
```
[master][~/code/temp/try-pre-commit/hooks]
$ tree -a -L 1
.
├── .git
├── .pre-commit-hooks.yaml
├── setup.py
└── walrus_ast.py
```

Try running it again
```
$ pre-commit try-repo ../hooks walrus-ast --verbose --all-files
```

![image6-failed-as-expected](https://thepracticaldev.s3.amazonaws.com/i/31diiyfj2fcpe24czzs3.png)

Reminder on how *experiments.py* looks:
```python
# .py file with `:=` (Python3.8 syntax)
(x := 123)
```

The ":=" is indeed on the second line, and the one from comments is not reported. Thanks, AST!

**Note:** And if you forgot that this is Python3.8 only thing - you'll get (hopefully) some indication. In that case hook doesn't work, but shouldn't stop others from doing their job either.

![image7-syntax-error](https://thepracticaldev.s3.amazonaws.com/i/8owrlgarvrdfkf5rolkz.png)


## Example 3: Less negativity, more opportunity!

It was not my intention to stop you from using assignment expressions, rather to play with new things and describe the process.

Let's approach the idea from the opposite direction and find a place where walrus can fit. Thanks to [this tweet](https://twitter.com/hchasestevens/status/1015237280481665025) and very interesting [library](https://github.com/hchasestevens/astpath) by Chase Stevens we can find such places.

Create *hooks/walrus_opportunity.py*
```python
import astpath

def main():
    search_path = "//Assign[targets/Name/@id = following-sibling::*[1][name(.) = 'If']/test/Name/@id]"
    found = astpath.search('.', search_path, print_matches=True, after_context=1)
    return len(found)

if __name__ == "__main__":
    exit(main())
```

Here *astpath* uses *xpath* to find AST nodes.

Update *setup.py*
```python
from setuptools import setup

setup(
    name="hooks",
    py_modules=["walrus_ast", "walrus_opportunity"],
    entry_points={
        'console_scripts': [
            "walrus-ast=walrus_ast:main",
            "walrus-opportunity=walrus_opportunity:main"
        ],
    },
    install_requires=[
        # Dep. for `walrus-opportunity`
        "astpath[xpath]",
    ],
)
```

Update *hooks/.pre-commit-hooks.yaml*
```yaml
- id: walrus-opportunity
  name: walrus-opportunity
  description: Warn if you could have used ":=" somewhere
  entry: walrus-opportunity
  language: python
  types: [python]
```

Update *code/experiments.py*
```python
# .py file with `:=` (Python3.8 syntax)
(x := 123)

# missed chance to use `:=`
x = calculate()
if x:
    print(f"found {x}")
```

and run it
```
$ pre-commit try-repo ../hooks walrus-opportunity --verbose --all-files
```
![image8-failed-opportunity](https://thepracticaldev.s3.amazonaws.com/i/3pxiw8pxufflbol6wlw5.png)



## Wrapping up

If you decide that local development is done:
Push *hooks* folder to *your-repo* (e.g. on *github*), create *code/.pre-commit-config.yaml*
```yaml
repos:
- repo: https://github.com/<your-repo>/pre-commit-hooks
  rev: <commit-sha> or <tag>
  hooks:
  - id: walrus-grep
  - id: walrus-ast
  - id: walrus-opportunity
```

And now you should be able to use git from *code* repo as usual: change files, stage and commit them. Before each commit git will run a script from *code/.git/hooks/pre-commit*. And *pre-commit* framework should show report and  allow commit if everything is OK, or show report and abort commit if some checks have returned non-zero or files were changed.

## Addendum

More about AST visitors and working on the tree [here](https://greentreesnakes.readthedocs.io/en/latest/manipulating.html#working-on-the-tree).
Thanks to Anthony Sottile for such a wonderful framework.