Static Analysis Tooling: how three fancy words are my new best friends
Tags: opensource, git, github, python
Published: 2020-11-09 01:56:33

This week in my OSD600 adventures, we're learning about Static Anaylsis Tooling: Three fancy words which basically equates to the smart kid in the class (*~~who you secretly hate~~*) that proof reads your poorly written paper and points out more errors than you'd like.

We don't really speak about this kid anymore. Does it really matter that Marvin now works as a software engineer for a FAANG? No, because Marvin is getting paid six figures to write quality code, drive a Tesla, put a down payment on a house, but **you** get to work with linting software, and quality sunday lectures.

___

Static Analysis Tools are pieces of software that are used in Static Code Analysis, a process of determing if the code you're about push into a dumpster is pizza crust, or rotten pizza crust. `🍕🗑️🔥`

For this week's lab I had to implement two tools my ever evolving [link checker project](https://github.com/chrispinkney/He-s-Dead-Jim): a code formatter called [Black](https://github.com/psf/black), and a code linter called [Flake8](https://gitlab.com/pycqa/flake8).

Black formats your code in such a way that makes it more readable and easier to understand. It turns your spaghetti into penne. Your mess into order. It's great.

Flake8 looks through your code and scans it for suspicious lines of code. These lines might be stylistic errors that may end up as a bug down the road.

Basically, you can think of Black as a spell checker, and Flake8 as a grammar checker.

Once I chose my tools it was time to RTFM and set them up in my project.

*Note: I am using Windows and thus these programs and set up files will vary based on your OS.*

___

The first step was to install and set up Black: `pip install git+git://github.com/psf/black`

> Since my project uses a `setup.py` file, I also added Black to the `install_requires` portion so that anyone who wants to set up the project simply types `pip install .`

The next step is testing: `black sample/hdj_util.py` which amazing *just worked*, a common phase I mutter creepily to myself when working with Python. Until things don't. Then I mutter profanities. It automatically prett-ified my `hdj_util.py` file.

![](https://imgur.com/8JDcjNN.gif)

Afterwards I set up PyCharm to run black on my project upon every save to whatever file I'm currently working on:

1. File -> Settings -> Tools -> External Tools
*Click the + icon to add a new external tool and add the following values:*
```
Name: Black
Description: Black is the uncompromising Python code formatter.
Program: C:\Users\Chris\AppData\Local\Programs\Python\Python39\Scripts
Arguments: $FilePath$
```

Note the `Program:` path will obviously be different based on your machine. Simply locate the `black.exe` file on your system and point it to that.

From here you can go to `Tools -> External Tools -> black` to run black on the currently open file, or you can configure PyCharm to run Black on every file save. This requires the [File Watchers plugin](https://plugins.jetbrains.com/plugin/7177-file-watchers) to work.

2. Go to Preferences or Settings -> Tools -> File Watchers and click + to add a new custom watcher:

```
Name: Black
File type: Python
Scope: Project Files
Program: C:\Users\Chris\AppData\Local\Programs\Python\Python39\Scripts\black.exe
Arguments: $FilePath$
Output paths to refresh: $FilePath$
Working directory: $ProjectFileDir$

Uncheck “Auto-save edited files to trigger the watcher” in Advanced Options
```

3. Create a `pyproject.toml` file to configure my project's black:
```
[tool.black]
line-length = 88
include = '\.pyi?$'
exclude = '''
/(
    \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''
```

`line-length`: Formats lines over a length of 88 characters to a prettier version. You can see this occur in the gif above.
`include` runs black on only `.py` files (thus excluding files like `.gitignore`.)
`exclude` is pretty self explanatory: "Don't run black on these folders or files".

___

With Black set up, let's move onto our next step and set up Flake8: `python -m pip install flake8`.

> Again, my project uses a `setup.py` file, So I also added Flake8 to the `install_requires` portion.

The next step is testing: `flake8 sample` (*sample is the directory which contains my code*) which yet again *just worked*... albeit maybe too well... **grumble grumble**

![](https://i.imgur.com/b1MOYs5.png)

The process to get this to run on save took a bit of google-fu but I managed to set it up.

We must simply add a file watcher just like above:

1. Go to Preferences or Settings -> Tools -> File Watchers and click + to add a new custom watcher:

```
Name: Flake8
File type: Python
Scope: Current File
Program: C:\Users\Chris\AppData\Local\Programs\Python\Python39\Scripts\flake8.exe
Arguments: $FileDir$/$FileName$

Check “Auto-save edited files to trigger the watcher” in Advanced Options
Output filters: $FILE_PATH$:$LINE$:$COLUMN$: $MESSAGE$
```

2. Go to Preferences or Settings -> Editor -> Inspections -> File Watchers -> File watcher problems, and change the severity to Warning.

Now Flake8 will auto run and highlight linting errors in yellow!

This is either wonderful or terrible depending entirely on your viewpoint, or however many glasses of wine you've had that evening, which for this author is, respectively, terrible and many.

3. Create a `.flake8` file to configure my project's flake8:
```
[flake8]
ignore=E501, W503
exclude =
    .idea,
    .git,
    __pycache__,
    assets,
    build,
    dist,
    docs

```

`ignore=E501, W503`: E501 simply ignores the `line too long` warning, and W503 ignores the `Line break occurred before a binary operator` rule. Because I'm anal but not *that* anal.
`exclude` again, is pretty self explanatory: "Don't run flake8 on these folders or files".

Great! We're all done.

___

Between the two tools neither of them find *too* many issues. Black altered all four Python files in my project, which was expected (pretend this says _changed_):

![](https://i.imgur.com/CCla9ss.png)

...and Flake8 screamed about... well, a lot of things. But it's fine. These were just either minor things, or things that [I wanted to fix up anyway](https://github.com/chrispinkney/He-s-Dead-Jim/issues/12).

![](https://i.imgur.com/b1MOYs5.png)

All in all, I didn't have too much to fix up. It's also pretty cool that with the code formatter and linter installed globally I can run either of them on the CLI and just specify a directory or file to run them on. Truly batteries included.

**_slaps monitor_** Now it's time to get these bad boys to run upon every `commit`, because suffering truly builds character. Or software.

___

Git hooks are awesome. They really are. Imagine forcing people to run a script before they can even commit something to your project. Amazing. I bet there's a lot of funny gimmick hooks out there that just insult the user upon a commit... **_writes down_**.

It was suprisingly easy to set up a git hook to run these programs upon commit, it just required a bit of reading. We're going to leverage something called a *pre-commit framework* to do this. It's also configured with `YAML`, which is pretty cool. It's yet another mark up language that I've never used:

1. Install pre-commit: pip install pre-commit .`
2. Added `pre-commit` to my `setup.py`.
3. Create `.pre-commit-config.yaml` with the hooks needed in the pipeline:
```
repos:
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
      language_version: python3.9
-   repo: https://gitlab.com/pycqa/flake8
    rev: 3.8.4
    hooks:
    -   id: flake8
```
4. Run `pre-commit install` which installs git hooks in .git/ directory of my project.

I mostly had to dig around for the proper repos and versions to use. I thought I'd have to config this some more because I'm using the new version of Python but surprisingly I didn't have to.

![](https://i.imgur.com/IVgsw0D.gif)

Neat.

___

I learned a LOT in this lab, and just as I predicted the tooling labs so far are really interesting and fun.

I also unfortunately learned about another tool called [Pylama](https://pylama.readthedocs.io/en/latest/) (*What is it with the Python community and animals?*) which is **both** a code formatter and a linter! I'll have to keep it in mind for my next big project.

I think we're doing CI tools next week! It's gonna be great, I can't wait.

The struggled wasn't too bad, but it's all worth it for that sweet, sweet fast forward `rebase` and `merge`.

![](https://i.imgur.com/9y4ei42.gif)