Linting and Formatting a Python Project
Tags: python, productivity, learning, programming
Published: 2023-11-04 03:34:32

There are mainly two types of tools/techniques to analyze the quality of a codebase in an automated manner, ones that run on your code (**static**) and ones that test your program when its running ([dynamic](https://totalview.io/blog/what-dynamic-analysis)).

In this post, I'll be talking about the importance of [static program analysis](https://en.wikipedia.org/wiki/Static_program_analysis) and how I set up appropriate tooling for it in **Python** [project](https://github.com/Amnish04/til-page-builder/tree/master).

## What is Static Code Analysis 🕵🏻

Static code analysis is a software testing technique that analyzes source code or compiled code without executing it.

{% embed https://www.youtube.com/embed/KXdm4eQdc5s?si=Od9BjAko_LrUB_l4 %}

The primary goal of static code analysis is to find potential issues, vulnerabilities, and maintainability problems in a program's source code before it is executed or compiled. This analysis helps identify defects early in the development process, leading to more robust and maintainable software.

## Significance of such tooling 🤔

Beginners, including me, tend to overlook the **importance** of static analysis tools, and take them for granted. But its important to realize how the rise of such tools, especially **code formatters**, has revolutionized the way contributions are made to projects, exponentially boosting the development productiviy.

The following **interview** of my open source **professor**, with the **early developers** of [prettier](https://prettier.io/), does a great job at highlighting the importance of such formatters, and how **blessed** we should feel to have them at our disposal.

> [Interview with Prettier’s Christopher Chedeau](https://humphd.github.io/pretty-effective/#interview-with-prettiers-christopher-chedeau)

There are various types of such tools focussing on different areas like **code formatters**, **linters**, **security scanners** and many more. But I'll only be focusing on the first two in this post.

## Integrating Static Analysis in my Project 🐍

The first thing I had to do was **decide** what [code formatter](https://www.freecodecamp.org/news/using-prettier-and-jslint/#what-are-code-formatters) and [linter](https://www.freecodecamp.org/news/using-prettier-and-jslint/#what-are-linting-tools) I would use in my **python** project.

I researched online and chose the most popular ones I could find.
- [Black](https://black.readthedocs.io/en/stable/index.html) Code Formatter
- [Pylint](https://docs.pylint.org/intro.html) for linting

### Adding Black

[Black](https://github.com/psf/black#used-by) is a highly opinionated code formatter with little configuration options widely used by major companies and projects.

![Black Users](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/grhrft2jxl2rpflraj5h.png)

I started out by reading the [official documentation](https://black.readthedocs.io/en/stable/getting_started.html) to see steps for **getting started**, and surprisingly, it was pretty simple to use.

Run the following command format a file or files within a folder,
```
black {source_file_or_directory}...
```

However, this is not enough if you would like to setup some **custom configurations**, like excluding some files or setting a `max-line-length` of your choice.

For this, you need to add a `pyproject.toml` file at the root of your project with the following configuration to get started
```toml
[tool.black]
line-length = 88
target-version = ['py37']
include = '\.pyi?$'
# 'extend-exclude' excludes files or directories in addition to the defaults
exclude = '^/src/version.py'
extend-exclude = '''
# A regex preceded with ^/ will apply only to files and directories
# in the root of the project.
(
  ^/foo.py    # exclude a file named foo.py in the root of the project
  | .*_pb2.py  # exclude autogenerated Protocol Buffer files anywhere in the project
)
'''
```

Refer [Configuration via a file](https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#configuration-via-a-file) for more options and details.

Now that the setup was done, it was time to run the formatter on my code.

![Running Black](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t9ckmrdmt2ir4fb3g9f8.jpg)

I was surprised to see the massive formatting changes to all my files, and the entire codebase instantly become way more readable.

### Adding Pylint

From Pylint [documentation](https://docs.pylint.org/intro.html),
> Pylint is a tool that checks for errors in Python code, tries to enforce a coding standard and looks for bad code smells. This is similar but nevertheless different from what pychecker provides, especially since pychecker explicitly does not bother with coding style. The default coding style used by Pylint is close to Guido’s style guide). For more information about code smells, refer to Martin Fowler’s refactoring book.

This process was pretty similar to configuring black. The first thing I did was learn how to [invoke pylint](https://docs.pylint.org/run.html) from the commandline.
```
pylint [options] module_or_package
```

Unlike **black**, which is an opinionated tool, **pylint** is highly configurable with a whole lot of options that can be found here:
https://docs.pylint.org/features.html#pylint-global-options-and-switches

To configure these options, I had to create a `.pylintrc` file, again in the root of my project with the following command.
```
pylint --generate-rcfile > ./.pylintrc
```
This was a long file all the supported options set to their **default** values.

Should like like this

![.pylintrc](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6kt4i5lazkg9wyhpx31q.png)

Now that the configuration was setup, it was time to run the tool on my project.

![Pylint Score](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gl2f392uh0tfa4ui0g5q.jpg)

As expected, there were dozens of issues in my project even though its very young. There were **various types** of issues like missing docstrings, [dangerous default values](https://pylint.pycqa.org/en/latest/user_guide/messages/warning/dangerous-default-value.html), extra general exceptions, long lines, too few public members, and the list goes on.

One thing I did was disable any formatting related settings **black** was handling for me to **avoid conflicts**. This part is really important to prevent **unexpected behaviors**.

```ini
disable=raw-checker-failed,
        bad-inline-option,
        locally-disabled,
        file-ignored,
        suppressed-message,
        useless-suppression,
        deprecated-pragma,
        use-symbolic-message-instead,
        use-implicit-booleaness-not-comparison-to-string,
        use-implicit-booleaness-not-comparison-to-zero,

        # From here
        line-too-long,
        missing-module-docstring,
        consider-using-from-import,
        unspecified-encoding,
        import-error
```

After this, I added some missing docstrings and fixed most of the linting issues present. This took me like 3 hours since while going through the code, I moved some method across classes where they belong better, leading to a **better score**.

![After Lint Fix](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hzsehwr7ahpjj7wmirnb.jpg)

Now mostly some missing docstrings were left, which I opened created an [issue](https://github.com/Amnish04/til-page-builder/issues/21) for.

## Integrating tools in the IDE

To get the benefits of these tools while you are writing code, it is recommended to integrate these tools into your code editor as well. This gives you suggestions and you're able to fix your code while developing.

This was pretty simple with **VSCode** as all you need to do is install [black-formatter](https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter) and [pylint](https://marketplace.visualstudio.com/items?itemName=ms-python.pylint) extensions.

The corresponding **configuration** needs to be set in a `settings.json` file (workspace settings) the `.vscode` directory.

[**.vscode/settings.json**](https://github.com/Amnish04/til-page-builder/blob/master/.vscode/settings.json)
```json
{
    "[python]": {
        "editor.defaultFormatter": "ms-python.black-formatter",
        "editor.formatOnSave": true,
    },
    "editor.codeActionsOnSave": {
        "source.organizeImports.ruff": true
    },
    "pylint.ignorePatterns": [
        "version.py"
    ],
    "pylint.args": [
        "--rcfile=./.pylintrc"
    ],
}
```
This will auto-format the code whenever a file is saved by **black** code formatter, and you'll contantly receive warnings from **pylint**. These settings are based on whatever we set earlier for the commadnline versions.
For more options, feel free to explore the official documentations of these extensions.

I also added an `extensions.json` file such that any new contributors are automatically recommend to install these extensions when they first open my project.

[**.vscode/extensions.json**](https://github.com/Amnish04/til-page-builder/blob/master/.vscode/extensions.json)
```json
{
	// See https://go.microsoft.com/fwlink/?LinkId=827846 to learn about workspace recommendations.
	// Extension identifier format: ${publisher}.${name}. Example: vscode.csharp

	// List of extensions which should be recommended for users of this workspace.
	"recommendations": [
		"ms-python.pylint",
		"ms-python.black-formatter"
	],
}
```

And with that, I was all set.

## Git Hooks 🪝

Many projects also add [pre-commit git hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) to their projects that run some code or script on every commit. However, I don't want to be **too strict** with my contributors by allowing them to write in their own style, while still maintaining these standards in my codebase.
I am planning to do this by configuring a **Github Actions CI**, that does this on every **pull request**. The [issue](https://github.com/Amnish04/til-page-builder/issues/22) has already been opened, feel free to follow up if you would like to contribute. Will be adding more details soon!

## Conclusion 🌟

In this post, I discussed about the importance of static code analysis and its importance, and how to add **linting** and **code formatting** tools to a python project.

These were all the changes I had to make to accomplish this.

![Git Log](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/euyx0hmfb1n8m4ueumy3.gif)

![Changes Count](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0dsa0056q5lwo4tmq45c.jpg)

Here's a [link](https://github.com/Amnish04/til-page-builder/commit/99dc50e50e72c122260de12f5b6b2a0832b93cbe) to the **squashed commit** if you're interested.

Hope this helped!