Type Annotate an existing Python Django Codebase with MonkeyType
Tags: python, django, typehints, refactoring
Published: 2020-07-21 17:50:10

---
title: Type Annotate an existing Python Django Codebase with MonkeyType
published: true
description: Why annotate existing code manually when Python already knows what type a variable is at runtime? A walkthrough to type annotate Python code with PyTest and Monkeytype. 
tags: Python, Django, Typehints, refactoring
cover_image: https://github.com/cleder/talks/blob/main/images/monkey-typing.jpg?raw=true
---

# Why Add Types?

There has been quite a buzz around type annotations in the recent past, so you may ask yourself if type checking is something you should do. This article is not a general introduction to type annotations and type checks, I recommend [Hypermodern Python: Typing](https://cjolowicz.github.io/posts/hypermodern-python-04-typing/) to get an overview of the background. 

In a nutshell citing the above article: 
*"Type annotations are a way to annotate functions and variables with types. Combined with tooling that understands them, they can make your programs easier to understand, debug, and maintain. A static type checker can use type annotations and type inference to verify the type correctness of your program without executing it, helping you discover many bugs that would otherwise go unnoticed."* or Dropbox puts it like this: *"If you aren’t using type checking in your large-scale Python project, now is a good time to get started — nobody who has made the jump I’ve talked to has regretted it. It really makes Python a much better language for large projects."* 

In addition to the above the tooling around type annotations is growing continuously and helps to keep your code DRY.
There are libraries like [Desert](https://desert.readthedocs.io/en/latest/) or [Typical](https://python-typical.org/) to help with data validation, [Typeguard](https://github.com/agronholm/typeguard) or [strongtyping](https://github.com/FelixTheC/strongtyping) to check types at runtime, [sphinx-autodoc-typehints](https://github.com/agronholm/sphinx-autodoc-typehints) to help build documentation, [Typer](https://typer.tiangolo.com/) to build CLIs, [FastAPI](https://fastapi.tiangolo.com/) a web framework for building APIs, and even transpilers to compile your python code with [Mypy to Python C](https://github.com/python/mypy/tree/master/mypyc) or [Rust](https://medium.com/@konchunas/monkeytype-type-inference-for-transpiling-python-to-rust-64fa5a9eb966).

For a more complete list please refer to [Awesome Python Typing](https://github.com/typeddjango/awesome-python-typing)

# Do Not Let Legacy Code Become Technical Debt

As Dropbox describes in [Our journey to type checking 4 million lines of Python](https://dropbox.tech/application/our-journey-to-type-checking-4-million-lines-of-python) it is quite daunting to type annotate an existing codebase. The article describes in depth how you can approach gradual improvements with type annotations to your codebase.

# Enter MonkeyType

But why do I have to write my annotations manually? After all Python knows what type a variable is at runtime. The folks at Instagram thought the same and created MonkeyType. A similar projects is [PyAnnotate](https://github.com/dropbox/pyannotate) created by Dropbox. [Pytype](https://google.github.io/pytype/) generates files of inferred type information, located by default in `.pytype/pyi`. [Pyre](https://pyre-check.org/) has a powerful feature for migrating codebases to a typed format. The [infer](https://pyre-check.org/docs/pysa-coverage/) command-line option ingests a file or directory, makes educated guesses about the types used, and applies the annotations to the files.

[MonkeyType](https://monkeytype.readthedocs.io/en/stable/) collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type annotations directly to your Python code based on the types collected at runtime.

Have a look at the  [Introduction to MonkeyType](https://instagram-engineering.com/let-your-code-type-hint-itself-introducing-open-source-monkeytype-a855c7284881) for an overview of its functionality or listen to the [MonkeyType podcast](https://www.pythonpodcast.com/monkeytype-with-carl-meyer-and-matt-page-episode-146/).

# Setup MonkeyType For a Django Project With PyTest

While you can run MonkeyType in a production like environment and get real world data on how you code is called, the most common use case is probably to get this data out of test runs. If you do not use pytest with django [Testing Your Django App With Pytest](https://djangostars.com/blog/django-pytest-testing/) should get you started.

Install the required dependencies by creating a file `monkeytype.txt` with the contents
```
mypy
mypy-extensions
MonkeyType
pytest-monkeytype
django-stubs
djangorestframework-stubs
```
and install the dependencies with `pip install -r monkeytype.txt` As of writing this `pytest-monkeytype` needs `MonkeyType==19.11.2`, this is fixed in this [Pull Request](https://github.com/mariusvniekerk/pytest-monkeytype/pull/3)

Configure mypy with `mypy.ini` e.g.
```
[mypy]
disallow_any_generics = True
disallow_untyped_calls = True
disallow_untyped_defs = True
disallow_untyped_decorators = True
ignore_errors = False
ignore_missing_imports = True
implicit_reexport = False
strict_optional = True
strict_equality = True
no_implicit_optional = True
warn_unused_ignores = True
warn_redundant_casts = True
warn_unused_configs = True
warn_unreachable = True
warn_no_return = True
warn_return_any = True

plugins =
    mypy_django_plugin.main,
    mypy_drf_plugin.main

[mypy.plugins.django-stubs]
django_settings_module = "test_settings"

[mypy-drfdapc.test_permissions]
ignore_errors = True
```


Create a `monkeytype_config.py` file: 
```
# Standard Library
import os
from contextlib import contextmanager
from typing import Iterator

# 3rd-party
from monkeytype.config import DefaultConfig


class MonkeyConfig(DefaultConfig):
    @contextmanager
    def cli_context(self, command: str) -> Iterator[None]:
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_settings")
        import django

        django.setup()
        yield


CONFIG = MonkeyConfig()
```
Change the `DJANGO_SETTINGS_MODULE` to the one that is used in tests.
Now you can run your tests with `pytest --monkeytype-output=./monkeytype.sqlite3` and `export MT_DB_PATH=./monkeytype.sqlite3` on Bash, or `setenv MT_DB_PATH ./monkeytype.sqlite3` on C Shell. 

Check which modules MonkeyType has collected infomation for with `monkeytype list-modules`.

```
drfdapc$ monkeytype list-modules
drfdapc.permissions
drfdapc.test_permissions
```

You can then use the `monkeytype stub some.module` command to generate a stub file for a module, or apply the type annotations directly to your code with `monkeytype apply some.module`.

```
drfdapc$ monkeytype stub drfdapc.permissions
from django.core.handlers.wsgi import WSGIRequest
from unittest.mock import Mock


def allow_all(*args, **kwargs) -> bool: ...


def allow_authenticated(request: WSGIRequest, *args, **kwargs) -> bool: ...


def allow_authorized_key(
    request: WSGIRequest,
    view: Mock,
    *args,
    **kwargs
) -> bool: ...


def allow_staff(request: WSGIRequest, *args, **kwargs) -> bool: ...


def allow_superuser(request: WSGIRequest, *args, **kwargs) -> bool: ...


def deny_all(*args, **kwargs) -> bool: ...


class DABasePermission:
    def has_object_permission(
        self,
        request: WSGIRequest,
        view: None,
        obj: Mock
    ) -> bool: ...
    def has_permission(self, request: WSGIRequest, view: None) -> bool: ...


class DACrudBasePermission:
    def has_object_permission(
        self,
        request: WSGIRequest,
        view: None,
        obj: Mock
    ) -> bool: ...
    def has_permission(self, request: WSGIRequest, view: None) -> bool: ...


class DARWBasePermission:
    def has_object_permission(
        self,
        request: WSGIRequest,
        view: None,
        obj: Mock
    ) -> bool: ...
    def has_permission(self, request: WSGIRequest, view: None) -> bool: ...
```

Checking the annotations with mypy shows that there is still some work to do and it demonstrates nicely which errors mypy catches.
```
drfdapc$ mypy drfdapc/permissions.py
...
drfdapc/permissions.py:71: error: Function is missing a type annotation for one or more arguments
drfdapc/permissions.py:86: error: Untyped decorator makes function "allow_superuser" untyped
...
drfdapc/permissions.py:162: error: Argument 2 of "has_permission" is incompatible with supertype "BasePermission"; supertype defines the argument type as "View"
drfdapc/permissions.py:162: note: This violates the Liskov substitution principle
drfdapc/permissions.py:162: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
...
Found 17 errors in 1 file (checked 1 source file)
```

Remember that MonkeyType’s annotations are an informative first draft, to be checked and corrected by a developer.
In the above example we never pass a view, all objects are mocked and `WSGIRequest` is too specific, so we need to adjust this. Also run `black` and `isort` on this file to have a nicely formatted code. You can see the workflow in [this pull request](https://github.com/PrimarySite/drf-deny-allow-pc/pull/11)

Start with small modules that have few (or even better no) dependencies on other parts of your code, apply the types, check the correctness of the annotations with `mypy some/module.py` fix the typing information, and move on to the next module.

# Little Helpers

You will want to keep your type information clean, readable and consistent. To achieve this there are some plugins for flake8.

[flake8-annotations-complexity](https://pypi.org/project/flake8-annotations-complexity/) reports on too complex, hard to read type annotations. Complex type annotations often means bad annotation usage, wrong code decomposition or improper data structure choice. 

[flake8-annotations-coverage](https://pypi.org/project/flake8-annotations-coverage/) reports on files with a lot of code without type annotations. This is mostly useful when you add type annotations to existing large codebase and want to know if new code in annotated modules is annotated.

[flake8-type-annotations](https://pypi.org/project/flake8-type-annotations/) is used to validate type annotations syntax as it was originally proposed 

[flake8-annotations](https://pypi.org/project/flake8-annotations/) detects the absence of function annotations.
What this won't do: Check variable annotations respect stub files, or replace mypy.