Flask / Quart - manage module loading and splitting
Tags: python, flask, quart
Published: 2022-06-28 18:02:50

Working with Flask or the fabulous Quart in Python, you sometime want to "split" your routes (or views) in several packages. That's something very common in Go, but a bit more complex with Python. I've got something to propose.

The reason is how Python defined a package. With Go, a package is a set of files that are all injected once. But with Python, it's a set of module that are independent.

So, the time has come when you begin to use `Blueprints`. A very nice way to write sub-routes that you can classify, prefix and activate.

> That's ok, let's see the problem.

> Note, I use [Quart](https://pypi.org/project/quart/) but you can change the entire examples to use Flask (remove `async` and change imports)


## Classification of routes

Let's imagine we want to create this application structure:
```
app.py
api/
    __init__.py
    auth.py
    content.py
```

The `app.py` will load the `api` package to register it as a `Blueprint`. That's common, and we will not touch this file anymore. So, write the `app.py` file like this:

```python
""" The main application """

from quart import Quart

from api import api

app = Quart(__name__)
app.register_blueprint(api, url_prefix="/api")

if __name__ == "__main__":
    app.run(debug=True)

```

## Manage the routes in modules

Now, let's take a look at the `routes` package. And see the problem happening.
First, declare a Blueprint in `api/__init__.py`

```python
# api/__init__.py
from quart import Blueprint

api = Blueprint('api', __name__)
```
The `api` variable is global and can be imported inside any modules.
That's not a bad idea.

```python
# in api/auth.py
from . import api

@api.route("/login")
async def login():
    return "Welcome"

@api.route("/logout")
async def logout():
    return "Bye !"
```

And...

```python
# in api/content.py
from . import api

@api.route("/")
async def index():
   return "The content !"
```

Back to `__init__.py`, how to import the modules?

There are many difficulties...

- You cannot import `auth` and `content` modules before the `api` declaration, because the module is not yet initialized (keep in mind that `api` should be initialized to use `@api.route` decorator...
- if you put imports after the `api` declaration, the linter will complain about the placement of imports
- and moreover, if you import modules without any use of them, so the linter will complain (and it is right, it's bad, ugly)

So, let's make a "programmatic import"! A good idea, isn't it?

```python
# in api/__init__.py

from quart import Blueprint

api = Blueprint('api', __name__)

for module in ["auth", "content"]:
    __import__(f"{__name__}.{module}")
```

> This is not that bad actually, everything is working as expected.

Yes, but... What if we forget a module or if we make a mistake in the name? And what if I move, rename or delete a module ?

## Make it a few more automatic

We need to detect the modules inside the package. And, believe it or not, that's not so easy. We could, for example, use `os.walk` or `glob.glob` to find files. But there is a complication: a module can be "compiled" as a library (`.so` file), and there are many other extensions to detect. By chance, we have a solution.

There is a `__loader__` inside each package. A lot of (even good) Python developers ignore this, and that's a pity. Because this variable contains a very useful `SourceFileLoader` that provides a `FileReader`. This is a generator that gives a solution to retrieve the **modules** inside the package.

> Ho !

So! It's time to make a nice and tricky loop!

```python
# app/__init__.py
from importlib import import_module
from pathlib import Path

from quart import Blueprint

api = Blueprint('api', __name__)

for mod in __loader__.get_resource_reader().contents():
    mod = Path(mod).stem
    if mod == "__init__":
        continue
    import_module(__package__ + "." + mod, __package__)

```

But... one more time, a problem may happen. 

> What if another file than a Python source file (or precompiled file) exists in the module ? For example, a `README.md` file.

There are plenty of ways to check this, for example having a list of possible extensions.

> But, are you sure to know the entire list of Python resource extensions?
> I mean, you think about `.py` and `.pyc` - but do you remember that the `.pyo` extention exists? And are they other forgotten names?

So, what I love to do is to guess the mimetype.

```python
# app/__init__.py
import mimetypes
from importlib import import_module
from pathlib import Path

from quart import Blueprint

api = Blueprint('api', __name__)

for mod in __loader__.get_resource_reader().contents():
    if "python" not in str(mimetypes.guess_type(mod)[0]):
        continue

    mod = Path(mod).stem
    if mod == "__init__":
        continue

    import_module(__package__ + "." + mod, __package__)
```

And there we are!

## Time to make it a "tool"

OK, copying this loop inside each package `__init__` file is a waste of time, and it is not comfortable if we need to fix the behavior in case of bug.

> Anyway, when you need to copy a portion of code, so it should be a function.

Note, the `__loader__` is a `SourceFileLoader` which is a bit tricky to find in modules, and not easy at all to use. So, I prefer to make a "representation" of the class I need.

It's time to create a "tool". Create a "utils" package at the root of the project, and make the `__init__.py` file:

```python
# utils/__init__.py

import mimetypes
from importlib import import_module
from pathlib import Path
from typing import Callable


class SourceFileLoader:
    """ Represents a SouceFileLoader (__loader__)"""
    name: str
    get_resource_reader: Callable

def load_modules(loader: SourceFileLoader):
    """Load the entire modules from a SourceFileLoader (__loader__)"""
    pkg = loader.name
    for l in loader.get_resource_reader().contents():

        if "python" not in str(mimetypes.guess_type(l)[0]):
            continue

        mod = Path(l).stem
        if mod == "__init__":
            continue

        import_module(pkg + "." + mod, pkg)
```

In the `api` and others packages, I can now do:

```python
# api/__init__.py

from quart import Blueprint
from utils import load_modules

api = Blueprint("api", __name__)

load_modules(__loader__)
```

And now, that's fantastic! Our all modules are now injected from the package. I don't forget any file, everything is automated by a simple call to `load_modules()`.

## I know...

Yes, Python is an awesome language, easy to use and Flask or Quart are wonderful. But, as you can see, sometimes you need to make some tricky things to make your project more concise.

Here, to make this loader tool, we needed to know some deep knowledge of the language. But that's also something I really love when I use a technology: not to be happy of the basics. I like to use `pdb` to observe and discover what's behind the scene. Using `dir()` on a object, checking types...

So, this loader is a little example of what we can do to automate development and to make less code in our project.

I hope you enjoyed my article and that may help you :smile: