Experimentations on Bazel: Python (1), FastAPI
Tags: bazel, python, fastapi
Published: 2021-04-18 16:53:38


## Goals

Setup a python project with:
- split the code into packages (eg web, services, models,...)
- use [FastAPI](https://fastapi.tiangolo.com/) as web framework to handle http request
- use pytest for test the code
- format the code with [black](https://github.com/psf/black)
- check/audit the code with [mypy](https://github.com/python/mypy) and other linters (managed by [pylama] 
- enforce the python version use to build, check,...
- integration with an IDE/Editor (VSCode in my case)


## Steps

### Setup python rules

Follow instructions from [bazelbuild/rules_python](https://github.com/bazelbuild/rules_python)

Add into `WORKSPACE.bazel`

```python
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")


#------------------------------------------------------------------------------
# Python
#------------------------------------------------------------------------------

# enable python rules
http_archive(
    name = "rules_python",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.2.0/rules_python-0.2.0.tar.gz",
    sha256 = "778197e26c5fbeb07ac2a2c5ae405b30f6cb7ad1f5510ea6fdac03bded96cc6f",
)
```

> :information_source: In every starlark source file (the language used for *.bazel, *.bzl), to be able to use a function, you should start by using `load(<from>, <function_a>, <function_b>,...)`, it's like `import` or `use` in other programming language.

To prepare usage of external dependencies (like fastapi, pytest,...), create the file `third_party/requirements.txt`.

```sh
mkdir third_party
touch third_party/BUILD.bazel
cat >third_party/requirements.txt <<EOF
# list externals dependencies available for every python packages
EOF
```

Update `WORKSPACE.bazel` to create the repo with externals dependencies for every python packages.

```python
# Create a central repo that knows about the dependencies needed for
# requirements.txt.
load("@rules_python//python:pip.bzl", "pip_install")

pip_install(
    name = "my_python_deps",
    requirements = "//third_party:requirements.txt",
)
```

But this setup use the python installed on your local environment, and we want to enforce the python version that will be used. I found 2 alternatives:
- building python from source as describe in [Hermetic Python with Bazel | The Thoughtful Koala](https://thethoughtfulkoala.com/posts/2020/05/16/bazel-hermetic-python.html)
- setup some pyenv via [digital-plumbers-union/rules_pyenv](https://github.com/digital-plumbers-union/rules_pyenv)

Both have pros & cons, because we'll use pyenv for integration with IDE/Editor, go for the second (it's main downside is to install 2 versions of python, because python rules should know interpreter for python 2.x and 3.x). So update `WORKSPACE.bazel` with

```python
# use pyenv to enforce python version
http_archive(
    name = "dpu_rules_pyenv",
    sha256 = "d057168a757efa74e6345edd4776a1c0f38134c2d48eea4f3ef4783e1ea2cb0f",
    strip_prefix = "rules_pyenv-0.1.4",
    urls = ["https://github.com/digital-plumbers-union/rules_pyenv/archive/v0.1.4.tar.gz"],
)

load("@dpu_rules_pyenv//pyenv:defs.bzl", "pyenv_install")

pyenv_install(
    hermetic = False,
    py2 = "2.7.18",
    py3 = "3.9.2",
)
```

### Add FastAPI code

Create the file `exp_python/webapp/main.py` with

```python
from fastapi import FastAPI

app = FastAPI()


@app.get("/status")
def read_root():
    return {"status": "UP", "version": "0.1.0"}
```

Create the file `exp_python/webapp/BUILD.bazel` with

```
load("@rules_python//python:defs.bzl", "py_library")
load("@my_python_deps//:requirements.bzl", "requirement")

py_library(
    name = "webapp",
    srcs = ["main.py"],
    srcs_version = "PY3",
    deps = [requirement("fastapi")],
)
```

If you try to build now `bazel build //exp_python/webapp`, you will have an error with:

```sh
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/BUILD.bazel:4:11: no such package '@my_python_deps//pypi__fastapi': BUILD file not found in directory 'pypi__fastapi' of external repository @my_python_deps. Add a BUILD file to a directory to mark it as a package. and referenced by '//exp_python/webapp:webapp'
```

Because `requirement("fastapi")` is not defined, to fix this, update `third_party/requirements.txt`change

```txt
# list externals dependencies available for every python packages
fastapi==0.63.0
```
### Test webapp

Add `exp_python/webapp/test.py`, with a code similar to the FastAPI guide, but with a `assert True == False` at the end, so the test will failed (because no error could also mean that test are not launch, especially when you're setup the test flow).

```python
from fastapi.testclient import TestClient

from .main import app

client = TestClient(app)


def test_read_main():
    response = client.get("/status")
    assert response.status_code == 200
    assert response.json() == {"status": "UP", "version": "0.1.0"}
    assert True == False
```

And modify `BUILD.bazel` to use `py_test`

```python
load("@rules_python//python:defs.bzl", "py_library", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

...

py_test(
    name = "test",
    srcs = [
        "test.py",
    ],
    # main = "test.py",
    python_version = "PY3",
    srcs_version = "PY3",
)
```

Calling `bazel test //exp_python/webapp:test` failed but said to look into a file under bazel-out to see the log. It's not convenient, so configure bazel to output error to console, by adding into `.bazelrc` (see previous article) the default cli option for test to display errors on stdout

```txt
test --test_output=errors
```

Now we can see the error on  stdout: `No module named 'fastapi'`. In fact to be able to run test, we should add as dependencies for the test:

- `:webapp` to be able to access the SUT (system under test)
- `fastapi` because test import it (in fact it is transitively available through `:webapp`)
- `requests`, maybe it's a bug; it's a dependencies of fastapi > starlette for testing but it doesn't seems to be transitively available.
- `pytest` because without, the test was not launch (always green)

```python
py_test(
    name = "test",
    srcs = [
        "test.py",
    ],
    # main = "test.py",
    args = [
        "--capture=no",
    ],
    python_version = "PY3",
    srcs_version = "PY3",
    deps = [
        ":webapp",
        requirement("requests"),
        requirement("fastapi"),
        requirement("pytest"),
    ],
)
```

`third_party/requirements.txt`

```txt
# list externals dependencies available for every python packages
fastapi==0.63.0

#test
requests==2.25.1
pytest==6.1.2
```

In fact, when running the test the command launch is something like `python test.py` with test.py id the value from the `main` argument of `py_test` (by default the value of `name` + `.py`) and it should be a member of srcs. So no way to call the module `pytest`. The workaround is to call `pytest` from `test.py`.

```python
from fastapi.testclient import TestClient

from exp_python.webapp.main import app

client = TestClient(app)


def test_read_main():
    response = client.get("/status")
    assert response.status_code == 200
    assert response.json() == {"status": "UP", "version": "0.1.0"}
    assert True == False

# if using 'bazel test ...'
if __name__ == "__main__":
    import sys
    import pytest
    sys.exit(pytest.main([__file__] + sys.argv[1:]))
```

Notice that `app` is now imported from `exp_python.webapp.main` and no longer relative. I didn't find how to handle it, without providing the full package name from root workspace.

Now when launch test `bazel test //exp_python/webapp:test`, it failed as expected (I let you fix the test)

```sh
INFO: From Testing //exp_python/webapp:test:
==================== Test output for //exp_python/webapp:test:
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/53/execroot/__main__/bazel-out/k8-fastbuild/bin/exp_python/webapp/test.runfiles/__main__
collected 1 item

exp_python/webapp/test.py F

=================================== FAILURES ===================================
________________________________ test_read_main ________________________________

    def test_read_main():
        response = client.get("/status")
        assert response.status_code == 200
        assert response.json() == {"status": "UP", "version": "0.1.0"}
>       assert True == False
E       assert True == False

exp_python/webapp/test.py:12: AssertionError
=========================== short test summary info ============================
FAILED exp_python/webapp/test.py::test_read_main - assert True == False
============================== 1 failed in 0.10s ===============================
================================================================================
Target //exp_python/webapp:test up-to-date:
  bazel-bin/exp_python/webapp/test
INFO: Elapsed time: 0.923s, Critical Path: 0.85s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//exp_python/webapp:test                                                 FAILED in 0.8s
  /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/test/test.log

INFO: Build completed, 1 test FAILED, 2 total actions
```

### Run webapp

The goal is also to be able to run the webapp. To launch a FastAPI webapp, the recommended way is to use uvicorn

```txt
# list externals dependencies available for every python packages
fastapi==0.63.0
uvicorn==0.13.4

#test
requests==2.25.1
pytest==6.1.2
```

But we can not launch uvicorn from command line, because we do not install it "globaly" on the system. On the other side, our target is to create an executable, that we can later launch on local or into a container.

Building executable is the goal of rule with suffix `_binary` in the Bazel ecosystem like `py_binary`.

```python
load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

...

# to add additionals parameters place them after "--" in bazel call, like:
# `bazel run //exp_python/webapp:run -- --reload`
py_binary(
    name = "run",
    srcs = ["run.py"],
    python_version = "PY3",
    srcs_version = "PY3",
    visibility = ["//visibility:public"],
    deps = [
        ":webapp",
        requirement("uvicorn"),
    ],
)
```

As you can see, we also introduce `run.py`, because like for py_test previously there is main attribute and only file from srcs can be used.

So create `exp_python/webapp/run.py`

```python
import uvicorn
import sys

if __name__ == '__main__':
    # freeze_support()
    sys.argv.insert(1, "exp_python.webapp.main:app")
    sys.exit(uvicorn.main())
```

And try it
```sh
bazel run //exp_python/webapp:run
# open into browser or via curl http://127.0.0.1:8000/status
```

If you launch with `bazel run //exp_python/webapp:run -- --reload` and do change into main.py they should be detected and apply.

### Editor

At this point, we have a bazel project that is working, but bazel is not really well supported by IDE / Editor (except for edition of bazel configuration file and sometimes launch of command).
What we can do to improve a little, is to create a python virtual env with same python version and python external dependencies (not scoped by packages, target, usage).

Currently I don't know, how to do it better (suggestions are welcomes), so we'll create a file at the root of the workspace `setup_localdev.sh`:

```sh
#!/bin/bash

PYENV_VERSION="3.9.2"

eval "$(pyenv init -)"
pyenv install ${PYENV_VERSION} --skip-existing
pyenv local ${PYENV_VERSION}
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r third_party/requirements.txt
python --version
```

This file could be generated by bazel, but I failed to :
- find a way to share the python version with `WORKSPACE.bazel`
- generate the shell script with `eval "$(pyenv init -)"`
(Maybe a suject for an other article)

Run this script, configure your editor to use the virtual environement `.venv` and continue to edit code. Re-run the script everytime you update `requirements.txt` (remove of dependencies, do not clean the virtual environment).

## To be continued

It's not the end, we have more stuff to setup (linters,...), but we're in a state enough to work.

The sandbox_bazel is hosted on github (not with the same history, due to errors), use tag to have the expected view at end of article: [article/4_python_1](https://github.com/davidB/sandbox_bazel/tree/article/4_python_1). I'll be happy to have your comments on this article, or [to discuss](https://github.com/davidB/sandbox_bazel/discussions) on github repo.