Experimentations on Bazel: Python (3), linter & pytest
Tags: bazel, python, pytest, pylint
Published: 2021-05-03 19:36:42

In the previous articles,we saw how to launch linter for python with a custom rule, and in the one before how to launch pytest test. In this article, we'll see how to combine both usage in more friendly way (IMHO).

Since the previous article, I did more experimentations (on my side), and I was limited by my current knowledge about how to implement rules (and managed dependency). But with bazel's [macro], we could fix (or reduce) some issues:

- duplication of pytest's boilerplate every `BUILD.bazel` and `*test.py`
- duplication between pytest and potential linter
- too many custom code to please bazel

## Create the macro

Create the empty `tools/pytest/BUILD.bazel` , just to initiate the package.

Create the macro `tools/pytest/defs.bzl`, it is like the existing `py_test` defined in `exp_python/webapp/BUILD.bazel` but with the call inside a function and some part configurable.

```python
"""experience wrap py_test"""

load("@rules_python//python:defs.bzl", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

def pytest_test(name, srcs, deps = [], args = [], **kwargs):
    """
        Call pytest
    """
    py_test(
        name = name,
        srcs = [
            "//tools/pytest:pytest_wrapper.py",
        ] + srcs,
        main = "//tools/pytest:pytest_wrapper.py",
        args = [
            "--capture=no",
        ] + args + ["$(location :%s)" % x for x in srcs],
        python_version = "PY3",
        srcs_version = "PY3",
        deps = deps + [
            requirement("pytest"),
        ],
        **kwargs
    )

```
- `load` to import definition of symbols
- `pytest_test` is the macro, in fact a function that call py_test
- `//tools/pytest:pytest_wrapper.py` is the wrapper to call pytest (see below)
- `name`, `srcs`, `deps`, `args` are the argument of that will customize
- `**kwargs` allow to forward some args from the caller side for the part that are not overwitten by the macro
- `["$(location :%s)" % x for x in srcs]` is to append at the end of the call the list of source file to process, to have the path of those file (srcs is a list of string at this point), we call `$(location ...)` to have the path in the execution context of the py_test and and the prefix `:` is to convert the file name into a label. It's hacky but it works

Then create the wrapper that will call pytest `tools/pytest/pytest_wrapper.py`

```python
import sys
import pytest

# if using 'bazel test ...'
if __name__ == "__main__":
    sys.exit(pytest.main(sys.argv[1:]))

```

Replace the current call to py_test by a call to hour macro in `exp_python/webapp/BUILD.bazel`.

```python
load("//tools/pytest:defs.bzl", "pytest_test")
...

# py_test(
#     name = "test",
#     srcs = [
#         "test.py",
#     ],
#     # main = "test.py",
#     args = [
#         "--capture=no",
#     ],
#     python_version = "PY3",
#     srcs_version = "PY3",
#     deps = [
#         ":webapp",
#         requirement("requests"),
#         requirement("fastapi"),
#         requirement("pytest"),
#     ],
# )

pytest_test(
    name = "test",
    srcs = ["test.py"],
    deps = [
        ":webapp",
        requirement("requests"),
        # requirement("fastapi"),
    ],
)

```

We keep commented the previous code for comparison, it's shorter and we move into a central shared place code that could be shared with other python package. Give a try...

```sh
❯ bazel test //exp_python/webapp:test
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/BUILD.bazel:32:12: no such target '//tools/pytest:pytest_wrapper.py': target 'pytest_wrapper.py' not declared in package 'tools/pytest'; however, a source file of this name exists.  (Perhaps add 'exports_files(["pytest_wrapper.py"])' to tools/pytest/BUILD?) defined by /home/david/src/github.com/davidB/sandbox_bazel/tools/pytest/BUILD.bazel and referenced by '//exp_python/webapp:test'
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/BUILD.bazel:32:12: no such target '//tools/pytest:pytest_wrapper.py': target 'pytest_wrapper.py' not declared in package 'tools/pytest'; however, a source file of this name exists.  (Perhaps add 'exports_files(["pytest_wrapper.py"])' to tools/pytest/BUILD?) defined by /home/david/src/github.com/davidB/sandbox_bazel/tools/pytest/BUILD.bazel and referenced by '//exp_python/webapp:test'
ERROR: Analysis of target '//exp_python/webapp:test' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.704s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (18 packages loaded, 19 targets configured)
FAILED: Build did NOT complete successfully (18 packages loaded, 19 targets configured)
```

Ok, we need to fix some stuff, because it's a macro we need to expose pytest_wrapper.py to caller of the macro. So just follow the recommendation and change the empty `tools/pytest/BUILD.bazel` into

```python
exports_files(["pytest_wrapper.py"])
```

Retry...

```sh
❯ bazel test //exp_python/webapp:test
INFO: Analyzed target //exp_python/webapp:test (26 packages loaded, 15198 targets configured).
INFO: Found 1 test target...
Target //exp_python/webapp:test up-to-date:
  bazel-bin/exp_python/webapp/test
INFO: Elapsed time: 2.242s, Critical Path: 0.99s
INFO: 5 processes: 3 internal, 2 linux-sandbox.
INFO: Build completed successfully, 5 total actions
//exp_python/webapp:test                                                 PASSED in 0.7s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these aINFO: Build completed successfully, 5 total actions
```

First victory.

We can now remove the boilerblate to call pytest at the end of `exp_python/webapp/test.py` because it is now provided by the shared `tools/pytest/pytest_wrapper.py`.

```python
from fastapi.testclient import TestClient
from exp_python.webapp.main import app

client = TestClient(app)


def test_read_main():
    response = client.get("/status")
    assert response.status_code == 200
    assert response.json() == {"status": "UP", "version": "0.1.0"}

```
You can mofidy the test to force the fail, and check if it'll fail as expected.

## Use Pytest to launch linter

Pytest have lot of plugin/extension to launch tool like black, pylint, flake8, mypy,... So instead of create one linter call we can configure pytest run all linter.

Add linter into `third_party/requirements.txt`

```txt
...
#test
requests==2.25.1
pytest==6.1.2

#tools
black==20.8b1
pytest-black==0.3.12
pytest-mypy==0.8.1
pytest-pylint==0.18.0
pylint==2.8.2
```

Configure `tools/pytest/defs.bzl` to also launch linter and configure the call to pytest (it's like a script and a configuration)

```python
"""experience wrap py_test"""

load("@rules_python//python:defs.bzl", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

def pytest_test(name, srcs, deps = [], args = [], **kwargs):
    """
        Call pytest
    """
    py_test(
        name = name,
        srcs = [
            "//tools/pytest:pytest_wrapper.py",
        ] + srcs,
        main = "//tools/pytest:pytest_wrapper.py",
        args = [
            "--capture=no",
            "--black",
            "--pylint",
            "--mypy",
        ] + args + ["$(location :%s)" % x for x in srcs],
        python_version = "PY3",
        srcs_version = "PY3",
        deps = deps + [
            requirement("pytest"),
            requirement("pytest-black"),
            requirement("pytest-pylint"),
            requirement("pytest-mypy"),
        ],
        **kwargs
    )

```

Run it

```sh
❯ bazel test //exp_python/webapp:test
INFO: Analyzed target //exp_python/webapp:test (20 packages loaded, 2332 targets configured).
INFO: Found 1 test target...
FAIL: //exp_python/webapp:test (see /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/test/test.log)
INFO: From Testing //exp_python/webapp:test:
==================== Test output for //exp_python/webapp:test:
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/3/execroot/__main__/bazel-out/k8-fastbuild/bin/exp_python/webapp/test.runfiles/__main__
plugins: black-0.3.12, pylint-0.18.0, mypy-0.8.1
collected 5 items
--------------------------------------------------------------------------------
Linting files
....Unable to create directory /home/david/.pylint.d
Unable to create file /home/david/.pylint.d/exp_python.__init__1.stats: [Errno 2] No such file or directory: '/home/david/.pylint.d/exp_python.__init__1.stats'

--------------------------------------------------------------------------------

exp_python/webapp/test.py FFF..

=================================== FAILURES ===================================
______________________ [pylint] exp_python/webapp/test.py ______________________
C:  1, 0: Missing module docstring (missing-module-docstring)
C:  7, 0: Missing function or method docstring (missing-function-docstring)
__________________________ exp_python/webapp/test.py ___________________________
1: error: Cannot find implementation or library stub for module named 'fastapi.testclient'
1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
_________________________________ test session _________________________________
mypy exited with status 1.
===================================== mypy =====================================
exp_python/webapp/main.py:1: error: Cannot find implementation or library stub for module named 'fastapi'
Found 2 errors in 2 files (checked 1 source file)
=========================== short test summary info ============================
FAILED exp_python/webapp/test.py::PYLINT
FAILED exp_python/webapp/test.py::mypy
FAILED exp_python/webapp/test.py::mypy-status
========================= 3 failed, 2 passed in 1.99s ==========================
================================================================================
Target //exp_python/webapp:test up-to-date:
  bazel-bin/exp_python/webapp/test
INFO: Elapsed time: 3.841s, Critical Path: 3.45s
INFO: 5 processes: 3 internal, 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 5 total actions
//exp_python/webapp:test                                                 FAILED in 3.0s
  /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/test/test.log

INFO: Build completed, 1 test FAILED, 5 total actions
```

Seems to work (or failed as expected, linter detect issue)

Now extend `pytest_test` to lint over every *.py of the package (and let pytest detects test). As we check every python file, we also need to add all their dependencies.

```python
pytest_test(
    name = "test",
    srcs = glob(["*.py"]),
    deps = [
        ":run",
        ":webapp",
        requirement("requests"),
        requirement("fastapi"),
    ],
)

```

To fix the linter issue (I comments mypy, not useful for the purpose of this article), provide a configuration file for pylint and disable a pylint rule localy into `run.py` (full code available into the repo, see bellow). So the final `tools/pytest/defs.bzl` looks like (with a configuration file for the demo, provide as data to `py_test` and as args)

```python
"""Wrap pytest"""

load("@rules_python//python:defs.bzl", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

def pytest_test(name, srcs, deps = [], args = [], data = [], **kwargs):
    """
        Call pytest
    """
    py_test(
        name = name,
        srcs = [
            "//tools/pytest:pytest_wrapper.py",
        ] + srcs,
        main = "//tools/pytest:pytest_wrapper.py",
        args = [
            "--capture=no",
            "--black",
            "--pylint",
            "--pylint-rcfile=$(location //tools/pytest:.pylintrc)",
            # "--mypy",
        ] + args + ["$(location :%s)" % x for x in srcs],
        python_version = "PY3",
        srcs_version = "PY3",
        deps = deps + [
            requirement("pytest"),
            requirement("pytest-black"),
            requirement("pytest-pylint"),
            # requirement("pytest-mypy"),
        ],
        data = [
            "//tools/pytest:.pylintrc",
        ] + data,
        **kwargs
    )

```

And modify `tools/pytest/BUILD.bazel` to export configuration files.

```python
exports_files([
    "pytest_wrapper.py",
    ".pylintrc",
])

```

## Clean up

Remove code and reference to pytest_check (that we introduce previously), it's no longer needed. Now we have

```txt
❯ tree -a -I '.vscode|bazel-*|.venv|.git'                                                                                      21:16:36
.
├── .bazelrc
├── .bazelversion
├── BUILD.bazel
├── exp_genrule
│   ├── a.txt
│   ├── b.txt
│   └── BUILD.bazel
├── exp_python
│   └── webapp
│       ├── BUILD.bazel
│       ├── main.py
│       ├── __pycache__
│       │   └── main.cpython-39.pyc
│       ├── run.py
│       └── test.py
├── .github
│   ├── dependabot.yml
│   └── workflows
│       └── ci.yml
├── .gitignore
├── .python-version
├── setup_localdev.sh
├── third_party
│   ├── BUILD.bazel
│   └── requirements.txt
├── tools
│   ├── pytest
│   │   ├── BUILD.bazel
│   │   ├── defs.bzl
│   │   ├── .pylintrc
│   │   └── pytest_wrapper.py
│   └── workspace_status.sh
└── WORKSPACE.bazel
```

## Limits of the solution

- Dependencies of test, runtime, tools are all mixed (no isolation), but it's the common (crappy ?) way to work in python ecosystem.
- The macro is harder to extract as a workspace, tools for reuse into an other project, but in the other side the code is small enough to be copied and customized for each need
- No split in the output of bazel every linter and test will be show as 1 test into report of bazel.

At the end, currently it's the most pleasant solution I found to have pytest, linters run by bazel without too many (and duplicate) boilerplate into each package. A central configuration place, and pretty simple to extends if pytest as the extension (else we can tweak the `pytest_wrapper.py` to do complementary stuff by example).

## To be continued

It's not the end, we'll continue to setup other stuff like depending of another package (a python lib), make a docker image,...

The sandbox_bazel is hosted on github (not with the same history, due to errors), use tag to have the expected view at end of article: [article/6_python_3](https://github.com/davidB/sandbox_bazel/tree/article/6_python_3). I'll be happy to have your comments on this article, or to discuss on github repo.

[macro]: https://docs.bazel.build/versions/master/skylark/macros.html