Tools to automate Python Tests
Tags: testing, python, productivity, devops
Published: 2021-05-04 14:09:27

Heya fellows,

The code of this post can be found on Github (see [here](https://github.com/NiklasTiede/tinyHTTPie/tree/6-Testing-and-CI)).

When projects grow a good test suite gives you confidence that new code you add don't cause parts of the application to break. It improves the maintainability of the project. The complexity of small projects is low and tests are only necessary when the size of the project increases. But for the sake of this tutorial we will write a small test to demonstrate the usage of `pytest`, `tox` and `github actions`.

We will store the tests within a separate folder. Here's the current structure of the project.

```
├── LICENSE
├── README.md
├── setup.py
├── tihttp.py
├── tests
│  ├── __init__.py
│  └── test_tihttp.py
```

We place a `test_tihttp.py` file within the `tests` folder. It will contain the test suite. The `main()` function which contains the logic for triggering the flags has to be imported from the `tihttp.py` file.

```python
from tihttp import main

def test_GET_body(capsys):
    main(["-B", "http://jsonplaceholder.typicode.com/todos?userId=1"])
    captured = capsys.readouterr()
    result = captured.out
    with open("tests/jsonplaceholder.json", "r") as f:
        output = f.read()
    assert result == output
```

We compare the response of a GET request to an API with the expected JSON data. We place a `jsonplaceholder.json` file into the same folder which contains our expected output. Then we install `pytest` and let it execute the test.

```
$ pip install pytest
$ pytest
```

Our test passes. To increase the tests verbosity `-v` is a useful flag. Furthermore, I like to use the `-s` flag to see the captured output. Next we add pytest to our extra requirements in the `setup.py` file.

```python
extras_require={
    'dev': [
        'pytest', # pip install tihttp[dev]
    ],
},
```

This gives us the possibility to install extra dependencies (testing, linting tools etc.) easily by adding a `[dev]` to the package name.

```
$ pip install .[dev]            # local install
$ pip install tihttp[dev]       # remote install, PyPI repo
```

We tested all of this with python 3.7.3. But how does our application behave when executed on a different interpreter version? So let's test it against different Python versions! We use `tox`. It let's us run tests in multiple virtual envs.

```
$ pip install tox
```

Tox needs a recipe to know which virtualenv/commands to create/execute. This recipe is named `tox.ini`.

```ini
[tox]
envlist = py36,py37,py38,py39

[testenv]
deps =
    pytest
commands =
    pytest
```

If some of the Python interpreters are missing on your system, install them from the deadsnakes archive:

```
$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt install python3.5 python3.6 python3.7 python3.8 python3.9
```

Now let's test across different interpreters!

```
$ tox
```

If you wanna test against a specific environment or execute only one file, then type:

```
$ tox -e py38
$ tox -e py38 -- test/main_test.py   # executes only a single test
```

Ok, we did the test locally, but when working in a team using continuous integration is pretty convenient. We set a `integrate.yaml` file up within a `.github/workflows` directory to tell github actions what jobs to execute. The following github actions file will test across different platforms and Python versions.

```yaml
name: Python package

on: [push]

jobs:
  run:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        python-version: [3.6, 3.7, 3.8, 3.9]
        os: [ubuntu-latest, macos-latest, windows-latest]
    steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Cache pip
        uses: actions/cache@v2
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-
            ${{ runner.os }}-
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install flake8 pytest pytest-cov
          pip install -r requirements.txt
      - name: Lint with flake8
        run: |
          # stop the build if there are Python syntax errors or undefined names
          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
          # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
      - name: Test with pytest
        run: |
          pytest
```

Don't be intimidated by the length of this job. It's just illustrating how powerful Github workflows can be. 🥰