Two Methods for Testing HTTPS API Calls with Python and pytest and Also Communicating with the in-Laws
Tags: python, pytest, webdev, tutorial
Published: 2021-03-16 23:19:32

API endpoints and web URLs are, thankfully, more secure than ever, usually requiring encrypted HTTPS. [Python] works well as an HTTPS client, and [pytest] simplifies testing Python-based tools or libraries. Tools like [VCR.py] or the combination of [pytest-httpserver] and [trustme] provide an additional testing layer that is fast and convenient, and well-suited for HTTPS work. This will help with family gatherings. Let me show you.

## A potentially awkward social situation, solved by HTTPS

Imagine this scenario: you recently married into a wonderful family, but your spouse's parents only speak [pig Latin](https://en.wikipedia.org/wiki/Pig_Latin). Unfortunately, you are not fluent in this exotic language, and you are worried about the next family gathering. As you can guess, this is a common situation for many, so, thankfully, [there is an API for that](https://api.funtranslations.com/translate/piglatin.json?text=Thank+you+for+your+hospitality+I+brought+a+turkey).

Being the enterprising Python developer that you are, you write a Python client to serve as a realtime translator.

It looks like this:

```python
import json
from urllib.parse import urlencode
from urllib.request import urlopen


def translate(text, server="api.funtranslations.com"):
    data = {"text": text}
    form_encoded_data = urlencode(data).encode()
    url = f"https://{server}/translate/piglatin.json"
    with urlopen(url, form_encoded_data) as response:
        result = json.load(response)
        print(result)
    return result["contents"]["translated"]
```

Less than a dozen lines of code for marital bliss. Seems worthwhile.

You might save the above as `pypiglatin.py` in a directory of your choosing.

The script and associated tests can be found [in the companion Github repo][repo].

## The tests

If you are like me, you have already tried the above enough times to receive a [429 "Too Many Requests" error](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429). The [funtranslations.com service](https://funtranslations.com/register) allows you to sign up and become a paying customer in order to remove the rate limiting, but the anonymous limits certainly make a point. Your many tests are using up someone's server bandwidth and compute cycles. Maybe we should stop doing that. Even if the API you are accessing isn't rate limited.

Instead, find ways of testing locally. It will be faster, and kinder. Tests will make sure that the pig Latin translation is working smoothly, in advance of any high-pressure family gatherings.

I believe tools for HTTP client testing should:

- As already mentioned, easily replicate API endpoints locally, for performance and respect of API limits.
- Locally test the same protocol (HTTPS or HTTP) that is used in production. Usually API endpoints are HTTPS only, and so the URLs, even in testing, should start with "https://"
- HTTP client library flexibility. Yes, [requests] pairs well with [responses] or [requests-mock], and [HTTPX] has [RESPX] or [pytest_httpx]. Those testing helpers are an excellent match for the corresponding library, and should certainly be recommended. However, I don't always want to face the risk of rewriting all my tests if I replace the client library some day. And sometimes I am using an altogether different tool (even though both [requests] and [HTTPX] are quite awesome) such as [urlopen](https://dev.to/bowmanjd/http-calls-in-python-without-requests-or-other-external-dependencies-5aj1), [urllib3], [httplib2], [tornado], or [aiohttp].

Let's explore two methods for meeting the above goals. Both assume the use of [pytest]:

1. An easy-to-configure HTTP server using [pytest-httpserver], coupled with [trustme] for quick SSL certificate setup.
2. While not an actual HTTP server, [pytest-vcr] records and then mocks HTTP requests using the [VCR.py] library.

## pytest-httpserver

It is hard to beat an actual HTTP server for testing reassurance, and [pytest-httpserver] provides a quick way to configure responses based on expected requests. The web application library [Werkzeug] (the foundation of the [Flask] framework) is used to build the server.

Here is an example that tests if a request to the "/hello" endpoint indeed returns "Hello, World!" as expected:

```python
from urllib.request import urlopen

import pytest


@pytest.fixture(scope="session")
def httpserver_listen_address():
    return ("localhost", 8888)


def test_hello(httpserver):
    body = "Hello, World!"
    endpoint = "/hello"
    httpserver.expect_request(endpoint).respond_with_data(body)
    with urlopen(httpserver.url_for(endpoint)) as response:
        result = response.read().decode()
    assert body == result
```

To use the above test, save it in your current directory as `test_http.py` or similar, then install `pytest` and `pytest-httpserver` in your preferred manner (`pip install pytest pytest-httpserver` works well in a [Python virtual environment](https://dev.to/bowmanjd/python-tools-for-managing-virtual-environments-3bko)). Then run `pytest` from the command line in that directory.

While not strictly necessary, I like defining the `httpserver_listen_address` fixture from `pytest-httpserver` to explicitly define the server and port. This makes it easy to define URLs later, as the port is known. Otherwise, `pytest-httpserver` will assign a random port.

The heart of the test server setup is the line

```python
httpserver.expect_request(endpoint).respond_with_data(body)
```

We tell the server what sort of request to expect, and the data with which to respond to the expected request. This is quite configurable, on both ends. See [the docs](https://pytest-httpserver.readthedocs.io/en/latest/tutorial.html#specifying-the-expectations-and-constraints) for more advanced examples. For instance, request and/or response headers can be defined, the method (GET, POST, PUT, DELETE) specified, and various formats of data can be expected or returned, such as JSON.

### Adding SSL with trustme

What the sample test above is missing is HTTPS. Thankfully, `pytest-httpserver` allows a Python [`ssl.SSLContext`][sslcontext] to be specified by defining the `httpserver_ssl_context` fixture.

However, building a functioning certificate authority and the rest of the cert chain is not exactly a trivial matter. Enter [trustme], a Python-based tool for enabling SSL for testing purposes. You can install it in your virtual environment with `pip install trustme`.

Here is a test mechanism for our previously-built `translate` function:

```python
import ssl
from urllib.parse import urlencode

import pytest
import trustme

import pypiglatin

SERVER = "localhost"
PORT = 8888
ENDPOINT = "/translate/piglatin.json"


@pytest.fixture(scope="session")
def httpserver_listen_address():
    return (SERVER, PORT)


@pytest.fixture(scope="session")
def httpserver_ssl_context():
    ca = trustme.CA()
    client_context = ssl.SSLContext()
    server_context = ssl.SSLContext()
    server_cert = ca.issue_cert("test-host.example.org")
    ca.configure_trust(client_context)
    server_cert.configure_cert(server_context)

    def default_context():
        return client_context

    ssl._create_default_https_context = default_context

    return server_context


def test_post(httpserver):
    text = "Thank you for your hospitality"
    translated = "ank-Thay ou-yay or-fay our-yay ospitality-hay "
    response = {
        "success": {"total": 1},
        "contents": {
            "translated": translated,
            "text": text,
            "translation": "pig-latin",
        },
    }

    httpserver.expect_request(
        ENDPOINT, method="POST", data=urlencode({"text": text})
    ).respond_with_json(response)
    assert pypiglatin.translate(text, f"{SERVER}:{PORT}") == translated
```

A few notables:

- We take advantage of the `httpserver_ssl_context` fixture to not only build a server SSL context and return it, but also a build a client context that trusts the server cert. We then make `ssl._create_default_https_context()` return that client context. Thus, any HTTPS client that seeks to utilize the default context from the `ssl` module will respect our new cert authority.
- If your client does not use the `ssl` module to provide an SSL context, you will want to find another way to pass the client SSL context to your HTTP client. If need be, see the [trustme] documentation on how to write the certs to files so they can be utilized later.
- The expected request is a little more complex than our `test_hello` example. It expects `application/x-www-form-urlencoded` POST data (encoded with [`urllib.parse.urlencode()`][urlencode]).
- The response is a Python dict populated with a known API response, then converted to JSON by the `respond_with_json()` method.
- In my tests, the server SSL context seemed to work as the client SSL context as well. It just didn't seem like the right way to do it, so I separated them. Feel free to post advice in the comments.
- This is only one test. We can add tests to make sure incorrect requests elicit an error, and other messages are translated correctly. Neither the production API server nor the in-laws will know how fast and how often we fail. Beauty.

The end result is a live local HTTPS service intended to behave very similarly to the remote production service when called with specific data. The primary difference is that the URL is "https://localhost:8888/translate/piglatin.json" when testing, but "https://api.funtranslations.com/translate/piglatin.json" in production.

For a nice introduction to `pytest-httpserver`, [see the tutorial](https://pytest-httpserver.readthedocs.io/en/latest/tutorial.html). For a deeper dive, [see the how-to](https://pytest-httpserver.readthedocs.io/en/latest/howto.html).

## VCR.py

I love the convenience and realism that `pytest-httpserver` provides. The only downside is the mild tedium in recording and setting up responses and expected requests manually.

An alternate approach involves using [VCR.py], a tool that records HTTP interactions in YAML files, then intercepts future HTTP requests and plays back the recorded responses. In this tutorial, we will use [pytest-vcr] to interface with [VCR.py], although [pytest-recording] is another good option for doing the same.

While this doesn't provide an actual HTTPS server like `pytest-httpserver` does, the URLs used in the client code do not need to change. They can still start with "`https://`"; in fact, because the HTTPS calls are intercepted, URLs can use the same domains as production, without fear that calls will be emitted to the production servers (after the initial recording is saved).

Unfortunately, VCR.py only supports [a certain number of HTTP client libraries][vcr-support]. Thankfully, some very popular implementations are included: the Python built-in `urllib` and `http.client`, as well as [requests], [urllib3], and [aiohttp]. And, while it may not be listed in the docs at the time of this writing, [the changelog notes](https://github.com/kevin1024/vcrpy/blob/v4.1.1/docs/changelog.rst) that [HTTPX] is also supported. Sadly, [pycurl] is not supported.

To use VCR.py with pytest, start by installing `pytest-vcr` in your virtual environment:

```sh
pip install pytest-vcr
```

Implementing a test for the `translate()` function we built is quite simple using [pytest-vcr]:

```python
import pytest

import pypiglatin


@pytest.mark.vcr()
def test_translate():
    text = "Thank you for your hospitality"
    translated = "ank-Thay ou-yay or-fay our-yay ospitality-hay "
    assert pypiglatin.translate(text) == translated
```

You might save this as `test_vcr.py` in the current directory. Now we can run `pytest`.

Afterward, there should be a `cassettes` directory with a `test_translate.yaml` file in it. Feel free to take a look: you will get an idea for the many facets of an HTTP request and response. The file is also easily editable, which could be useful, for instance, to munge authentication/authorization information (just do that before saving the file to version control, of course).

If you run `pytest` again, the test should run much faster, using the existing cassette.

This YAML cassette can be saved to version control, alongside the tests, so that later runs on any machine will not need to hit the live server at all.

And there it is. [VCR.py] is actually this easy.

## Improved family dynamics

Hopefully, you are now gaining fluency in pig Latin. Meanwhile, you can be confident in your fallback plan: a thoroughly-tested translator, available at a moment's notice. Feel the reliability.

Please feel free to leave comments with your experiences, questions, or marital advice!

## Alternatives

Thes two methods are solid and flexible.

Of course, as noted earlier, there are other options out there, especially if you use [requests] or [HTTPX]:

For [requests]:

- [responses]
- [requests-mock]

For [HTTPX]:

- [RESPX]
- [pytest_httpx]

## Also of interest

If this article was helpful, you may also be interested in some of my other articles:

- [Using pytest_httpx with HTTPX](https://dev.to/bowmanjd/getting-started-with-httpx-part-2-pytest-and-pytesthttpx-2jef)
- [Using pytest in a Python dev environment](https://dev.to/bowmanjd/python-dev-environment-part-2-testing-with-pytest-5a1h)

[aiohttp]: https://docs.aiohttp.org/en/stable/client.html
[flask]: https://flask.palletsprojects.com/
[httplib2]: https://github.com/httplib2/httplib2
[httpx]: https://www.python-httpx.org/
[pycurl]: http://pycurl.io/
[pytest-httpserver]: https://pytest-httpserver.readthedocs.io/
[pytest-recording]: https://github.com/kiwicom/pytest-recording
[pytest-vcr]: https://pytest-vcr.readthedocs.io/
[pytest]: https://docs.pytest.org/
[pytest_httpx]: https://colin-b.github.io/pytest_httpx/
[python]: https://python.org
[repo]: https://github.com/bowmanjd/pypiglatin
[requests-mock]: https://requests-mock.readthedocs.io/
[requests]: https://requests.readthedocs.io/
[responses]: https://github.com/getsentry/responses
[respx]: https://lundberg.github.io/respx/
[sslcontext]: https://docs.python.org/3/library/ssl.html#ssl.SSLContext
[tornado]: https://www.tornadoweb.org/en/stable/httpclient.html
[trustme]: https://trustme.readthedocs.io/
[urlencode]: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlencode
[urllib3]: https://urllib3.readthedocs.io/
[vcr-support]: https://vcrpy.readthedocs.io/en/latest/installation.html#compatibility
[vcr.py]: https://vcrpy.readthedocs.io/
[werkzeug]: https://werkzeug.palletsprojects.com/