Getting Started with HTTPX, Part 3: Building a Python REST Client (Asynchronous Version)
Tags: python, httpx, asyncio, pythonpoetry
Published: 2020-08-13 15:34:14

[HTTPX] is a modern HTTP client library for Python. Its interface is similar to the old standby [Requests], but it supports asynchronous HTTP requests, using Python's [asyncio] library (or [trio](https://trio.readthedocs.io/)). In other words, while your program is waiting for an HTTP request to finish, other work does not need to be blocked.

In [Part 1], we built a simple Wikipedia search tool using Python and [HTTPX]. Even though [HTTPX] was used, the tool was only synchronous. In other words, each HTTP request was sent sequentially, and subsequent requests only start after the previous one is complete. A lot of waiting in line.

Now, let's do what [HTTPX] is good for: asynchronous HTTP requests.

## `async` and `await`

Python's [asyncio] allows tasks to collaborate. When a task is busy waiting on input/output, it can give other tasks room to do their business.

To designate such a function, precede it with the `async` keyword. To _call_ such a function, precede the call with the `await` keyword.

We can create another python module (a file), `src/pypedia/asynchronous.py` with the following code that uses`async` and `await`. It is nearly the same as the code from [Part 1], with a few differences. Feel free to compare the two.

```python
"""Proof-of-concept asynchronous Wikipedia search tool."""
import asyncio
import logging
import time

import httpx

EMAIL = "your_email@provider"  # or Github URL or other identifier
USER_AGENT = {"user-agent": f"pypedia/0.1.0 ({EMAIL})"}

logging.basicConfig(filename="asyncpedia.log", filemode="w", level=logging.INFO)
LOG = logging.getLogger("asyncio")


async def search(query, limit=100, client=None):
    """Search Wikipedia, returning a JSON list of pages."""
    if client:
        close_client = False
    else:
        client = httpx.AsyncClient()
        close_client = True
    LOG.info(f"Start query '{query}': {time.strftime('%X')}")
    url = "https://en.wikipedia.org/w/rest.php/v1/search/page"
    params = {"q": query, "limit": limit}
    response = await client.get(url, params=params)
    if close_client:
        await client.aclose()
    LOG.info(f"End query '{query}': {time.strftime('%X')}")
    return response


async def list_articles(queries):
    """Execute several Wikipedia searches."""
    async with httpx.AsyncClient(headers=USER_AGENT) as client:
        tasks = [search(query, client=client) for query in queries]
        responses = await asyncio.gather(*tasks)
    results = (response.json()["pages"] for response in responses)
    return dict(zip(queries, results))


def run():
    queries = [
        "linksto:Python_(programming_language)",
        "incategory:Computer_programming",
        "incategory:Programming_languages",
        "incategory:Python_(programming_language)",
        "incategory:Python_web_frameworks",
        "incategory:Python_implementations",
        "incategory:Programming_languages_created_in_1991",
        "incategory:Computer_programming_stubs",
    ]
    results = asyncio.run(list_articles(queries))
    for query, articles in results.items():
        print(f"\n*** {query} ***")
        for article in articles:
            print(f"{article['title']}: {article['excerpt']}")
```

Note the use of [`httpx.AsyncClient`](https://www.python-httpx.org/async/) rather than `httpx.Client`, in both `list_articles()` and in `search()`.

In `list_articles()`, the client is used in a context manager. Because this is asynchronous, the context manager uses `async with` not just `with`.

In `search()`, if the client is not specified, it is instantiated, not with the context manager, but with `client = httpx.AsyncClient()`. When using this method, the responsibility is on us to close the client with `await client.aclose()`. Bad news if we forget to do this.

Our two primary functions have been preceded by the `async` keyword to indicate that they are async-friendly. In other words, they are willing to share control of the [event loop] when twiddling their thumbs.

If there was a need to call `search()` individually, then we could do so with `await search()`.

However, in this case, we need to concurrently run several calls to `search()`.

## [`asyncio.gather()`][gather]

The `list_articles()` function calls the awaitable `search()` function using [the function `asyncio.gather()`][gather]. This will create tasks for the event loop and run them concurrently.

Conveniently, [`asyncio.gather()`][gather] returns a list of each task's return values, in the exact order the functions were passed in.

> Note: put `await` before [`asyncio.gather()`][gather], but do not put `await` before the functions passed to it. The awaiting of each call will be handled by [`asyncio.gather()`][gather].

## Event loop

I have already mentioned the [event loop] a couple times. I think of the event loop as _the_ (there should be only one) task runner for [asyncio] applications. It handles the tasks.

Instantiating the event loop is done from the only non-awaitable function in our script. I named the function `run()`, coincidentally, and it calls [the high level function `asyncio.run()`][run].

Put another way, a synchronous function cannot `await` an asynchronous function. But it can `asyncio.run()`][run] it.

This creates a new event loop that then handles the various awaitable tasks, and returns the result of the called awaitable function.

## Enable the command runner

Our `run()` function executes whatever we want to have executed when called as a script. In this case, it creates a list of search terms, then sends the list to `list_articles()`, then parses and prints the result.

With Poetry, the entry point for a script is defined in `pyproject.toml`. So we add this to that file. Assuming you already had the synchronous `syncpedia` defined, that section should now look like this:

```toml
[tool.poetry.scripts]
asyncpedia = "pypedia.asynchronous:run"
syncpedia = "pypedia.synchronous:run"
```

So, the script `asyncpedia` will call the `run` function of the `asynchronous` submodule of the package `pypedia`. And, as already defined, the script `syncpedia` will call the `run` function of the `sync` submodule of the package `pypedia`.

Try it out:

```bash
poetry run asyncpedia
```

Assuming all works well, titles and excerpts of many Wikipedia articles should scroll by.

## Performance benefits of async

Unlike the script from [Part 1], the calls to the Wikipedia API now happen _asynchronously_, sharing the event loop concurrently. One request, while waiting for Wikipedia to respond, can share control of the event loop with the others. This can be seen in the log file.

```bash
$ cat asyncpedia.log
INFO:asyncio:Start query 'linksto:Python_(programming_language)': 06:03:39
INFO:asyncio:Start query 'incategory:Computer_programming': 06:03:39
INFO:asyncio:Start query 'incategory:Programming_languages': 06:03:39
INFO:asyncio:Start query 'incategory:Python_(programming_language)': 06:03:39
INFO:asyncio:Start query 'incategory:Python_web_frameworks': 06:03:39
INFO:asyncio:Start query 'incategory:Python_implementations': 06:03:39
INFO:asyncio:Start query 'incategory:Programming_languages_created_in_1991': 06:03:39
INFO:asyncio:Start query 'incategory:Computer_programming_stubs': 06:03:39
INFO:asyncio:End query 'incategory:Python_implementations': 06:03:39
INFO:asyncio:End query 'incategory:Python_(programming_language)': 06:03:39
INFO:asyncio:End query 'incategory:Programming_languages_created_in_1991': 06:03:39
INFO:asyncio:End query 'incategory:Python_web_frameworks': 06:03:39
INFO:asyncio:End query 'incategory:Computer_programming_stubs': 06:03:39
INFO:asyncio:End query 'incategory:Computer_programming': 06:03:40
INFO:asyncio:End query 'linksto:Python_(programming_language)': 06:03:40
INFO:asyncio:End query 'incategory:Programming_languages': 06:03:40
```

Note that start/end times are no longer sequential (or, perhaps, predictable). They are intermixed.

On my machine, the synchronous version completes in about 7 seconds, while this asynchronous version only takes around 2 seconds to complete.

That is a performance improvement!

Success isn't success, though, until we have repeatable tests constructed, as we will in [the next article][Part 4].

[run]: https://docs.python.org/3/library/asyncio-task.html#asyncio.run
[event loop]: https://docs.python.org/3/library/asyncio-eventloop.html
[gather]: https://docs.python.org/3/library/asyncio-task.html#asyncio.gather
[pytest]: https://docs.pytest.org/
[Python]: https://www.python.org/
[Part 1]: https://dev.to/bowmanjd/getting-started-with-httpx-building-a-python-rest-client-234n
[Part 2]: https://dev.to/bowmanjd/getting-started-with-httpx-part-2-pytest-and-pytesthttpx-2jef
[Part 3]: https://dev.to/bowmanjd/getting-started-with-httpx-part-3-building-a-python-rest-client-asynchronous-version-3lb0
[Part 4]: https://dev.to/bowmanjd/getting-started-with-httpx-part-4-pytest-asyncio-and-pytesthttpx-asynchronous-version-454d
[HTTPX]: https://www.python-httpx.org/
[asyncio]: https://docs.python.org/3/library/asyncio.html
[Poetry]: https://python-poetry.org/
[pytest_httpx]: https://github.com/Colin-b/pytest_httpx
[Requests]: https://requests.readthedocs.io/