Getting Started with HTTPX, Part 1: Building a Python REST Client (Synchronous Version)
Tags: python, httpx, pythonpoetry
Published: 2020-08-08 19:32:37

[HTTPX] is a modern HTTP client library for Python. Its interface is similar to the old standby [Requests], but it supports asynchronous HTTP requests, using Python's [asyncio] library (or [trio](https://trio.readthedocs.io/)). In other words, while your program is waiting for an HTTP request to finish, other work does not need to be blocked.

In this first article, we will first build a client that is synchronous. In other words, each request will complete before the next one starts. [The second article][Part 2] will build tests for this client, then [Part 3] will adapt it to make the requests asynchronously overlap, and then test asynchronously in [Part 4].

To interact with [HTTPX], let's build a mini-project called `pypedia`. It will be a command-line tool to list a few Python-related articles from [Wikipedia](http://wikipedia.org/).

[Poetry] eases Python project and dependency management, so I use that to quickly get a project up and running. If new to Poetry, you may appreciate [an article in which I introduce it](https://dev.to/bowmanjd/getting-started-with-python-poetry-3ica).

## Setup with [Poetry]

```bash
poetry new --src pypedia
cd pypedia
poetry add httpx
```

## Two functions and a command runner

In the `src/pypedia` directory, create a Python file called `synchronous.py`.

```python
"""Proof-of-concept Wikipedia search tool."""
import logging
import time

import httpx

EMAIL = "your_email@provider"  # or Github URL or other identifier
USER_AGENT = {"user-agent": f"pypedia/0.1.0 ({EMAIL})"}

logging.basicConfig(filename="syncpedia.log", filemode="w", level=logging.INFO)
LOG = logging.getLogger("syncpedia")


def search(query, limit=100, client=None):
    """Search Wikipedia, returning a JSON list of pages."""
    if not client:
        client = httpx
    LOG.info(f"Start query '{query}': {time.strftime('%X')}")
    url = "https://en.wikipedia.org/w/rest.php/v1/search/page"
    params = {"q": query, "limit": limit}
    response = client.get(url, params=params)
    LOG.info(f"End query '{query}': {time.strftime('%X')}")
    return response


def list_articles(queries):
    """Execute several Wikipedia searches."""
    with httpx.Client(headers=USER_AGENT) as client:
        responses = (search(query, client=client) for query in queries)
    results = (response.json()["pages"] for response in responses)
    # results = (response.json() for response in responses)
    return dict(zip(queries, results))


def run():
    """Command entry point."""
    queries = [
        "linksto:Python_(programming_language)",
        "incategory:Computer_programming",
        "incategory:Programming_languages",
        "incategory:Python_(programming_language)",
        "incategory:Python_web_frameworks",
        "incategory:Python_implementations",
        "incategory:Programming_languages_created_in_1991",
        "incategory:Computer_programming_stubs",
    ]
    results = list_articles(queries)
    for query, articles in results.items():
        print(f"\n*** {query} ***")
        for article in articles:
            print(f"{article['title']}: {article['excerpt']}")
```

In summary, the above has two significant functions and a command runner.

## Using `Client.get()`

The `search` function accepts a reusable [HTTPX Client] instance and the query string, then performs a [client GET request](https://www.python-httpx.org/api/#client) to the [Wikipedia search endpoint](https://www.mediawiki.org/wiki/API:REST_API/Reference#Search_pages).

The [HTTPX Client] is passed into the `search()` function as the `client` variable, so we can use methods like `client.get()`, passing two arguments: `url` and `params`. Whatever key:value pairs are in the `params` dict will make up the query string appended to the url, such as `q` (the Wikipedia search terms) or `limit`.

In case a pre-existing client is not passed to `search()`, the subsequent requests will use `httpx.get()`. This makes it easy to use `search()` by itself and test it in an isolated fashion.

## `httpx.Client` in a context manager

The `list_articles()` function opens an HTTPX Client as a context manager, so that cleanup is assured and automatic. It accepts one parameter, `queries`, and then iterates over that list, calling `search()` with every query. It does this _inside the context manager_. This way, all the `client.get()` calls in the `search()` function should benefit from the re-use of a single [HTTP persistent connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection). 

For those familiar with [Requests], this is the equivalent to [Requests's Session object](https://requests.readthedocs.io/en/master/user/advanced/).

The [HTTPX Advanced Usage guide](https://www.python-httpx.org/advanced/) has excellent rationale and instructions for using the Client.

## Enable the command runner

The `run()` function executes whatever we want to have executed when called as a script. In this case, it creates a list of search terms, then sends the list to `list_articles()`, then parses and prints the result.

With Poetry, the entry point for a script is defined in `pyproject.toml`. So we add this to that file:

```toml
[tool.poetry.scripts]
syncpedia = "pypedia.synchronous:run"
```

So, the script `syncpedia` will call the `run` function of the `synchronous` submodule of the package `pypedia`.


```bash
poetry install
```

## Synchronous execution

To run:

```bash
poetry run syncpedia
```

Assuming all works well, titles and excerpts of many Wikipedia articles should scroll by.

The calls to the Wikipedia API happened synchronously, in a sequence. One completed before the next began. This can be seen in the log file.

```bash
$ cat syncpedia.log
INFO:root:Start query 'linksto:Python_(programming_language)': 05:39:16
INFO:root:End query 'linksto:Python_(programming_language)': 05:39:17
INFO:root:Start query 'incategory:Computer_programming': 05:39:17
INFO:root:End query 'incategory:Computer_programming': 05:39:18
INFO:root:Start query 'incategory:Programming_languages': 05:39:18
INFO:root:End query 'incategory:Programming_languages': 05:39:19
INFO:root:Start query 'incategory:Python_(programming_language)': 05:39:19
INFO:root:End query 'incategory:Python_(programming_language)': 05:39:19
INFO:root:Start query 'incategory:Python_web_frameworks': 05:39:19
INFO:root:End query 'incategory:Python_web_frameworks': 05:39:20
INFO:root:Start query 'incategory:Python_implementations': 05:39:20
INFO:root:End query 'incategory:Python_implementations': 05:39:20
INFO:root:Start query 'incategory:Programming_languages_created_in_1991': 05:39:20
INFO:root:End query 'incategory:Programming_languages_created_in_1991': 05:39:20
INFO:root:Start query 'incategory:Computer_programming_stubs': 05:39:20
INFO:root:End query 'incategory:Computer_programming_stubs': 05:39:21
```

In the instance above, each call took 1 second or less, executing in a clear order.

In other words, everybody had to wait in line.

[HTTPX] has the ability to make calls asynchronously, in which each call does not need to wait its turn in line. This can potentially have performance benefits. We will explore the async possibilities [later in the series][Part 3].

For now, we cannot forget to write tests, and [the next article][Part 2] will engage this.

[Part 1]: https://dev.to/bowmanjd/getting-started-with-httpx-building-a-python-rest-client-234n
[Part 2]: https://dev.to/bowmanjd/getting-started-with-httpx-part-2-pytest-and-pytesthttpx-2jef
[Part 3]: https://dev.to/bowmanjd/getting-started-with-httpx-part-3-building-a-python-rest-client-asynchronous-version-3lb0
[Part 4]: https://dev.to/bowmanjd/getting-started-with-httpx-part-4-pytest-asyncio-and-pytesthttpx-asynchronous-version-454d
[HTTPX]: https://www.python-httpx.org/
[HTTPX Client]: https://www.python-httpx.org/api/#client
[Requests]: https://requests.readthedocs.io/
[asyncio]: https://docs.python.org/3/library/asyncio.html
[Poetry]: https://python-poetry.org/