HTTP Calls in Python Without Requests or Other External Dependencies
Tags: python, tutorial, webdev, programming
Published: 2021-03-07 22:44:17

In addition to great [Python] HTTP client tools such as [Requests] and [HTTPX], the standard library itself supplies the necessary ingredients to make a working HTTP client for API calls. This tutorial shares how to construct and customize such a tool for your own scripts.

## Consider installing a library

Before proceeding, I should note that in many cases, the approach in this article is not best practice. Instead, I highly recommend using a third-party Python library for features, security, and reliability.

Some suggested libraries:

- [urllib3] is the dependency for many other tools, including [requests]. By itself, [urllib3] is quite usable. It may be all you need.
- [requests] is ubiquitous and well documented.
- [HTTPX] has an interface almost identical to [requests], but with the added benefit of [asyncio] support. You may be interested in [a series of articles I wrote on using HTTPX](https://dev.to/bowmanjd/getting-started-with-httpx-building-a-python-rest-client-234n) both synchronously and asynchronously.
- [pycurl] is less popular as a Python library, but interfaces with the well-known [libcurl](https://curl.se/libcurl/).
- [aiohttp] has an asyncio-based HTTP client that is well-documented and well-liked.

If, however, you find yourself needing a solution that does not require external dependencies other than what is already available in the Python standard library, then you may wish to read on.

## Summary code

```python
import json
import typing
import urllib.error
import urllib.parse
import urllib.request
from email.message import Message


class Response(typing.NamedTuple):
    body: str
    headers: Message
    status: int
    error_count: int = 0

    def json(self) -> typing.Any:
        """
        Decode body's JSON.

        Returns:
            Pythonic representation of the JSON object
        """
        try:
            output = json.loads(self.body)
        except json.JSONDecodeError:
            output = ""
        return output


def request(
    url: str,
    data: dict = None,
    params: dict = None,
    headers: dict = None,
    method: str = "GET",
    data_as_json: bool = True,
    error_count: int = 0,
) -> Response:
    if not url.casefold().startswith("http"):
        raise urllib.error.URLError("Incorrect and possibly insecure protocol in url")
    method = method.upper()
    request_data = None
    headers = headers or {}
    data = data or {}
    params = params or {}
    headers = {"Accept": "application/json", **headers}

    if method == "GET":
        params = {**params, **data}
        data = None

    if params:
        url += "?" + urllib.parse.urlencode(params, doseq=True, safe="/")

    if data:
        if data_as_json:
            request_data = json.dumps(data).encode()
            headers["Content-Type"] = "application/json; charset=UTF-8"
        else:
            request_data = urllib.parse.urlencode(data).encode()

    httprequest = urllib.request.Request(
        url, data=request_data, headers=headers, method=method
    )

    try:
        with urllib.request.urlopen(httprequest) as httpresponse:
            response = Response(
                headers=httpresponse.headers,
                status=httpresponse.status,
                body=httpresponse.read().decode(
                    httpresponse.headers.get_content_charset("utf-8")
                ),
            )
    except urllib.error.HTTPError as e:
        response = Response(
            body=str(e.reason),
            headers=e.headers,
            status=e.code,
            error_count=error_count + 1,
        )

    return response
```

You are certainly welcome to copy and use the above function, or browse or clone [the Github repo][repo].

If, however, you are reading this article for a do-it-yourself approach, I encourage you to build your own function that suits your needs. It may grow simpler or more flexible than the above.

Let's discuss the building blocks.

## An introduction to `urllib.request.urlopen()`

The recommended high-level function for HTTP requests is [`urlopen()`][urlopen], available in the standard [`urllib.request`][urllib.request] module.

Unlike the lower-level [`http.client`][http.client] module, [`urlopen()`][urlopen] provides error handling, follows redirects, and provides convenience around headers and data.

An example:

```python
from urllib.request import urlopen, Request

url = "https://jsonplaceholder.typicode.com/posts/1"

if not url.startswith("http"):
    raise RuntimeError("Incorrect and possibly insecure protocol in url")

httprequest = Request(url, headers={"Accept": "application/json"})

with urlopen(httprequest) as response:
    print(response.status)
    print(response.read().decode())
```

I highly appreciate and recommend [JSONPlaceHolder]'s free fake API, used above. Useful precisely for what we are doing here: testing HTTP clients intended for API work.

Please note the security measures in the above code. Before passing a URL to `urlopen()`, [make sure that it is a web url and not a local file ("`file:///`")][bandit310]. If you want a wake-up call, try `urlopen("file:///etc/passwd").read()` on a Linux system (not in production code, though!) and see what happens. Of course, this protocol check is only necessary if the URL comes from user input. If you control the URL string and can assure that it does not start with "`file:`" then that is a good thing. You may also be interested in an another approach to [hardening `urlopen()` by redefining the list of protocol handlers](https://dev.to/bowmanjd/hardening-and-simplifying-python-s-urlopen-4gee).

Protocol checking aside, the `urlopen()` call is fairly simple, as you can see in the above example. I recommend using it alongside [`with` in a context manager](https://docs.python.org/3/reference/compound_stmts.html#with) for tidiness, so that closing the response is handled automatically.

We passed a [Request] object to the `urlopen()` function. While we could simply pass a URL string, the [Request] object offer much more flexibility: we can specify HTTP `method` (GET, POST, PUT, HEAD, DELETE), [request `headers`](https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Request_fields), and request `data`.

The [response][urllib.response] returned by `urlopen` has 4 useful attributes:

1. It has a file-like interface that can be `read()`, returning bytes
2. `url`
3. `status` returns the [HTTP status code][httpstatus]
4. `headers` returns an [EmailMessage] object. This functions somewhat like a `dict` but with case-insensitive keys. It also has some helpful methods such as [`get_content_type()`](https://docs.python.org/3.8/library/email.message.html#email.message.EmailMessage.get_content_type) and [`get_content_charset()`](https://docs.python.org/3.8/library/email.message.html#email.message.EmailMessage.get_content_charset). The [`get_all()`](https://docs.python.org/3.8/library/email.message.html#email.message.EmailMessage.get_all) method is another useful one, for when there may be multiple key/value pairs for the same header name. See [the helpful Wikipedia article](https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Response_fields) for a list of possible reponse headers.

## HTTP errors

Out of the box, `urlopen` handles redirects (status codes 301, 302, 303, or 307). Other than these codes, though, if the status code is not [between 200 and 299](https://github.com/python/cpython/blob/v3.9.2/Lib/urllib/request.py#L1954) (HTTP "OK" codes according to [RFC 2616](https://tools.ietf.org/html/rfc2616#section-10.2)) then an [HTTPError] exception is raised.

The [HTTPError] can be captured and analyzed with the appropriate `try... except...` block, such as this:

```python
from urllib.error import HTTPError
from urllib.request import urlopen

try:
    urlopen("https://github.com/404")
except HTTPError as e:
    print(e.status)
    print(e.reason)
    print(e.headers.get_content_type())
```

The error (assigned to the "`e`" variable in the above) has the following useful properties:

- `status` to get the error code (such as 404)
- `headers` as an [EmailMessage] object. Again, this can be treated like a case-insensitive `dict`.
- `reason` with the text of the error

In the function at the top of this article, I catch and silence all errors, to make the response uniform, and pass error-handling responsibility downstream. However, this may not be desirable. Perhaps, instead of continuing no matter the error, you want to fail on anything other than a 401 or 429 error:

```python
except urllib.error.HTTPError as e:
    if e.code in (401, 429):
        response = Response(
            body=str(e.reason),
            headers=e.headers,
            status=e.code,
            error_count=error_count + 1,
        )
    else:
        raise e
```

Of course, logic could be added to deal with errors as appropriate, depending on the status code.

Note the entirely optional auto-incrementing `error_count` attribute in my code. Sometimes, I wish to call the http request function recursively. This allows the number of calls to be tracked, and dealt with downstream, hopefully preventing infinite recursion. For instance, I may want to catch 401 errors, parse the "Www-Authenticate" header for a token, then retry the request with the token. But if this fails repeatedly (say, 5 tries), it should stop. I could test for `error_count >= 5` and raise and exception if so, meanwhile making sure to pass the current `error_count` back to the request function as a parameter, so it continues to be incremented appropriately.

An alternative way to customize error handling is to construct your own subclasses of [`BaseHandler`](https://docs.python.org/3/library/urllib.request.html#urllib.request.BaseHandler), then [build an OpenerDirector](https://docs.python.org/3/library/urllib.request.html#urllib.request.build_opener) chain of handlers as appropriate. For instance, you could subclass `BaseHandler` and add a method `http_error_401` to handle authorization as desired, then pass an instance of that custom class to `build_opener()`. Obviously, this requires a deeper dive into the opener innards.

## A versatile `Response` object

I find it helpful to create a Python class that can contain the bits of the HTTP response that I care about. This could be a `dict`, but I like to add a method or two, such as a JSON decoder.

If using Python 3.7 or later, consider using a [`dataclass`][dataclass]. An example:

```python
@dataclass(frozen=True)
class Response():
    body: str
    headers: Message
    status: int
    error_count: int = 0

    def json(self) -> typing.Any:
        """
        Decode body's JSON.

        Returns:
            Pythonic representation of the JSON object
        """
        try:
            output = json.loads(self.body)
        except json.JSONDecodeError:
            output = ""
        return output
```

Enabling `frozen` is strictly optional, and reflects my preference for this object being immutable (attributes can't be changed after initialization).

Another option, as demonstrated in the code at the beginning of the article, is a [typed NamedTuple](https://docs.python.org/3/library/typing.html#typing.NamedTuple). I chose this for its immutability, ease of setup, and backwards compatibility.

Of course, a custom class will work, or [attrs], or whatever container works for you.

## Requests with data

Depending on the API with which you are interfacing, you may encounter various scenarios for accepting data. In each scenario, we can start with a Python `dict` and convert it into the required format.

### The query string

Sometimes, data is passed in through the query string. To encode a Python `dict` as a query string that can be appended to the URL (after the "`?`"), use [`urllib.parse.urlencode`][urlencode]:

```python
from urllib.parse import urlencode
from urllib.request import urlopen

url = "https://jsonplaceholder.typicode.com/posts"
params = {"userId": 1, "_limit": 3}
url += "?" + urlencode(params, doseq=True, safe="/")

with urlopen(url) as response:
    print(response.read().decode())
```

While not relevant to the above request, I did pass two parameters to `urlencode` that I have found helpful:

- `doseq` will allow lists to be encoded as multiple parameters. For instance, if we passed in `{"usernames": ["John Doe", "Jane Doe"]}`, then the end result would be "`usernames=John+Doe&usernames=Jane+Doe`".
- `safe` defines the characters that will not be url-encoded. In some APIs I encounter, such as the Docker API, it is better to leave slashes unencoded, so I added that to the `safe` string. Adapt as you see fit.

### Sending data in the request body

Similarly, data can be encoded with [`urllib.parse.urlencode`][urlencode] and then passed into the Request object via the `data` parameter:

```python
from urllib.parse import urlencode
from urllib.request import Request, urlopen

url = "https://api.funtranslations.com/translate/yoda.json"
data = {"text": "HTTP POST calls are remarkably easy"}

postdata = urlencode(data).encode()

httprequest = Request(url, data=postdata, method="POST")

with urlopen(httprequest) as response:
    print(response.read().decode())
```

## Sending JSON in the request body

Many APIs accept and even require the request parameters to be sent as JSON. In these cases, it is important to first encode the Python `dict` (or other object) as JSON, then set the "`Content-Type`" request header appropriately:

```python
import json
from urllib.parse import urlencode
from urllib.request import Request, urlopen

url = "https://jsonplaceholder.typicode.com/posts"
data = {
    "userid": "1001",
    "title": "POSTing JSON for Fun and Profit",
    "body": "JSON in the request body! Don't forget the content type.",
}

postdata = json.dumps(data).encode()

headers = {"Content-Type": "application/json; charset=UTF-8"}

httprequest = Request(url, data=postdata, method="POST", headers=headers)

with urlopen(httprequest) as response:
    print(response.read().decode())
```

In the above, we used [Python's built-in JSON module][pythonjson] to dump the data `dict` into a string, then encode it to bytes so it could then handled as POST data.

We set the content type header to `application/json`. In addition, we specified the character encoding as UTF-8. Given that UTF-8 is [the required JSON character encoding](https://tools.ietf.org/html/rfc8259#section-8.1), this is redundant and probably [unnecessary](https://tools.ietf.org/html/rfc8259#section-11), but it never hurts to be explicit.

## Parsing JSON in the response body

Because most APIs I use return JSON, and some return other formats such as XML unless JSON is specified, I typically set the `Accepts` header in the request to `application/json`. If you are pulling other types of data, such as `text/csv`, you would want to tweak that header. Set it to `*/*` if you don't care.

In the Response object we created earlier, there is an example of a JSON decoder, the result of which will likely be a Python `dict` or `list`, but could conceivably be a string or boolean, depending on what the server returns. Here is another example of similar functionality, but loading the JSON directly from the file-like [response][urllib.response]:

```python
import json
from urllib.parse import urlencode
from urllib.request import Request, urlopen

url = "https://jsonplaceholder.typicode.com/posts?_limit=3"

with urlopen(url) as response:
    try:
        jsonbody = json.load(response)
    except json.JSONDecodeError:
        jsonbody = ""

print(jsonbody)
```

In the above example I decided to fail silently, offering an empty string when JSON decoding fails. If this is not desired, just use `json.load()` (or `json.loads()` from a string) and let any exceptions float up as they occur.

## Other tricks or suggestions?

I am very curious if you use `urlopen` and how. Are there optimizations to the above that I am missing? Does this raise any questions or confusion? Feel free to post in the comments.

[aiohttp]: https://docs.aiohttp.org/en/stable/client.html
[asyncio]: https://docs.python.org/3/library/asyncio.html
[attrs]: https://www.attrs.org/
[bandit310]: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html?highlight=urlopen#b310-urllib-urlopen
[dataclass]: https://docs.python.org/3/library/dataclasses.html
[emailmessage]: https://docs.python.org/3.8/library/email.message.html
[http.client]: https://docs.python.org/3/library/http.client.html
[httperror]: https://docs.python.org/3.8/library/urllib.error.html#urllib.error.HTTPError
[httpstatus]: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
[httpx]: https://www.python-httpx.org/
[jsonplaceholder]: https://jsonplaceholder.typicode.com/
[pycurl]: http://pycurl.io/
[pythonjson]: https://docs.python.org/3/library/json.html
[python]: https://www.python.org/
[repo]: https://github.com/bowmanjd/pysimpleurl
[request]: https://docs.python.org/3/library/urllib.request.html#urllib.request.Request
[requests]: https://requests.readthedocs.io/
[urlencode]: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlencode
[urllib.parse]: https://docs.python.org/3/library/urllib.parse.html
[urllib.request]: https://docs.python.org/3/library/urllib.request.html
[urllib.response]: https://docs.python.org/3/library/urllib.request.html#module-urllib.response
[urllib3]: https://urllib3.readthedocs.io/
[urlopen]: https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen