Completely Type-Safe Error Handling in Python
Tags: python, functional, productivity
Published: 2020-08-04 15:20:19

In this post in my series on functional programming in Python with [pfun](https://pfun.dev/),
we'll take a look at the Python type system, and how it can be used to make error handling with `pfun` completely type safe. In a [previous post in this series](https://dev.to/suned/purely-functional-python-with-static-types-41mf), we introduced the `pfun.effect.Effect` type and discussed how it models both successful computations (through the `A` type parameter called _the success type_) and errors (through the `E` type parameter called _the error type_). 

Recall that `Effect` represents functions that:

- Take exactly one argument of type `R`
- Returns a result of type `A` or raises an error of type `E`
- May or may not perform side effects

Let's investigate how this abstraction enables completely type-safe error handling using a small example. In order to understand how `Effect` enables this feature, let's re-examine the method `and_then` that chains together an `Effect` with a function that returns a new `Effect`:


```python
from typing import TypeVar, Generic, Union, Callable, Any


R = TypeVar('R', contravariant=True)
E = TypeVar('E', covariant=True)
A = TypeVar('A', covariant=True)

E2 = TypeVar('E2')
B = TypeVar('B')


class Effect(Generic[R, E, A]):
    def __call__(self, r: R) -> A
        ...

    def and_then(self, 
                 f: Callable[[A], Effect[Any, E2, B]]
                 ) -> Effect[Any, Union[E, E2], B]:
        ...
```
We've left out the implementation since it's not particularly important (you can see the details in the [previous post](https://dev.to/suned/purely-functional-python-with-static-types-41mf) if you want). What _is_ important is the _signature_ of `and_then`. To see how the type of `and_then` enables type-safe error handling, consider this code:
```python
first: Effect[R, E, A]
second: Effect[R, E2, B]

f = lambda _: second
last: Effect[R, Union[E, E2], B] = first.and_then(f)

result: B  = last(...)
```

When `last` is called in the last line,  it calls `first` which might fail with a value of type `E`. If `first` succeeds, `last` calls `f` with the result of `first`. `f` returns `second`, which is also called called by `last`. But `second` might fail with a value of type `E2`, which ultimately means that `last` can fail in exactly two ways:

- If `first` fails
- if `second` fails

And so, the correct error type of `last` must be `Union[E, E2]`, which is what you'll find expressed in the signature of `and_then`.

The use of `Union` in the return type of `and_then` means that complex error types are built up by combining effects with simple error types _automatically_. This is extremely useful because it allows us to reason about complex errors of combined effects with the help of a type-checker like [MyPy](https://mypy.readthedocs.io/en/stable/) without any special effort on our part: we can simply describe effects with simple error types, combine them with `and_then` (or any other function in the `pfun.effect` api that accepts multiple effects) and let the type-checker do the complicated work of keeping track of which errors may be raised when an `Effect` is called.

Automatic tracking of error types isn't particularly useful if we can't eliminate error types by handling them with type safety. `pfun.effect.Effect` provides [several ways of handling errors](https://pfun.dev/effectful_but_side_effect_free/#error-handling). We'll study one in particular through the following example:

Imagine you want to use `pfun` to call an HTTP api and parse the result as JSON in functional style. To make HTTP requests you can use [`pfun.http`](https://pfun.dev/http_api/):

```python
from http.client import HTTPException

from pfun.effect import Effect
from pfun.http import HTTP

http = HTTP()
call_api: Effect[object, HTTPException, bytes] = http.get('http://foo-api.com')
```
(Of course you don't have to type out the type of `call_api` as it can be inferred by `MyPy`, we do it here only because it's instructive to look at the types).

Since not all byte strings are valid JSON data, let's write a function `parse_json` that parses JSON data as an `Effect`. We can do this using the `effect.success` function that creates an effect that does nothing but return its argument, and `effect.error` that creates an effect that does nothing but fail with its argument:

```python
from json import JSONDecodeError, loads

from pfun.effect import success, error


def parse_json(s: bytes) -> Effect[object, JSONDecodeError, dict]:
    try:
        return success(loads(b))
    except JSONDecodeError as e:
        return error(e)
```
Or simply using the `effect.catch` decorator:
```python
from json import JSONDecodeError, loads

from pfun.effect import catch


parse_json = catch(JSONDecodeError)(loads)
```
With `parse_json` in hand, our entire program looks like:
```python
from typing import Union


program: Effect[object, Union[HTTPException, JSONDecodeError], dict]
program = call_api.and_then(parse_json)
```
Now, imagine that we want to be sure that all errors are handled at the time `program` is assigned its value. The function we'll use to eliminate errors is called [`Effect.recover`](https://pfun.dev/effect_api/#pfun.effect.Effect.recover). This function allows us to pass in a function that inspects the error and returns a new effect (which might fail in new ways).

Let's demonstrate by handling the `HTTPException`. This might be done by calling a backup api. In this example, we'll simply imagine that we can use a default backup response
```python
from typing import NoReturn


def handle_http_error(reason: HTTPException) -> Effect[object, NoReturn, bytes]:
    return success(b'{}')


program: Effect[object, JSONDecodeError, dict] = call_api.recover(handle_http_error).and_then(parse_json)
```
`typing.NoReturn` is a special type that indicates that `handle_http_error` can't return a value for the `E` type variable of `Effect`, which is let's the type checker verify that `handle_http_error` can't fail.

We could of course proceed and handle the `JSONDecodeError` in a similar fashion, but I think you get the idea.

In summary, `pfun.effect` enables type safe error handling through pure functional programming without any special effort from the developer. This is useful because it enables the type checker to verify that our error handling is correct (at least in terms of the types). To learn more about `pfun` check out the [documentation](https://pfun.dev/) or head over to the github repository.

{% github suned/pfun no-readme %}