What to Except When You're Excepting: Python error handling dos & don'ts
Tags: python, errors, handling, exceptions
Published: 2020-07-10 21:16:19

---
title: What to Except When You're Excepting: Python error handling dos & don'ts
published: true
description: Python error handling dos & don'ts
tags: Python, error, handling, exceptions
---
Exceptions in Python are a great part of the language. They're easy to use, easy to subclass, and [pretty fast](http://gerg.ca/blog/post/2015/try-except-speed/). Unfortunately, convenience can lead to sloppy thinking and sloppier programming. In my 5+ years of Python programming in various startups, I've encountered a _lot_ of bad exception handling, so here are some recommendations for dos and don'ts.


### **Do** think about your exceptions
Towards the bottom of the [post I linked to above](http://gerg.ca/blog/post/2015/try-except-speed/), the author provides evidence that `try`/`catch` can be slower than `if` statements _when the errors are frequent._ His somewhat flippant advice is "to look into more reliable infrastructure," which is helpful in some contexts. I've found, however, that the convenience of Python exceptions, coupled with the dynamically-typed nature of the language itself, leads to developers either taking exceptions for granted, over-using `try`/`catch`, or both.

One advantage of statically-typed languages like Haskell and Rust is that they force the programmer to think more deliberately about errors, and provide them with the language constructs to do so in the form of `Maybe`/`Either` (in Haskell) or `Option`/`Result` (in Rust.)

[For those unfamiliar, `Maybe`/`Option` are ways to avoid `null`, aka `undefined` aka `None` aka [Hoare's 'billion-dollar mistake'](https://medium.com/@hinchman_amanda/null-pointer-references-the-billion-dollar-mistake-1e616534d485); in Haskell, a value of type `Maybe a` can either be `Just a` (e.g. `Just "a friend"`), or `Nothing`. Similarly, a value of type `Either a b` can be either (get it?) `Left a` (e.g. `Left "for dead"`) or `Right b` (e.g. `Right 1`). Both languages use one half of their `Either`-like types for "success" values, and the other for "error" values; it's common to see things like `Result<Int,String>` in Rust or `Either String Int` in Haskell. (I know why Haskell orders it the way it does (ask me in the comments if you care), I don't know why Rust switched the order.)]

Both of these languages have "real" exceptions, but most Haskell and Rust programmers eschew them in favor of `Maybe`/`Either`/`Option`/`Result`. Why? Because using these types allows you, and _people reading your code_, to differentiate between **errors** and **exceptions**. Errors are anticipatable, semi-routine, unfortunate but not disastrous. Exceptions are _exceptional_&mdash;they shouldn't happen, their very presence represents some serious derivation from the usual course of events.

Even in Python, which doesn't have anything like `Maybe`/`Either`, it's still worth thinking about which of your "exceptions" are actually exceptional, and which are merely errors. Viz:

```python
try:
    middle_name = user['middle_name']
except KeyError:
    middle_name = ''


try:
    birthday = user['birthday']
except KeyError:
    birthday = ''
```
We're treating two pieces of missing information the same way, but they're actually _completely different_. Lots of people don't have middle names; _everybody_ has a birthday. (Of course, some users may be wary of divulging such information to strangers on the internet, but let's assume you've earned their trust.) It's much clearer to treat different cases differently:

```python
middle_name = user.get('middle_name', '')
birthday = user['birthday']
```

Now we're treating **errors** and **exceptions** with the relative alarm they each deserve.

### **Do** handle specific errors appropriately
This is an extension of the previous point. You can use multiple `catch`es per `try`, and if there's cause to, you should.

```python
# instead of this...
try:
    do_whatever()
except SomeException as err:
    if isinstance(err, FooException):
        handle_foo_exception(err)
    elif isinstance(err, BarException):
        handle_bar_exception(err)

# ...do this
try:
    do_whatever()
except FooException as err:
    handle_foo_exception(err)
except BarException as err:
    handle_bar_exception(err)
```


### **Do** write your own exception subclasses
You can facilitate and extend the pattern above by writing and throwing your own exception subclasses. It's easy and aids in clarity.

```python
# instead of this...
def check_birthday(user):
    if not user.birthday:
        raise ValueError('User birthday missing!')
    elif user.birthday.year > 2002:
        raise ValueError('User is too young!')
    elif user.birthday.year < 1900:
        raise ValueError('User is probably dead!')

try:
    check_birthday(user)
except ValueError as err:
    msg = str(err)
    if 'missing' in msg:
        redirect_to_birthday_input()
    elif 'young' in msg:
        redirect_to_site_for_teens()
    elif 'old' in msg:
        handle_probable_fraud()

# ...do this
class UserBirthdayException(ValueError):
    pass

class UserBirthdayMissing(UserBirthdayException):
    pass

class InvalidUserBirthday(UserBirthdayException):
    def __init__(self, problem, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.problem = problem


def check_birthday(user):
    if not user.birthday:
        raise UserBirthdayMissing
    elif user.birthday.year > 2002:
        raise InvalidUserBirthday(problem='young')
    elif user.birthday.year < 1900:
        raise InvalidUserBirthday(problem='old')

try:
    check_birthday(user)
except UserBirthdayMissing:
    redirect_to_birthday_input()
except InvalidUserBirthday as err:
    if err.problem == 'young':
        redirect_to_site_for_teens()
    else:
        handle_probable_fraud()
```

Sure, it's more boilerplate, but the added clarity and control is worth it. It's especially worth considering if you're doing a lot of validation, or find yourself repeatedly handling a set of errors from an API&mdash;write some custom error classes and a response handler function, and your days of grepping through your codebase to find all the places you need to change your error handling because some third-party has switched from colons to tabs in their API responses are over.

### **Do** Make use of exception class hierarchies
Like everything else in Python, exceptions are objects, and they have their own classes and inheritance. `catch` will handle subclasses of errors appropriately.

```python
class AError(Exception):
    pass

class BError(AError):
    pass

class CError(AError):
   pass


try:
    raise BError
except CError:
    print('Caught CError')  # won't happen
except AError:
    print('Caught AError')  # will happen
```
Familiarize yourself with Python's [builtin exceptions](https://docs.python.org/3/library/exceptions.html#bltin-exceptions), and don't be shy about using inheritance when defining your own error classes.

### **Do** keep it tight
Quick, tell me which of these functions could throw `SomeException`:

```python
def f(x):
    try:
        a = do_something(x)
        b = do_something_else(a)
        c = lastly_do(b)
        log_or_whatever(c)
        return c
    except SomeException as err:
        log_exception(err)
        return None
```

Give up? Me too. How about here:

```python
def f2(x):
    a = do_something(x)
    try:
        b = do_something_else(a)
    except SomeException as err:
        log_exception(err)
        return None
    else:
        c = lastly_do(b)
        log_or_whatever(c)
        return c
```
Better, no? Keep the scope of your `try`/`catch` blocks as narrow as possible. Yes, nested `try`/`catch`/`else` blocks can get annoying, but why not just refactor each of the potential culprits into separate functions?

### **Don't** over-generalize
This should go without saying&mdash;by me, because your tooling (`pep8`/`flake8`/`pylint`, etc.) should be saying it at you instead. But I'm saying it anyway, because it bears repeating (and because I've seen&mdash;and written!&mdash;far too many `noqa`/`pylint: disable=bare-except` comments.)

A common excuse I've seen for `catch Exception` is ignorance&mdash;there's a 3rd-party API with bad documentation, or some complicated code with multiple points of failure. Unfortunately, as with the law, ignorance of potential exceptions is not a valid excuse. Do as much research as you can, add logging, and then refactor to catch the actual exceptions you want to catch.

And _whatever_ you do, **_DO NOT_** `catch BaseException`. Trust me.

Lastly,

### **Don't** be afraid to let it ride
Computers are incredibly complex; the world is a terrifying, unpredictable place; chaos is the only constant. Stuff (as it were) will happen. You don't have to `catch` everything; you don't even have to try. Think about what what handling every exception would even _mean_: if your DB can't be reached, or a config file doesn't exist, or Jeff Bezos decides overnight to get out of cloud storage and into self storage, what's the point of carrying on like your user can go about their business as normal? [Five 9s](https://en.wikipedia.org/wiki/High_availability#%22Nines%22) is the goal, but [Potemkin](https://en.wikipedia.org/wiki/Potemkin_village)-oriented programming serves no-one. Print that TB, return that 500, move on.