Iterable gotcha's in Python
Tags: python, todayilearned, typehints
Published: 2022-11-06 09:56:17


## What are Iterables?

Typehints have been [introduced](https://www.python.org/dev/peps/pep-0484/) in
Python since Python 3.5. As part of this introduction, a new module `typing` was
introduced which contained various types for built-in collections and Abstract
Base Classes. `Iterable` is a type that is prominantly used in the PEP for Type hints and is implemented by many of the most used Python collections. Since Python 3.9, this type can be imported from `collections.abc` and can be used as a type as well as an Abstract Base Class.

## When to use Iterables?

If you enthousiastically started using type hints in Python and have been following best practices to accept the most generic type possible in your arguments, you might have been using `Iterable`. `Iterable` can be used at any place where your code expects the variable to implement `__iter__`, in other words, where you code wants to iterate through a collection of some sorts.

Say you want to send an email to multiple receivers:

```python
class MailApi:
    def sendmail(self, sender: str, receiver: str, message: str) -> None:
        ...


api = MailApi()


def send_emails(message: str, sender: str, receivers: Iterable[str]) -> Optional[T]:
    if not receivers:
        raise ValueError("you need atleast one receiver")

    for receiver in receivers:
        api.sendmail(sender, receiver, message)
```

If you send an email using the following code you will send an email:

```python
send_emails(
    "Hello",
    "me@example.com",
    ["steve@example.com", "larry@example.com", "elon@example.com"],
)
```

If you use the following code, you will get an exception:

```python
send_emails("Hello", "me@example.com", [])
```

## Pitfalls

But what if you did this?

```python
send_emails(
    "Hello",
    "me@example.com",
    filter(
        lambda x: not x.endswith("example.com"),
        ["steve@example.com", "larry@example.com", "elon@example.com"],
    ),
)
```

You would intuitively expect an exception, because the filter will remove all matching emails, but **Nothing** will happen!

`filter` returns a valid `Iterable` and MyPy correctly asserts that there are no type errors. What is going on?

The pythonic way, to check if a collection is empty, is to use `if not <collection_name`. This works because Python returns `False` when `__len__` returns `0`. However, `Iterable` is not required to implement `__len__` as well as `__bool__`. The default truth value is `True`, so even though your code is perfectly Type safe, it will not do what you might expect. A solution to this problem is not simple, but you could prevent similar bugs by using a slightly less generic type such as `Collection`, which does require `__len__` to be implemented, if you want to catch these errors with a type checker.

## Conclusion

The new Python type hints are a welcome addition to the language and boost maintainability, reduce bugs and allow better editor support for Python. However, due to historic design decsicions there are subtle cases which can result in bugs in ways you wouldn't expect from type safe code. If you use `Iterable` in Python, consider using `Collection`.