What to Expect from Python 3.11?
Tags: python, discuss
Published: 2022-03-30 14:06:21


Here are the most important updates on our favourite programming language.


## Even Better Error Messages

When dealing with highly nested data structures (such like API responses) you might encounter a `KeyError` when trying to access undefined dictionary keys.

Before Python 3.11 this has been particularly painful. Consider this code snippet:

```python
users = [
    {'id': 1, 'name': 'Bas', 'social': {'twitter': '@bascodes'}},
    {'id': 2, 'name': 'John Doe', 'social': {}},
]

def get_twitter(user):
    return user['social']['twitter']

get_twitter(users[1])
```

which will produce the following error message on Python 3.10:

```
Traceback (most recent call last):
  File "/Users/sebst/tmp311/py311.py", line 9, in <module>
    get_twitter(users[1])
  File "/Users/sebst/tmp311/py311.py", line 7, in get_twitter
    return user['social']['twitter']
KeyError: 'twitter'
```

In this small example, we immediately see that the error is because of the missing twitter account of user 2. The more nested these structures become, the more helpful is the new error message format in Python 3.11:

```
Traceback (most recent call last):
  File "/tmp/py311.py", line 9, in <module>
    get_twitter(users[1])
    ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/py311.py", line 7, in get_twitter
    return user['social']['twitter']
           ~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'twitter'
```

The error message immediately tells us which part of the dictionary is missing. This will become even more helpful when we have two dictionary accesses in one line.


## Typing: Adding the Self Type

[PEP 673](https://peps.python.org/pep-0673/) introduces the `Self` type.

Here is an example from the PEP:

```python
from typing import Self

class Shape:
    def set_scale(self, scale: float) -> Self:
        self.scale = scale
        return self


class Circle(Shape):
    def set_radius(self, radius: float) -> Self:
        self.radius = radius
        return self
```


## Typing: Variadic Generics

[PEP 646](https://peps.python.org/pep-0646/) discusses the introduction of Variadic Generics in the context of numerical libraries, such as NumPy. In that context, a variable is not only characterized by its type but also by its shape.

Particularly, the introduction of `TypeVarTuple` allows using an arbitrary-length type variable like so:

```python
from typing import TypeVar, TypeVarTuple

DType = TypeVar('DType')
Shape = TypeVarTuple('Shape')

class Array(Generic[DType, *Shape]):

    def __abs__(self) -> Array[DType, *Shape]: ...

    def __add__(self, other: Array[DType, *Shape]) -> Array[DType, *Shape]: ...
```


## TOML

The TOML format (maybe popular by using it in `pyproject.toml` files) can now be parsed with a tomllib, which will become part of Python 3.11's standard library through [PEP 680](https://peps.python.org/pep-0680/).

```python
import tomllib
with open("pyproject.toml", "rb") as f:
    data = tomllib.load(f)
```

<!-- ## Star Expressions in Loops

```python
result = []
a = b = c = [1, 2, 3]
for x in *a, *b, *c:
    result.append(x)

print(result)
# [1, 2, 3, 1, 2, 3, 1, 2, 3]
``` -->


## Exception Groups / `except*`

One important feature of Python 3.11 is discussed in [PEP 654](https://peps.python.org/pep-0654/).

This PEP introduces `ExceptionGroup`s. With them, we're able to raise multiple exceptions simultaneously like so:

```python
try:
    raise ExceptionGroup("Validation Errors", (
        ValueError("Input is not a valid user name."),
        TypeError("Input is not a valid date."),
        KeyError("Could not find associated project.")
    ))
except* (ValueError, TypeError) as exception_group:
    ...
except* KeyError as exception_group:
    ...
```

## AsyncIO Task Groups


The idea of `TaskGroup`s is to run nested tasks and continue running them even if one fails. Errors are raised using exception groups, so no error will pass silently.

In fact, the previously mentioned `ExceptionGroup`s were needed to implement the `TaskGroup`s feature.

For more information on TaskGroups, look at this tweet:

{% embed https://twitter.com/1st1/status/1493748843430567942 %}




<!-- ```python
try:
    async with trio.open_nursery() as nursery:
        # Make two concurrent calls to child()
        nursery.start_soon(child)
        nursery.start_soon(child)
except* ValueError:
    pass
```

```python
def handle_ValueError(exc):
    if isinstance(exc, ValueError):
        return None
    else:
        return exc   # reraise exc

with MultiError.catch(handle_ValueError):
    async with trio.open_nursery() as nursery:
        # Make two concurrent calls to child()
        nursery.start_soon(child)
        nursery.start_soon(child)
``` -->


```python
try:
    async with asyncio.TaskGroup() as tg:
        tg.create_task(coro1())
        tg.create_task(coro2())
except* ValueError as e:
    pass # Ignore all ValueErrors
```


## Performance Improvements

The release notes of [Python 3.11.0a6](https://www.python.org/downloads/release/python-3110a6/) claim that there is a performance improvement of approx. 19% in CPython 3.11 vs. CPython 3.10.

For a quick overview have a look at this tweet:

{% embed https://twitter.com/methane/status/1501108529297432577 %}