Top Features Added to Python From 3.7 to Now
Tags: python, datascience, webdev, news
Published: 2021-01-14 22:11:26

2018 brought us a plethora of new features with the release of Python 3.7, followed by 3.8 in 2019, and 3.9 in 2020.

Many of those changes were behind the scenes. Optimizations and upgrades that the vast majority of us will never notice, despite their benefits.

Others are more obvious, additions to syntax or functionality that can change how we write our code. But even these visible changes can be hard to keep up with.

In this article, we will run through the more apparent upgrades to provide a brief but hopefully invaluable refresher on everything new to Python from the past few years.

```
> Python 3.7
  - Breakpoints

> Python 3.8
  - Walrus Operator
  - F-string '=' Specifier
  - Positional-only Parameters

> Python 3.9
  - New Parser
  - More Type Hinting
  - Dictionary Unions
```

If you prefer video, I've run through everything (code included) here:

{% youtube KTFWNI0qL28 %}

---

# Python 3.7
## Breakpoints

Sometimes, or realistically, almost always - code debugging is frustrating. Python 3.7 made it slightly less annoying though with the inclusion of `breakpoint()`.

This new function allowed us to add breakpoints directly into our scripts, if we see an error that we simply cannot figure out - we can add `breakpoint()` to stop the code and allow us to try and figure out what is causing the error.

![breakpoint() makes use of the pdb debugger by default, but can make use of other alternative debuggers too.](https://dev-to-uploads.s3.amazonaws.com/i/rsdhrtnt7xa4gusjs2sg.png)

We can then run the code, which will break at our breakpoint and provide us with an interactive shell to test our code with.

It allows us to print, test our functions with different datatypes/variables - and simply do what we need to do to debug our code.

Read more about this in [PEP 553](https://www.python.org/dev/peps/pep-0553/).

---

# Python 3.8
## Walrus Operator

Most people have already seen the walrus operator, which was the go-to for anyone covering new features in 3.8.

As well as being a fun addition to Python syntax, the operator is genuinely very useful. It allows us to assign a value to a variable 'on the fly'.

This means that we can write more compact code. Although the following is not a good use-case, it makes it very clear what the operator can do.

![Without the walrus operator, we need to assign t_len before entering into our if-statement.](https://dev-to-uploads.s3.amazonaws.com/i/elm1npj9m9gs38ax7zxm.png)

Here we need to check for the length of a list, and if it is over a certain length, print a statement about its length being greater than three.

The walrus operator allows us to combine the variable assignment and if-statement into a single line like so:

![We can write more compact code using the walrus operator.](https://dev-to-uploads.s3.amazonaws.com/i/7f2rd84k95lnfaj5wgil.png)

Or perhaps a more useful example is the use of this with RegEx, where we need to check for the existence of a pattern and - if it matches - returns the matched pattern:

![Example of the walrus operator used alongside RegEx.](https://dev-to-uploads.s3.amazonaws.com/i/ur5fvu5r6u3s87lvu4o5.png)

[PEP 572](https://www.python.org/dev/peps/pep-0572/) covers everything you could ever want to know about the walrus operator.

## F-string '=' Specifier

Another good feature for debugging. When printing f-strings we can add `=` to print both the variable name and value, like so:

!['=' specifier can be used in f-strings to return both the variable name and value.](https://dev-to-uploads.s3.amazonaws.com/i/g61u77i8dkn631onuime.png)

A smaller, but still useful addition. Covered comprehensively in [bpo-36817](https://bugs.python.org/issue36817).

# Positional-only Parameters

Our final feature from 3.8 is the inclusion of syntax to specify function input parameters that cannot be called by name, but instead by position. For example:

![We can specify the input parameters by their given keywords, or not - sometimes it can be beneficial to enforce that users do not.](https://dev-to-uploads.s3.amazonaws.com/i/xdbtwpz0sazt3a861ht5.png)

Here we can specify the input parameters by their given names - and in most cases this makes sense.

However, sometimes this might be a behavior that we would like to block. For example, if we are coding a function wherein the name of the input parameters is likely to change in the future.

![We may want to avoid allowing users to specify the input parameter name so that we can change the variable name in the future without issue.](https://dev-to-uploads.s3.amazonaws.com/i/12maslu1qofdxk9tsdrw.png)

If we and other users of our functions specify the input parameter names, and we then update that function to use a different variable name - we could break all other code-bases using that function.

To stop users from using input parameter names, we can add the positional-only syntax /. Let's see how this behaves when added to our original code:

![We will receive a TypeError if we try to specify a positional-only parameter by its keyword.](https://dev-to-uploads.s3.amazonaws.com/i/s95elf1yyn8fpdcw5udk.png)

We see that a TypeError is raised for all parameters defined before /. This prevents users from using these variable names from the start. Now, users will have to use the position of op rather than the keyword:

![Adding the positional-only parameter means users will be forced to specify the parameter by position rather than keyword.](https://dev-to-uploads.s3.amazonaws.com/i/4h5x6kr15padebo6eay1.png)

So that when we do change op to method in the future, our users are protected:

![Now, when users switch from the old function using op, to the new one using method - they won't notice a difference - and it won't break their code.](https://dev-to-uploads.s3.amazonaws.com/i/gtc4efpgg15jhz2sqjmo.png)

Read all about it in [PEP 570](https://www.python.org/dev/peps/pep-0570/).

---

# Python 3.9

I have written more about Python 3.9 in [this article](https://towardsdatascience.com/python-3-9-9c2ce1332eb4), but I will summarize everything here.

## New Parser

Although this doesn't bring in any new syntax immediately, it is still significant and can lead to some big changes further down the road.

Guido van Rossum wrote the previous Python parser pgen 30 years ago. It was one of the first pieces of code ever written for Python.

Pgen uses a variant of LL(1)-based grammar. Meaning the parser reads the code top-down, left-to-right, with a lookahead of just one token.

This creates a few limitations:
* A one-token lookahead limits the expressiveness of grammar rules.
* Non-LL(1) grammar implementation requires a lot of messy workarounds for the Python devs.
* Left-recursive syntax can cause an infinite loop in the parse tree, causing stack overflow ([explained here](https://medium.com/@gvanrossum_83706/left-recursive-peg-grammars-65dab3c580e1)).

From Python 3.9 onwards. CPython will use the PEG-based parser, which we will begin to see the impact of from Python 3.10 onwards. You can read more about it in [PEP 617 here](https://www.python.org/dev/peps/pep-0617/).

## More Type Hinting

![Timeline of additions to typing in Python.](https://dev-to-uploads.s3.amazonaws.com/i/w6kvo9m4l8n1sxalkbfo.png)

The steady introduction and improvement of type hints in Python are a recurring theme in the past few years of Python.

In short, they introduce an optional layer of syntactical sugar that helps us understand which datatypes are expected in our code.

![No type hinting (left), Python 3.9 type hinting (right).](https://dev-to-uploads.s3.amazonaws.com/i/s7o5zb5636mj6thbtki0.png)

In the image above, we have code without type hints (left), and with (right). The Python linter will also read through this code and identify where there seems to be a mismatch between the datatype defined by our type hint, and the actual data types being used.

![Specifying the expected input and output data types.](https://dev-to-uploads.s3.amazonaws.com/i/fz8hsak72j3whavwmkc7.png)

We can also use the -> type syntax to determine function output datatypes too.

![We can couple together different types to create more complex annotations.](https://dev-to-uploads.s3.amazonaws.com/i/91hletlglihacxbrcm1t.png)

Or we can define more complex datatypes too. [PEP 585](https://www.python.org/dev/peps/pep-0585/) covers type hinting in Python 3.9.

## Dictionary Unions

Another interesting addition to Python operators. Both are used for dictionary unions. We have the merge operator `|`:

```python
a = {1: 'a', 2: 'b', 3: 'c'}
b = {4: 'd', 5: 'e'}
c = a | b
print(c)
```

`[Out]: {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e'}`

And the update operator `|=`, which allows us to do the merge in-place:

```python
a = {1: 'a', 2: 'b', 3: 'c'}
b = {4: 'd', 5: 'e'}

a |= b
print(a)
```

`[Out]: {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e'}`

Read more about dictionary union operators in [PEP 584 here](https://www.python.org/dev/peps/pep-0584/).

---

And that is it for this article covering some of the biggest changes and upgrades to Python over the past few years.

Of course, there is a huge range of features that we haven't covered here. But, these are those that are most obvious and create the biggest difference when working with Python.

![Python 3.10 development started mid-2020, and we can expect the full release in October 2021.](https://dev-to-uploads.s3.amazonaws.com/i/f7ft71u2kvn1icj6kwv3.png)

Nonetheless, this selection of new features is more than enough to keep you updated with the growth and evolution of Python. And amidst the upcoming Python 3.10 changes, there is plenty to learn.

I hope you enjoyed this article! If you have any questions, let me know via [Twitter](https://twitter.com/jamescalam) or in the comments below. If you'd like more content like this, I post on [YouTube](https://www.youtube.com/channel/UCv83tO5cePwHMt1952IVVHw) too.

Thanks for reading!

---

# Resources

[1] Guido van Rossum, [PEG Parsers](https://medium.com/@gvanrossum_83706/peg-parsers-7ed72462f97c) (2019).