Iterators & Generators in Python
Tags: python, iterator, generator
Published: 2020-12-27 15:54:22


This post aims to describe the basic mechanisms behind iterators and generators.

## Iterator protocol

As in many programming languages, Python allows to iterate over a collection. The iteration mechanism is often useful when we need to scan a sequence, operation that is very common in programming.
In Python the iterator protocol involves two components: an *iterable* and an *iterator*.

## Iterable

The *iterable* is the container through which we want to iterate. It is the object that needs to be scanned to retrieve all the elements (or some of them). Some of well-known iterables are lists, tuples, dictionaries, ranges. In the *iterator protocol* the *iterable* exposes a **__iter__** method that returns an *iterator* object.

## Iterator

The *iterator* is the data structure that allows the scan through the container. It could seem a complication, but actually, with the separation of concerns, it lets the developer separate the concept of container, from the concept of iteration. The container object doesn't need to keep the state of an iteration and furthermore, on the same object, many iterations at the same time can take place, so keeping the iteration state in a different object is a must. The container is a collection of elements, while the iterator is kind of an handler of the container, it exposes the same elements (owned by the container) one-by-one with a specific order and . In the *iterator protocol*, the *iterator* exposes two methods:  **__iter__** and **__next__**. While the first one return the object itself (it allows the usage of both container and iterator in *for* and *in* statements), the latter return the next item from the container. What does it make the iterator end the iteration? The **StopIteration** exception.

## Iterable and Iterator Examples

Below an example of iterator protocol implementation:
{% gist https://gist.github.com/antomor/c26816773b028eb9c3b5be8835d36669 file=TenIntegers.py %}

And its usage:
{% gist https://gist.github.com/antomor/c26816773b028eb9c3b5be8835d36669 file=iterable_iterator_examples.py %}

## Generator

*Generators* are methods with yield statements. The *yield* statement has the power to suspend the function execution and store its state, so that it can be resumed. Behind the scenes, python returns the control to the function caller and save the function state; in this way, at the next execution, the function will start where it left, without letting the developer worrying about the state of the function. *Generators* ARE *iterators*, but not vice-versa.

Here an example of generator:
{% gist https://gist.github.com/antomor/c26816773b028eb9c3b5be8835d36669 file=ten_integers_generator.py %}

and its usage:
{% gist https://gist.github.com/antomor/c26816773b028eb9c3b5be8835d36669 file=ten_integers_generator_examples.py %}


### Async generator

Like *generators* they are **async** function with a *yield* statement.

## Conclusions

- Iterators are iterables.
- Iterators are objects that implement the `iterator protocol` consisting in implementing both `__iter__` and `__next__`.
- Iterator iterations stop when `StopIteration` is raised.
- Generators are methods with yield statements.
- Async generators are async methods with yield statements.
- Whenever possible, generator must be the preferred method due to its simplicity, while the protocol implementation gives much more control.

### Resources

- [Iterator types](https://docs.python.org/3.7/library/stdtypes.html#iterator-types)
- [Generator types](https://docs.python.org/3.7/library/stdtypes.html#generator-types)
- [Yield expression](https://docs.python.org/3.7/reference/expressions.html#yieldexpr)
- [Generators and iterators](https://stackoverflow.com/questions/2776829/difference-between-pythons-generators-and-iterators)