[Python] A Journey to Python Async - 4. Native Coroutines
Tags: webdev, python, network
Published: 2023-07-16 07:42:03

## What is Native Coroutine?
From the last article, we have talked about how generators *could be considered* as coroutines, but **cannot** be fully accepted as coroutines, and since PEP 492 we have finally native coroutine objects in Python.

If you read the [PEP 492](https://peps.python.org/pep-0492/) or the [official docs about the coroutine objects](https://docs.python.org/3/reference/datamodel.html#coroutines), it says that a coroutine object is
- constructed with `async… await…` keywords
-  basically built on generator without such iterator features(no `__next__()` or `__iter__()`), but sharing the interface with generator(such as `send()`, `throw()` or `close()`)
- equipped with `__await__()` such that it is usable with `await` keyword(`__await__()` returns an iterator)

To see what exactly those things mean, we’ll observe a few simple examples that demonstrate the behaviors of native coroutines.

But let’s first look into how the structure of coroutine looks like. 

## Internal Structure of Native Coroutine

In fact, they are not brand new objects, but has an object structure very similar with that of generators, where the core method `send()` is [shared](https://github.com/python/cpython/blob/8c177294899b621fe04ae755abd41b4d319dd4b5/Objects/genobject.c#L1251). So we can expect that the actual logic of a coroutine executed under the hood are somewhat similar to that of a generator, having suspended execution states at some points.

This expectation is in a sense true, although it doesn’t have to be - a native coroutine doesn’t need to suspend at all, but it could. This is largely because native coroutines provide the programmers [freedom to choose what to do with them](https://github.com/python/cpython/blob/8c177294899b621fe04ae755abd41b4d319dd4b5/Objects/genobject.c#L1033) - as long as the objects following the `await` expression are either native coroutines or objects that return iterators through `__await__()`. 

Thus, a coroutine is actually a set of chained coroutines, such that at the bottom level of the chain those awaited objects [must return non-coroutine iterators](https://github.com/python/cpython/blob/8c177294899b621fe04ae755abd41b4d319dd4b5/Objects/genobject.c#L1050). So it is up to the programmer to return generators as those iterators(well, this is why we use native coroutines), but it doesn’t have to(she or he can just return non-generator iterators).

Now let’s observe how native coroutines work using an actual native coroutine object. But let’s briefly discuss `yield from`, which is related to `await`.

### Interlude: `yield from`
While reading PEP 492, you must have come across the syntax `yield from <subgenerator>`. I haven’t had time to discuss this keyword, but it is simply handing in the current control flow to `<subgenerator>`, so that we can have a *chain of generators*. The reason why I introduce this syntax is because it is essential to call another coroutine inside a coroutine using `await`, which - according to PEP 492 - uses the same implementation as `yield from`. 

## Generator-like Behaviors of Coroutine Objects
So we have `send()`, `throw()`, and `close()` for a native coroutine object as well as a generator. Since the docs keep saying “delegating” those methods to the iterator returned from `__await__()`, I guess looking into only `send` method will be sufficient for our purpose.

Let’s augment a little bit of our example(a part of code is from https://stackoverflow.com/a/60118660):

```py
async def await1() -> str:
  print("await 1!!!")
  return "foo"

class Await2:
  def __await__(self) -> str:
    print("before executing Await2")
    val = yield "await from Await2!!!"
    print("value received from send(): ", val)
    val2 = yield "await from Await2!!! - 2"
    print("value received from send(): ", val2)
    return "actual return value"


async def example() -> int:
  print("coroutine runs!")
  print(await await1())
  print("await 1 ended")
  print(await Await2())
  print("Await 2 ended")
  print("end!")
  return 1

if __name__ == "__main__":
  coro = example()
  print("---send None---")
  coro.send(None)

  print("---send 1---")
  coro.send(1)

  print("---send 2---")
  coro.send(2)
```

which will give this result:

```sh
---send None---
coroutine runs!
await 1!!!
foo
await 1 ended
before executing Await2
---send 1---
value received from send():  1
---send 2---
value received from send():  2
actual return value
Await 2 ended
end!
Traceback (most recent call last):
# omitted
```

The result is in line with our [previous discussion about the internal structure of native coroutines](#internal-structure-of-native-coroutine). If a coroutine doesn’t suspend with `yield`(`await1()`), the execution continues up until it hits a `yield` or `return`(`”before executing Await2”`). Whereas generators halt at the next `yield`, native coroutines traverse their chained sub-coroutines(yes, `await` is not `yield` -  it is more like `yield from`, allowing a chained sequence of coroutines). After coming across `yield` in `__await__()`, in `Await2`, the result is almost similar to that of generators. One difference is that we can’t see those yielded values(`”await from Await2!!!”`) outside `__await__()`, but only the returning value(`"actual return value”`). 

So here we can confirm that the coroutine object doesn’t do more than what is [specified in the documentation](https://docs.python.org/3/reference/datamodel.html#coroutines), and it is the programmer who implement the details. Perhaps that is why most of the references(including the ones I recommended in the preface of this series of articles) don’t mention much about these methods of the coroutine object - there is nothing much to talk about!

So our takeaways from the discussion so far are:
- native coroutines are similar in structure with generators: instead of `yield from` they have `await` for chaining executions
- for `await`, not only coroutine for chaining, you can use any `Awaitable` object
- at the bottom of a chaining of coroutines, we have iterators such as generators with `yield`

Therefore you can understand a native coroutine object in Python - under the context of asynchronous programming - as an object with execution information that is chainable with other coroutines and other awaitable objects, to which it yields the current control flow and waits until those awaitable objects returns values, just like generator objects do with `yield from`.

## Conclusion - Now, what remains?
You might have thought why most of the discussions about coroutines and asyncio out there don’t explain much about the native coroutine object but suddenly move to the [asyncio](https://docs.python.org/3/library/asyncio.html) library(or any other similar libraries like [trio](https://trio.readthedocs.io/en/stable/) or [curio](https://curio.readthedocs.io/en/latest/)). Now that makes sense - a native coroutine is simply a frame or an interface, and those libraries fill in the actual behaviors. 

However, it should be daunting for us to investigate even a single library in details. Hence, in the next article, we will pick one of them to briefly sketch how asynchronous workflows could be implemented with several relevant and important concepts involved.