Extending Python with Go
Tags: go, python, tutorial
Published: 2020-01-27 23:26:24

One of the most interesting features of `CPython` is the ability to add new `built-in modules to Python written in C and C++`. Python provides its API to do that, a set of headers and core types for writing extensions.

In this tutorial I’ll give you an overview of how to extend Python 3 with C, and then how to do the same thing using a modern language like Go.

Go is a valid alternative to C if you want to extend Python: is easier than C, has a Garbage Collector, provides great performances and executing code in parallel is really straightforward with Go routines.

## Setup your environment

This tutorial is based on:

- Python 3.8.1
- Golang 1.13.5

Setup `LIBRARY_PATH` and `PKG_CONFIG_PATH` on your system. I run the code under macOS Catalina 10.15.1 so I have to set

```sh
export PKG_CONFIG_PATH=/Library/Frameworks/Python.framework/Versions/3.8/lib/pkgconfig/
export LIBRARY_PATH=/Library/Frameworks/Python.framework/Versions/3.8/lib
``` 

## Extending Python with C

Suppose you want to create a new module to make sums, something easy like

```py
from newmath import sum

print (sum(5,4))
```

Following the [Python section in the docs](https://docs.python.org/3.8/extending/extending.html), you can write the first extension `newmath.c`

```c
#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject *sum(PyObject *self, PyObject *args) {
    const long a, b;

    if (!PyArg_ParseTuple(args, "LL", &a, &b))
        return NULL;

    return PyLong_FromLong(a + b);
}

static PyMethodDef MathMethods[] = {
    {"sum", sum, METH_VARARGS, "Add two numbers."},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef newmathmodule = {
    PyModuleDef_HEAD_INIT, "newmath", NULL, -1, MathMethods
};

PyMODINIT_FUNC PyInit_newmath(void) {
    return PyModule_Create(&newmathmodule);
}
```

And compile it as a shared library

```sh
gcc -shared -o newmath.so `pkg-config --cflags --libs python3` `python3-config --libs --embed` newmath.c
```

The result is a new module `newmath.so` that can be used in Python code as a module.


## Do the same using Go

Go SDK comes with an amazing toolset called [cgo](https://golang.org/cmd/cgo/) which allows Go programs to interoperate with C and enables you to build shared libraries from Go.

Thanks to the magic C.* namespace is possible to use anything from the C world, so the C code above can be rewritten in Go in this way

```go
package main

// #cgo pkg-config: python3
// #cgo LDFLAGS: -L/Library/Frameworks/Python.framework/Versions/3.8/lib -lpython3.8 -ldl -framework CoreFoundation
// #define PY_SSIZE_T_CLEAN
// #include <Python.h>
import "C"

//export sum
func sum(self, args *C.PyObject) *C.PyObject {
    var a, b C.longlong
    return C.PyLong_FromLongLong(a + b)
}

func main() {}
```
In this case cgo needs a preamble before `import “C”`: it may contain any C code, functions, includes statements, variables declarations and definitions and may then be referred to from Go code as though they were defined in the package “C”. Compile everything with

```sh
go build -buildmode=c-shared -o newmath.so
```

And `newmath` module is ready to be used in Python

```py
from newmath import sum

print(sum(2, 40))
```

It works! ✌🏻

Anyway this is not the method I prefer to extend Python with Go because there are some limitations: `cgo doesn’t support variadic functions`! In this piece of code I don't use `PyArg_ParseTuple` because variadic functions like that need to be wrapped in another function.

```c
int PyArg_ParseTuple_LL(
    PyObject * args,
    long long * a,
    long long * b
) {
    return PyArg_ParseTuple(args, "LL", a, b);
}
```

Macro functions need to be wrapped as well, e.g. `PyLong_Check` (source: include/python3.8/listobject.h).

```c
int is_a_long(PyObject * p) {
    return PyLong_Check(p);
}
```

Another way to extend Python with Go is moving all the Python stuff into C and just call the Go function inside C. To do that, define `sum` function as a normal Go function using `export sum` preamble.


```go
package main

import "C"


//export sum
func sum(a, b int) int {
    return (a + b)
}

func main() {}
```

and then compile everything with

```sh
go build -buildmode=c-archive -o libnewmath.a
```
This command generates two files: 

- `libnewmath.a` we need to link later in the final step
- `libnewmath.h` containing some definitions, including

```c
extern GoInt sum(GoInt p0, GoInt p1);
```

Now rewrite `_newmath.c` (prepend `_` to exclude this file from go build) including `libnewmath.h` and using `sum` function we created with Go

```c
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "libnewmath.h"


static PyObject *sum_wrapper(PyObject *self, PyObject *args) {
    const long a, b;

    if (!PyArg_ParseTuple(args, "LL", &a, &b))
        return NULL;

    return PyLong_FromLong(sum(a, b));
}

static PyMethodDef MathMethods[] = {
    {"sum", sum_wrapper, METH_VARARGS, "Add two numbers."},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef newmathmodule = {
    PyModuleDef_HEAD_INIT, "newmath", NULL, -1, MathMethods
};

PyMODINIT_FUNC PyInit_newmath(void) {
    return PyModule_Create(&newmathmodule);
}
```

And then generate the module `newmath.so`, linking `libnewmath` 

```sh
gcc _newmath.c -shared -o newmath.so `pkg-config --cflags --libs python3` `python3-config --libs --embed` -L . -lnewmath
```

Easier! 😉


## Performances

Having to deal with different universes has a cost in terms of performances: when a Go function is used from another runtime, it spins up the Go runtime in parallel with the caller's runtime getting the goroutine threads, GC and all that other nice stuff that would normally be initialized up when running a Go program on its own. 

Once you cross the boundary, try to do as much on the other side as you can! If you call a Go function inside a Python or C "for" cycle you spin up and destroy Go env on any iteration, it's more performant writing a single Python interface that wraps the execution of the entire "for" cycle inside Go.

When extending Python with Go is faster than using Python? The answer is `parallel execution`!

I tried to make a simple application to countdown from 25000000 to 0 with a thread and do the same thing with another thread. In Python I get the following results:

- Using simple threads ~3.7 sec.
- Using Multiprocessing Pools ~2.4 sec.
- Using Python JobLib ~2.2 sec.

Extending Python with Go and doing the parallel computation inside Go takes only ~0.02 sec.! Python is not performant running CPU-bound multithread programs, it's limited by [Global Interpreter Lock - GIL](https://realpython.com/python-gil/).

You can [check the code on Github](https://github.com/astagi/pygoexamples/tree/master/4_parallel_execution)

## More on cgo

- [cgo documentation](https://golang.org/cmd/cgo/)
- [Adventures With cgo: Part 1 — The Pointering](https://dzone.com/articles/adventures-with-cgo-part-1-the-pointering) by Sean Allen if you have to deal with pointers between C and Go.
- [cgo is not Go](https://dave.cheney.net/2016/01/18/cgo-is-not-go) by Dave Cheney.