Speed Up Your Python Codebases With C Extensions
Tags: python, c, performance, cpp
Published: 2022-04-23 17:28:31

### Bring the speed of C to your Python programs

![Photo by [PAUL SMITH](https://unsplash.com/@sumo?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/54ny2w90x8xyhpyjrs0e.jpg)

Python, being both easy and powerful, has become one of the most popular programming languages. Nonetheless, it sometimes lacks the much-valued speed of statically typed and precompiled programming languages like C and Java.

## Why is Python Slow?

As you may know, Python is generally implemented through an [interpreter](https://g.co/kgs/eJyYmN). This can cause the code execution to be rather slow compared to languages, like C and Java, with compiled implementations and whose source code is compiled in advance into [machine](https://g.co/kgs/rAZgv2)/[byte](https://g.co/kgs/WuXQ2o) code. However, this topic is beyond the scope of the article.

<br>

## How Can You Speed Up Python Code?

![](https://cdn-images-1.medium.com/max/2000/1*sYcWbMDK22JLyXNIFgQ0FA.png)

Unless you have to perform computationally heavy operations, Python’s speed is not usually a problem. This is where C extensions come in handy.

*C extensions are a way to code functions in C, compile them into a Python module and use them in your source code as a normal Python library.*

Many popular modules are actually written in C or C++ (e.g. [numpy](https://numpy.org/), [pandas](https://pandas.pydata.org/), [tensorflow](https://www.tensorflow.org/)…) for better performance and/or low-level functionalities.

A few quick disclaimers:

* C extensions do not work for Python implementations other than [Cpython](https://en.wikipedia.org/wiki/CPython). This should not be a problem since Cpython is the default one.

* It is recommended you have basic knowledge of the C programming language. However, if you only know Python you should be able to follow along without any problems.

## How to Build a C Extension

As an example, let’s implement the classic `fib(n)` function. `fib(n)` takes in a number, `n`, and returns the corresponding number in the Fibonacci sequence. Then we will compare the performance of the Python and the C version.

First of all, you need the Python C API `Python.h`. It’s a C [header file](https://gcc.gnu.org/onlinedocs/cpp/Header-Files.html#:~:text=A%20header%20file%20is%20a,Header%20files%20serve%20two%20purposes.) that contains everything that’s needed to interface with Python.

### Python API installation

* On Linux you usually have to install the `python-dev` or `python3-dev` package, if not already present. (Note that on some distros the package name may be different)

* By default, if installed with the default installer, Windows should come with Python.

* MacOs should also come with Python. If it does not, `brew reinstall python` should do the trick.

Now, open the code editor of your choice and create the C module file. It should be named by convention — something along the lines of `module_name.c`, although you can name it whatever you want. Here we’ll call it `c_module.c`.

Before you start coding your extension, you need to include some core definitions and declarations.
{% gist https://gist.github.com/nic-obert/8e4a3d1c15bb21dcb3789d1209558647.js %}
It is recommended that you put these lines at the beginning of your file for [compatibility purposes](https://docs.python.org/3/c-api/intro.html#:~:text=Since%20Python%20may%20define%20some%20pre-processor%20definitions%20which%20affect%20the%20standard%20headers%20on%20some%20systems%2C%20you%20must%20include%20Python.h%20before%20any%20standard%20headers%20are%20included.).

Since in Python everything is an object, our `c_fib(n)` function should return one, precisely a `PyObject` pointer (defined in `Python.h`).
{% gist https://gist.github.com/nic-obert/f8529f460fa20f4fc01b6d89e5092392.js %}
Then it’s necessary to *declare which functions to export* from the module to make them accessible to Python.
{% gist https://gist.github.com/nic-obert/ef08f81852cb09342c0edf1c42138819.js %}
### Defining the module methods

Every exported method is represented as a struct comprised of:

* The exported method name (in this case "`c_fib`”)

* The actual method to be exported (`c_fib`)

* The type of arguments the method takes (in this case `METH_VARARGS`). This is from the documentation on `METH_VARARGS`: *“This is the typical calling convention, where the methods have the type `PyCFunction`. The function expects two `PyObject*` values. The first one is the `self` object for methods; for module functions, it is the module object. The second parameter (often called `args`) is a tuple object representing all arguments.”*

* A `const char*` describing the method

### Defining the module

The module is represented as a struct, as shown in the code above. It’s self-documenting, except for the [m_size](https://docs.python.org/3/c-api/module.html#:~:text=PyDoc_STRVAR%20is%20used.-,Py_ssize_t%20m_size,-%C2%B6) argument, which we set to `-1`. From the documentation:
>  Setting `m_size` to `-1` means that the module does not support sub-interpreters, because it has global state.

### Module initialization function

The `PyMODINIT_FUNC` gets called when the module is imported, initializing it. Note that the function name must start with `PyInit_` and end with your module’s name, hence `PyInit_c_module()`.

These are just a few of the Python API’s functionalities. For more information and features visit the API’s [documentation page](https://docs.python.org/3/c-api/index.html).

## Compile Your Extension to a Python Module

Once the C code is complete, you have to compile it to a Python module. Luckily there are a bunch of built-in tools that allow you to do exactly that.

Create a Python script, traditionally named `setup.py`, and insert the following code:
{% gist https://gist.github.com/nic-obert/ab996b2c98a0324e4c46c21d09bf563c.js %}
This script has many functionalities, but we’ll be using only the `build` and `install` commands. For more info, take a look at the [documentation](https://docs.python.org/3/distutils/setupscript.html) or just run the script with the help flag:

     python3 setup.py --help

From the command line run the following:

    python3 setup.py build

This will create a directory named `build` and put the compiled libraries in it. When done, run this:

    python3 setup.py install

This will install the just-built libraries on your system, making them accessible from anywhere.

*Note that you might need root/admin privileges in order to do that. You don’t have to install it system-wide, but if you skip the install process you’ll have to use [relative imports](https://realpython.com/absolute-vs-relative-python-imports/) in order to use the extension.*

## Use C Extensions Inside a Python Program

Inside your Python file, import the newly created module using the name you chose. In our case, it is `c_module`:
{% gist https://gist.github.com/nic-obert/dd305577bbc46362addb5ace13910d39.js %}
As you can see, the extension can be used the same way as any other Python module would.

## How Does It Compare With the Plain Python Version?

Now let’s compare the `c_fib` function with its plain Python counterpart. We’ll use the built-in `time` module:
{% gist https://gist.github.com/nic-obert/3d5737420d4e4404afbc1b4ef52eef91.js %}
Output:

    Input: 5
    py_res=5, py_time=5.245208740234375e-06
    c_res=5, c_time=1.6689300537109375e-06

As expected, the C function is faster.

*Note that you might get different timings on a different machine, but the C version of the same code will always be faster.*

Let’s now try with some larger numbers:

    Input: 10
    py_res=55, py_time=3.147125244140625e-05
    c_res=55, c_time=2.6226043701171875e-06

    Input: 30
    py_res=832040, py_time=0.40490126609802246
    c_res=832040, c_time=0.004115581512451172

    Input: 40
    py_res=102334155, py_time=50.17047834396362
    c_res=102334155, c_time=0.4414968490600586

The C version clearly outperforms the Python one when it comes to large numbers. If you just have to do a few simple calculations, it might not be worth implementing them in C as the performance difference would be minimal. However, given a really time-consuming operation or a function that has to be repeated many times, Python’s speed might not be enough.

*This is where C extensions really shine. You can leave all the heavy lifting to C, but still use Python as your main language.*

## Real-Life Use-Cases for C Extensions

Say you have to perform some heavy calculations, be it for a cryptographic algorithm, deep learning model training, or processing large amounts of data. C extensions can relieve the burden from Python’s vertebrae and speed up your application.

What if you instead wanted to build a low-level interface or work directly on memory from Python? C extensions are the way to go, given you know how to work with raw pointers.

What about optimizing an already-existing Python application that performs badly but you don’t want to (or can’t) rewrite it in another language? C extensions are the answer.

Or what if you’re just an optimization diehard who wants his code to run as fast as possible, but you still want some high-level abstractions for networking, a GUI, and so on. In this case, C extensions are definitely your best friend.
> ###*Time is the thing we always wish we had more of. Invest it wisely.*

## Conclusion

Whether you are a Python developer addicted to performance and efficiency, someone who likes mixing in different technologies, or you just want to experiment with something new, C extensions for Python are a great addition to your developer toolbox. Not only do they provide you with virtually free performance, but they can extend the functionalities of Python, saving it from the obsolete technologies stack.

Thank you for reading.