This post highlights an interesting dichotomy in the Python scientific computing community. Everyone knows that PyPy runs faster than CPython for many common tasks [1].

[1] PyPy is on average 7x faster than CPython: http://speed.pypy.org/

But those in the Python community who are serious about scientific computing (or image processing, like my startup) are already using Numpy & Scipy, which provide hand-coded C implementations of most matrix-related operations. Everyone knows that Python for-loops are "slow" [2], and storing a large 2-D matrix as a list-of-list-of-Python-int-object would require a huge amount of memory & indirection. So, Numpy offers an N-dimensional array type, implemented in C: C arrays of densely-packed C primitive types, with for-loops in C to iterate over the matrix elements. Then Scipy builds a lot of Matlab-like functionality as modules on top of this fundamental Numpy array type.

[2] Python for-loops are slow: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow...

So the most expensive operations in a Python number-crunching program are likely already implemented using Numpy & Scipy operations, which run in compiled C (and additionally, often make use of Blas/Atlas/LAPACK/etc, for even greater speedups in sustained number-crunching).

But unfortunately, Numpy & PyPy do not naturally work together. Being written in C, Numpy makes substantial use of the CPython C-API -- and in fact, Numpy provides its own C-API [3]! The official Numpy package doesn't work with PyPy; the PyPy project very thoughtfully provides its own PyPy-compatible Numpy package [4].

[3] Numpy C-API: http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html

[4] PyPy-compatible Numpy package: http://pypy.org/download.html#installing-numpy

Furthermore, Numpy is fantastic, but it can't offer all possible permutations of matrix operations. In particular, there are certain image-processing operations that are awkward (and thus, computationally-inefficient) to express using Numpy operations. So you might ultimately need to go to the Numpy C-API anyway.

This is why we created (and, just a few days ago, open-sourced) Pymod: https://github.com/jboy/nim-pymod

Pymod is a Nim+Python project that auto-generates all the Python C-API boilerplate & auto-compiles a Python C extension module that wraps the functions in a Nim module. Pymod enables us to write our Numpy array-processing code in Nim, then compile it (for C++-like speeds) as a well-behaved Python module. Nim made this very easy, because it compiles to C.

After considering our Python-integration options (CPython C-API, `ctypes` and `cffi`), we decided to go with the CPython C-API & Numpy C-API. We explained this decision in greater detail in the "Implementation details" section of the Pymod README [5]; the executive summary is that `ctypes` seems better suited to wrapping C types in Python, rather than exposing existing Python types in C, while the CPython C-API code could be generated & compiled with the C code that Nim was going to produce anyway.

[5] Pymod implementation details: https://github.com/jboy/nim-pymod#implementation-details

That said, we would be delighted for Pymod-produced Python modules to be able to run under PyPy. We've been strongly considering implementing a `cffi` back-end for Pymod, but this won't necessarily solve the Numpy issue. It would be even better if PyPy could support all the CPython C-API extension modules in the world in one fell swoop.