Homebrew, pyenv, ctypes... oh my!
Tags: homebrew, python, pyenv
Published: 2022-01-01 02:10:57

---
title: Homebrew, pyenv, ctypes... oh my!
published: true
description: After "fixing" my most recent Homebrew-related build failure, I felt a nagging bit of lingering confusion. Time to explore exactly why the fix fixed anything!
tags: homebrew, python, pyenv
---

- [Background](#background)
- [Skipping to the End](#skipping-to-the-end)
- [The Problem](#the-problem)
- [The Solution?](#the-solution)
- [The Itch](#the-itch)
- [Isolating the Useful Change](#isolating-the-useful-change)
  - [One Level Down](#one-level-down)
  - [Another Level Down](#another-level-down)
  - [Finally An Answer](#finally-an-answer)
- [In Other, Fewer Words...](#in-other-fewer-words)
- [Lingering Questions](#lingering-questions)

## Background

I usually have a number of Python versions, environments and tools in play. I generally use [pyenv](https://github.com/pyenv/pyenv) to manage Python versions, and [Homebrew](https://docs.brew.sh/Homebrew-on-Linux) to manage pyenv.

## Skipping to the End

If you've come across this post because you're having problems installing Python via Homebrew-managed pyenv, you have my sympathies. And in case it "just works" for you, feel free to give something like this a try:

```
CC="$(brew --prefix gcc)/bin/gcc-11" \
pyenv install --verbose 3.10.0
```

If you're interested in the circuitous road of troubleshooting and exploration that led to that command, you're welcome to keep reading and come along for the ride. Nice to have company :).

## The Problem

After successfully installing Python 3.10 with pyenv, I found that I couldn't `pip install` packages with C extension modules. They complained about `_ctypes` being missing. And trying to `import ctypes` from the Python REPL confirmed the issue:

```
Python 3.10.0 (default, Dec 31 2021, 16:09:32) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/python-build.20211231145343.269699/Python-3.10.0/Lib/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
```

Looking at the logs of my `pyenv install` command, I found this error:

```
*** WARNING: renaming "_ctypes" since importing it failed: libffi.so.8: cannot open shared object file: No such file or directory
```

Which took things a bit further and pointed at an underlying problem. Now, I've seen that issue (or one of its kin) before, and it usually means something like "go muck with LDFLAGS until it works". And after a bit of searching I found a number of relevant resources with plenty of suggestions:

- https://stackoverflow.com/questions/27022373/python3-importerror-no-module-named-ctypes-when-using-value-from-module-mul
- https://github.com/pyenv/pyenv/issues/2159
- https://github.com/pyenv/pyenv/issues/2137

Along with the always-helpful [suggested build environment](https://github.com/pyenv/pyenv/wiki#suggested-build-environment) information from the pyenv wiki.

That was plenty of information to fuel a few cycles of "try stuff and see what happens".

## The Solution?

Exploring those linked issues was productive, but my Python environment was still broken. I also wasn't sure why my previous pyenv-installed versions of Python could import ctypes just fine.

But I noticed that many of the suggested fixes worked because they took homebrew out of the equation. What I really wanted was to use the build tooling Homebrew already had installed. And so I thought "Why am I passing Homebrew paths in for CFLAGS and LDFLAGS but still letting `pyenv install` use my system gcc?". So I tried:

```
CC="$(brew --prefix gcc)/bin/gcc-11" \
CFLAGS="$(pkg-config --cflags libffi)" \
LDFLAGS="$(pkg-config --libs libffi)" \
pyenv install --verbose 3.10.0
```

And it worked! I commented about it [in this issue](https://github.com/pyenv/pyenv/issues/2159#issuecomment-983960026) because I suspected other folks with a similar problem might benefit from the same solution. Then I went back to whatever the heck I was trying to do _with_ Python in the first place :).

## The Itch

The "solution" got my Python 3.10 environments working perfectly, but it also felt unsatisfying. I had some hunches about why it worked, but I wasn't _really_ sure. And then I saw...

- A few other folks hit the same issue, and note that my suggestion worked for them
- A really great [post](https://jvns.ca/blog/2021/11/17/debugging-a-weird--file-not-found--error/) from Julia Evans debugging a very similar issue

Which made me think, "This is a great problem to explore during the last week of December".

## Isolating the Useful Change

I had been thinking that since adding a custom `CC` effectively fixed my Python build, the `CFLAGS` and `LDFLAGS` bits might be unnecessary. So I tried without them:

```
CC="$(brew --prefix gcc)/bin/gcc-11" \
pyenv install --verbose 3.10.0
```

And yes, that still worked. But why?

### One Level Down

The next step was to drop down a level and [use python-build](https://github.com/pyenv/pyenv/tree/master/plugins/python-build#using-python-build-standalone) to set up side-by-side build directories: one working, one broken. To get a broken one, I used:

```
python-build --keep --verbose 3.10.0 ./py310
```

And for a working one:

```
CC="$(brew --prefix gcc)/bin/gcc-11" \
python-build --keep --verbose 3.10.0 ./py310-2
```

From the broken directory, I could run `make` and see the ctypes warning. I could also see the object file renamed to `..._failed.so`:

```
❯ fd "ctypes.*so"
build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu_failed.so
build/lib.linux-x86_64-3.10/_ctypes_test.cpython-310-x86_64-linux-gnu.so
```

Compared to the working directory which showed the expected file names:

```
❯ fd "ctypes.*so"
build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.10/_ctypes_test.cpython-310-x86_64-linux-gnu.so
```

The earlier error messages mentioned an error finding `libffi`, and `ldd` helped highlight and reproduce that. From my broken build, I see "not found":

```
> ldd build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu_failed.so
        linux-vdso.so.1 (0x00007ffd97fbf000)
        libffi.so.8 => not found
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f478258f000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f478256c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f478237a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f47825d8000)
```

While the working file references homebrew's libffi:

```
> ldd build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffcd35fb000)
        libffi.so.8 => /home/linuxbrew/.linuxbrew/lib/libffi.so.8 (0x00007f306da2d000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f306da08000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f306d9e5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f306d7f3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f306dc5f000)
```

I was providing custom CFLAGS and LDFLAGS options that I _thought_ would avoid this issue, but obviously that didn't do it. What was that custom CC doing that the CFLAGS/LDFLAGS change didn't do?

### Another Level Down

I looked in the python build log to get the gcc command that built `_ctypes.cpython-310-x86_64-linux-gnu.so`. It looked something like this (noise alert):

```
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/home/linuxbrew/.linuxbrew/opt/zlib -I/home/linuxbrew/.linuxbrew/opt/zlib -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I/home/linuxbrew/.linuxbrew/Cellar/libffi/3.4.2/include -I./Include -I. -I/home/linuxbrew/.linuxbrew/opt/readline/include -I/home/aj/code/scratch/pyenv-builds/./py310/include -I/home/linuxbrew/.linuxbrew/include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/tmp/python-build.20211231172024.425603/Python-3.10.0/Include -I/tmp/python-build.20211231172024.425603/Python-3.10.0 -c /tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/stgdict.c -o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/stgdict.o -DPy_BUILD_CORE_MODULE -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1
gcc -pthread -shared -L/home/linuxbrew/.linuxbrew/opt/readline/lib -L/home/aj/code/scratch/pyenv-builds/./py310/lib -L/home/linuxbrew/.linuxbrew/lib build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/_ctypes.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/callbacks.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/callproc.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/cfield.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/stgdict.o -L/home/linuxbrew/.linuxbrew/opt/readline/lib -L/home/aj/code/scratch/pyenv-builds/./py310/lib -L/home/linuxbrew/.linuxbrew/lib -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -lffi -ldl -o build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu.so
```

There's a lot going on there, but as a test I ran the same command with all the same command line options, but using Homebrew's gcc-11 instead of my system gcc. And... that still worked. That was another helpful input.

### Finally An Answer

Once I found that I could fail or succeed to build `_ctypes.cpython-310-x86_64-linux-gnu.so` by varying only the gcc version, I ran both versions of gcc with the `--verbose` option to look for more clues. And I found my answer in the form of gcc specs. The verbose logs from my system gcc included this bit:

```
Using built-in specs.
```

While the Homebrew gcc log included:

```
Reading specs from /home/linuxbrew/.linuxbrew/Cellar/gcc/11.2.0_3/lib/gcc/11/gcc/x86_64-pc-linux-gnu/11/specs
```

And peeking into `/home/linuxbrew/.linuxbrew/Cellar/gcc/11.2.0_3/lib/gcc/11/gcc/x86_64-pc-linux-gnu/11/specs` revealed some custom options:

```
*cpp_unique_options:
+ -isysroot /home/linuxbrew/.linuxbrew/nonexistent -idirafter /home/linuxbrew/.linuxbrew/include -idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include

*link_libgcc:
+ -L/home/linuxbrew/.linuxbrew/lib/gcc/11 -L/home/linuxbrew/.linuxbrew/lib

*link:
+ --dynamic-linker /home/linuxbrew/.linuxbrew/lib/ld.so -rpath /home/linuxbrew/.linuxbrew/lib/gcc/11

*homebrew_rpath:
-rpath /home/linuxbrew/.linuxbrew/lib
```

I've never messed around with gcc spec files to be honest. But it sure looks like that's what was causing my Homebrew gcc to work where my system gcc didn't.

## In Other, Fewer Words...

My condensed takeaway after all of this exploration and troubleshooting is:

I'm trying to install Python using Homebrew-managed build dependencies. Those tools know where their pieces live and how they should work together. I should get out of their way and let them work. By pointing at Homebrew's gcc with `pyenv install`, I'm letting pyenv operate in the world it knows. The world where it was born.

## Lingering Questions

There are some things I'm still not sure about. If you're reading this, you are a lovely human and a patient angel. And maybe you can help shed additional light on my busted system? Notably...

* My Homebrew-managed pyenv on this system previously installed Python versions from 3.6 through 3.9 without any issues and without requiring any CC-fiddling. Did something change with Homebrew or pyenv, or is this possibly something specific to Python 3.10?
* The options in the gcc spec look similar to the CFLAGS and LDFLAGS I was trying... is there some other magic combination of flags that would have allowed my system gcc to find everything it needed?

I may come back to these questions myself in a later troubleshooting phase. But I suspect I'm missing something obvious that someone else would catch at a glance!