Troubleshoot dependencies in Pipfile/pipenv
Tags: python, pipenv, pipfile
Published: 2024-05-09 03:05:00

The Docker build failed in one of my projects on building pip wheel. As a Python newbie, I was not sure what it refers to, all I got is a dependency build stack trace that ends with `AttributeError: cython_sources`. I assume this is one of the packages hidden in the pip dependency tree. Eventually I managed to locate it but it took me for a while, thus I think it might be helpful to note down the troubleshooting process that I went through - at least for future me who has gold fish memory. 

First I scratched the pip surface:

- `pip list` to print the installed dependencies, got a looong list but nothing exactly matched the errored source. (Also tried `pip freeze` but didn't get any luck.)
- searched in Pipfile.lock file for `cython` but nothing popped up. 

Then I went to Uncle Google for `cython` and got this issue from last year: 
https://github.com/yaml/pyyaml/issues/724
Good news - it's closed!

From this issue, I learnt that `cython` is a dependency for `pyyaml`. I probably can look for this thing instead. 

I figured that we can install and print out the pip dependency tree from the [pipfile project](https://github.com/pypa/pipfile), some Q&As on Stack Overflow, and the helpful manual of `pipenv` cli.

1. `pipenv shell` to spawn a shell within the virtualenv;
2. `pipenv verify` to ensure the Pipfile.lock is up-to-date;
3. `pipenv sync -d` to install all packages and *dev packages* specified in the Pipfile.lock;
4. `pipenv graph` to display currently installed dependency tree.

MY Pipfile:

```

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
flask = "*"
requests = "*"

[requires]
python_version = "3.10"

[dev-packages]
pre-commit = "*"
black = "*"

```

The dependency tree I got:

```

$ pipenv graph
black==22.10.0
├── click [required: >=8.0.0, installed: 8.1.3]
├── mypy-extensions [required: >=0.4.3, installed: 0.4.3]
├── pathspec [required: >=0.9.0, installed: 0.10.1]
├── platformdirs [required: >=2, installed: 2.5.2]
└── tomli [required: >=1.1.0, installed: 2.0.1]
Flask==2.2.2
├── click [required: >=8.0, installed: 8.1.3]
├── itsdangerous [required: >=2.0, installed: 2.1.2]
├── Jinja2 [required: >=3.0, installed: 3.1.2]
│   └── MarkupSafe [required: >=2.0, installed: 2.1.1]
└── Werkzeug [required: >=2.2.2, installed: 2.2.2]
    └── MarkupSafe [required: >=2.1.1, installed: 2.1.1]
pre-commit==2.20.0
├── cfgv [required: >=2.0.0, installed: 3.3.1]
├── identify [required: >=1.0.0, installed: 2.5.7]
├── nodeenv [required: >=0.11.1, installed: 1.7.0]
│   └── setuptools [required: Any, installed: 65.5.0]
├── PyYAML [required: >=5.1, installed: 6.0]
├── toml [required: Any, installed: 0.10.2]
└── virtualenv [required: >=20.0.8, installed: 20.16.5]
    ├── distlib [required: >=0.3.5,<1, installed: 0.3.6]
    ├── filelock [required: >=3.4.1,<4, installed: 3.8.0]
    └── platformdirs [required: >=2.4,<3, installed: 2.5.2]
requests==2.28.1
├── certifi [required: >=2017.4.17, installed: 2022.9.24]
├── charset-normalizer [required: >=2,<3, installed: 2.1.1]
├── idna [required: >=2.5,<4, installed: 3.4]
└── urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]

```

Finally I found PyYAML is one of the dependencies of the `pre-commit` dev packages. 

What solved my issue is to re-sync Pipfile.lock file from Pipfile, even tho I didn't update anything in it. Because I use wildcard `*` to pull the latest version for dev packages, when I do `pipenv install`, there are many dependency changes in the lock file. PyYAML has bumped from `6.0` to `6.0.1` - and this fixed the docker build issue. Yay!