Reproducible environment in nbdev with Pipenv
Published: 2023-01-06 21:04:15

-   <a href="#introduction" id="toc-introduction">Introduction</a>
-   <a href="#some-background-on-pipenv"
    id="toc-some-background-on-pipenv">Some background on Pipenv</a>
-   <a href="#creating-a-pipenv-environment"
    id="toc-creating-a-pipenv-environment">Creating a Pipenv environment</a>
-   <a href="#syncing-your-pipfile-to-setup.py"
    id="toc-syncing-your-pipfile-to-setup.py">Syncing your
    <code>Pipfile</code> to <code>setup.py</code></a>
-   <a href="#pre-commit-hook" id="toc-pre-commit-hook">pre-commit hook</a>
-   <a href="#extra-reading" id="toc-extra-reading">Extra reading:</a>

## Introduction

If you’re reading this, you’re probably familiar with
[nbdev](https://nbdev.fast.ai/), a system for creating Python libraries
from Jupyter notebooks.

The only trouble I have with it is keeping track of dependencies,
especially when you have multiple projects or expect other people to use
your code.

In this post, I’ll show you how to set up a new nbdev project using
[Pipenv](https://pipenv.pypa.io/en/latest/), a tool for managing Python
dependencies. This will allow you to easily create a new project,
install dependencies, and run your code without having to worry about
setting up a virtual environment.

## Some background on Pipenv

If you’re not familiar with Pipenv, it’s a tool for managing Python
dependencies. It’s similar to
[virtualenv](https://virtualenv.pypa.io/en/latest/),
[conda](https://docs.conda.io/en/latest/) or
[poetry](https://python-poetry.org/), but it’s a bit more user-friendly.

Setting up a new project with Pipenv is easy. You just need to run
`pipenv install` in the project directory. This will create a new
virtual environment and install all the dependencies listed in the
`Pipfile`.

If you want to add a new dependency, you can run
`pipenv install <package_name>`. This will install the package and add
it to the `Pipfile`, keeping track of the version you installed.

If you want to install a specific version of a package, you can run
`pipenv install <package_name>==<version>`. This will install the
package and add it to the `Pipfile` with the specified version.

## Creating a Pipenv environment

install pipenv:

``` bash
pip install pipenv
```

install some generic dependencies for our project (optional):

``` bash
pipenv install "requests==2.28.1" "pandas==1.5.2"
```

Now install the **required** dependencies for our guide:

``` bash
pipenv install pipenv-setup "vistir==0.6.1" nbdev --dev
```

There is now a `Pipfile` and `Pipfile.lock` in the root of our project.
Let’s add the following to the `Pipfile`:

**Pipfile**

``` python
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "==2.28.1"
pandas = "==1.5.2"

[dev-packages]
nbdev = "*"
pipenv-setup = "*"
vistir = "==0.6.1"

[requires]
python_version = "3.10"
```

Activate your new environment by running:

``` bash
pipenv shell
```

## Syncing your `Pipfile` to `setup.py`

Lucky for us, this has already been solved in
[pipenv-setup](https://github.com/Madoshakalaka/pipenv-setup) package.

First, install it by running

``` bash
pipenv install pipenv-setup "vistir==0.6.1" --dev
```

Now let’s sync our `setup.py` with `Pipfile`:

``` bash
$ pipenv-setup sync --pipfile --dev

No setup() call found in setup.py
can not perform sync
```

Huh, an error. What’s going on here? Let’s check the `setup.py` file
that was generated by nbdev:

**setup.py**

``` python
setuptools.setup(
    ...
    install_requires = requirements,
    extras_require={ 'dev': dev_requirements },
    dependency_links = cfg.get('dep_links','').split(),
    ...
    )
```

The issue is that `pipenv-setup` expects us to call `setup(...)`, not
`setuptools.setup(...)`.

That is easy enough to fix by changing our function call:

**setup.py**

``` python
from setuptools import setup
setup(
    ...
    install_requires = requirements,
    extras_require={ 'dev': dev_requirements },
    dependency_links = cfg.get('dep_links','').split(),
    ...
    )
```

Running the command, we get another error:

``` bash
$ pipenv-setup sync --pipfile --dev

Error parsing setup.py: install_requires is not a list
can not perform sync
```

Let’s make our setup.py compatible with `pipenv-setup` by changing
several arguments, namely:

-   `install_requires` should be a list
-   `extras_require` should be a dictionary
-   `dependency_links` should be a list

**setup.py**

``` python
from setuptools import setup
setup(
    ...
    install_requires=[],
    extras_require={},
    dependency_links=[],
    ...
    )
```

Running again:

``` bash
$ pipenv-setup sync --pipfile --dev

reformatted setup.py
setup.py was successfully updated
5 default packages from Pipfile synced to setup.py
13 dev packages from Pipfile synced to setup.py
```

🎉 SUCCESS! 🎉

Our `setup.py` is now in sync with our `Pipfile`

now every time someone installs our package, they will get the same
dependencies as we do.

Installation with only the default dependencies:

``` bash
pip install -e .

OR

pip install your_package_name # if you've already published your package
```

Installation with all the dependencies:

``` bash
pip install -e '.[dev]'
```

updated setup.py example:

**setup.py**

``` python
    install_requires=["requests==2.28", "pandas", "bleak", "rich", "aranet4"],
    extras_require={
        "dev": [
            "black[jupyter]==22.12.0",
            "blacken-docs==1.12.1",
            "isort==5.10.1",
            "jupyter==1.*",
            "mypy",
            "nbdev==2.3.9",
            "nbqa==1.5.3",
            "pre-commit==2.20",
            "types-requests",
            "types-toml",
            "ipykernel",
            "pipenv-setup",
            "vistir==0.6.1",
        ],
    },
    dependency_links=[],
```

## pre-commit hook

Odds are you’ll forget to run `pipenv-setup sync` every time you add a
new dependency.

To fix this, we can add a pre-commit hook to our project. This will run
`pipenv-setup sync` every time we commit changes to our `Pipfile`.

First, install and activate `pre-commit` in your repo:

``` bash
pipenv install pre-commit --dev

pre-commit install
```

Then, add the following to your `.pre-commit-config.yaml` file:

**.pre-commit-config.yaml**

``` yaml
repos:
  - repo: https://github.com/Madoshakalaka/pipenv-setup
    rev: v3.2.0
    hooks:
      - id: pipenv-setup
        args: [--dev, --pipfile]
        additional_dependencies: ["vistir==0.6.1"]
```

Now on every commit, `pipenv-setup` will check that your `Pipfile` and
`setup.py` are in sync.

If you forget to run `pipenv-setup sync --pipfile --dev`, you’ll get an
error and won’t be able to commit until you fix it.

## Extra reading:

-   [nbdev docs](https://nbdev.fast.ai/)
-   [Pipenv docs](https://pipenv.pypa.io/en/latest/)
-   [setuptools
    docs](https://setuptools.pypa.io/en/latest/userguide/quickstart.html)
-   [pipenv-setup docs](https://github.com/Madoshakalaka/pipenv-setup)