I move from pipenv to poetry in 2023 - Am I right ?
Tags: python, devops, architecture
Published: 2023-01-03 14:36:00

Package management in python is evolving continuously. This evolution is strewn with initiatives of all kinds, each with their pitfalls. It's frustrating to see all these ways of doing things coexist, to have to question what we know about the subject every year. At the same time, it is the mark of an active community that is looking for new ways of doing things, an environment that evolves with the times. It reminds us of this long transition that took place over more than 10 years to go from python 2 to python 3. It was a hard choice but come with great benefits.

In 2019, I chose to use the `pipenv + setuptools` combination as a package manager. This combination has had served its time. I decided at the end of 2022 to migrate all the projects I work on to `poetry`.

I still have doubts while writing this article. The purpose of this article is to share my knowledge and my findings at the time of writing. I can be wrong or even make false assumptions. Share in comments if you have a different vision, and share in comments in you know interesting PEP (python enhancement proposal) on this topic not mentioned in this article.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9gatfuid39eg3ogmijss.jpg)

{%- # TOC start -%}
- [What am I expecting from a package manager at the end of 2022?](#what-am-i-expecting-from-a-package-manager-at-the-end-of-2022)
  * [standard manifest](#standard-manifest)
  * [manifest scaffholding](#manifest-scaffholding)
  * [virtual environment (.venv layout)](#virtual-environment-venv-layout)
  * [dependencies locking](#dependencies-locking)
  * [update of one / all dependencies](#update-of-one-all-dependencies)
  * [organization of the source code with src layout](#organization-of-the-source-code-with-src-layout)
  * [sharing a library on pypi](#sharing-a-library-on-pypi)
  * [version management](#version-management)
  * [multiple python projects in a mono repository](#multiple-python-projects-in-a-mono-repository)
- [Moving to poetry](#moving-to-poetry)
{%- # TOC end -%}

## What am I expecting from a package manager at the end of 2022?

A python project without a manifest is a car without air conditioning. After having tasted it, it is difficult to do without it at certain key moments. 

A manifest contains the declaration of dependencies, the version number, the exposed python packages, the development dependencies and a whole bunch of other metadata... The manifest is mandatory to publish a library on pypi. It avoids a lot of headaches when the project grows, for example the hell of relative packages or dependencies that evolve and become incompatible with each other.

```python
from .util import filter
from ..core import local
```

The only problem is that each package manager has its own manifest format. Here are the manifests that you may have come across and which are more or less exclusive: the `requirement.txt`, `pyproject.toml`, `setup.cfg`, `setup.py` and `Pipfile`

> A package manager may transpile its manifest to other formats as needed. For example, I think that poetry transpiles the `pyproject.toml` to `setup.py` manifest to build a wheel, the format expected by pypi to distribute a library, and publish it. It's transparent to us.

In 2022, a package manager is a tool that from a manifest will install dependencies, manage a virtual environment, lock dependencies to a given version, ... Things that at first are optional in python.

Here is a grid analysis that represents my expectations for a package manager in 2023 with a subjective evaluation of the benefits of each combination.

|                                                | Poetry      | pipenv + setuptools | pipenv      | setuptools |
| ---------------------------------------------- | ----------- |-------------------- | ----------- | -----------|
| standard manifest                              | 4           | 4                   | 2           | 5          |
| manifest scaffholding                          | 5           | 1                   | 3           | 0          |
| virtual environment (.venv layout)             | 5           | 3                   | 3           | 0          |
| dependencies locking                           | 5           | 3                   | 3           | 0          |
| update locked dependencies                     | 5           | 2                   | 2           | 0          |
| source code organization (src layout)          | 4           | 4                   | (?)         | 4          |
| publication of library                         | 4           | 3                   | 0           | 4          |
| version management                             | 3           | 4                   | 3           | 4          |
| mono-repository                                | 1           | 1                   | 0           | 0          |

projects that use setuptools : [requests](https://github.com/psf/requests/blob/main/setup.py), [django](https://github.com/django/django/blob/main/setup.cfg), [flask](https://github.com/pallets/flask/blob/main/setup.cfg), [boto3](https://github.com/boto/boto3/blob/develop/setup.py)...
projects that use poetry : [ward](https://github.com/pallets/flask/blob/main/setup.cfg), ... (if you have some with more than 500 stars, share them in comments)
projects that use pipenv : ... (if you have some with more than 500 stars, share them in comments)

`setuptools` is awesome, but it is doing only one things, but it is doing it almost perfectly. Unfortunately, it miss features I want to have in my project as managed virtual environment, locked dependencies, ... I won't talk about it in this article.

### standard manifest

The `pyproject.toml` manifest has been official since PEP 621. It is still young. `poetry` uses this format with its own guidelines. `setuptools` is supported it for few months. The support is pretty good but there is some limitation. I tried this manifest two months ago and I fail to install a python application in editable mode with `pyproject.toml` manifest. I don't know if it's a limitation of `pip` or `setuptools`, but it's a problem for development.

`pipenv` uses the `Pipfile` format. He is the only one to use it. A `Pipfile` manifest can reference a `setup.cfg` or `setup.py` manifest. It is this feature that led me 2 years ago to choose `pipenv + setuptools`.

### manifest scaffholding

I instantiate a manifest with every python project I start. With setuptools, I have to write this manifest by hand. The scaffholding brings together assistants who assist us in writing a manifesto. This is the case with `poetry` with the `init` command. pipenv also offers scaffholding but on the `Pipfile` manifest.

```bash
poetry init
```

The package manager can offer a more advanced scaffholding. `poetry` can declare new dependencies in the manifest with `poetry add`.

```bash
poetry add requests
poetry add --dev pytest
```

The scaffholding is essential for a newcomer who comes from nodejs or java. This is his first contact with the environment. This is the guarantee of creating a python project that can be used by other developers without reading 10 tutorials that contradict each other...

*pyproject.toml manifest*
```toml
[tool.poetry]
name = "myproject"
version = "1.0.0"
description = "this awesome project"
authors = ["Fabien Arcellier <>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.8"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```

### virtual environment (.venv layout)

`pipenv` and `poetry` share the same DNA. The 2 tools bind a virtual environment to a python project. A virtual environment, independent of the python installed on the system, is automatically associated with the manifest and mounted as soon as the user issues a command in the package manager, for example ``poetry install``.

On all the projects that I have initiated for 2 years, I use the `.venv` convention. the virtual environment is created at the root of the python project in `.venv` directory.  `poetry` uses the default .venv layout. As soon as you create a project with `poetry init`, the associated virtual environment is automatically declared in the `.venv` folder next to `pyproject.toml`. This convention allows me to share run configurations in the IDE with other developers, whether on pycharm or vscode. 

```
.
├── pyproject.toml
└── .venv
    ├── bin
    ├── .gitignore
    ├── lib
    └── pyvenv.cfg
```

### dependencies locking

locking dependencies ensures that all dependencies will be reinstalled with the same version regardless of which developer clones the project locally.

I happened to clone a repo several months after the last commit and noticed dependency problems because a 2nd or 3rd level library evolved and this evolution "broke" a level 1 library that uses it. It's a frustrating problem. I had the case this year with [label-studio-converter](https://github.com/heartexlabs/label-studio-converter) one of the repo of the excellent label-studio

Dependency locking is provided by `poetry` and `pipenv`. `poetry` is faster and seems more robust.
`pipenv` fails to install dependencies on machine learning pocs that used pytorch or yolo.

*poetry.lock*
```
[[package]]
name = "altair"
version = "4.2.0"
description = "Altair: A declarative statistical visualization library for Python."
category = "main"
optional = false
python-versions = ">=3.7"

[package.dependencies]
entrypoints = "*"
jinja2 = "*"
jsonschema = ">=3.0"
numpy = "*"
pandas = ">=0.18"
toolz = "*"
```

Locking dependencies is a plus to guarantee the deployment on a PaaS hosting that deploys source code instead of docker image like `heroku`, .... It guarantees the installation in production of the same dependencies that we use locally in our developments.

### update of one / all dependencies

Once the dependencies are locked, we are guaranteed to always install the same versions, even a year later. However, it may become required to update these dependencies. `poetry` does this with `poetry update` command. Note that this command only updates dependencies locked in the `poetry.lock` file and does not touch the `pyproject.toml` manifest.

`poetry` also allows you to update just one library rather than all libraries in the project at once, for example, just `requests`. It allows more flexibility in the update policy, especially on security alerts that can be raised by github for example. This is something I missed with `pipenv+setuptools`. I had to update all the locked dependencies at once.

### organization of the source code with src layout

Relative imports are bad practice in python. This pattern feel wrong, you can import a module of one directory above but not more.

```python
# to avoid
from .util import foo

# to do
from myproject.util import foo
```

The project manifest definitely solves this problem. The package manager installs our project in the virtual environment and allows us to use our modules like the other modules of the standard library from their full path.

For 2 years, I have the impression that the `src` layout takes precedence over other methods of organizing source code such as the `flat layout`. (see discussion [src layout vs flat layout](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/)).

A project organized according to `src layout` implements the packages in a `src` folder. The layout looks like the following.

```
.
├── poetry.lock
├── pyproject.toml
├── README.md
├── src
│   ├── my_project
│   │   ├── __init.py__
│   │   └── util.py 
└── tests
```

Package are declared in the manifest `pyproject.toml` with the directive ``packages``.

*pyproject.toml*
```toml
[tool.poetry]
name = "myproject"
version = "1.0.0"
description = "this awesome project"
authors = ["Fabien Arcellier <>"]
readme = "README.md"
packages = [
  { include = "my_project", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.8"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```

`src layout` seems to be supported by all package managers, including `poetry`. This layout is not linked to a PEP. I don't know yet if this trend will continue. It is similar to what is found in nodejs or java. I find that it facilitates the organization of projects in python, makes it easier to get started with the project. Today, I apply it in libraries, web projects like flask or streamlit but it's one of the trends that could fall. I kept the flat layout only with django. `manager.py` does not work easily with src layout.

### sharing a library on pypi

I think that having the ability to share a native python library helps to democratize python and encourage sharing in the community. Everyone can share their work. The overall quality is not terrible, but this ocean of contribution pulls the ecosystem upwards and encourages its evolution.

`poetry` allows you to publish a library on pypi with the `poetry publish` command. For `pipenv + setuptools` , I had to install `twine` in the development dependencies and use it to orchestrate the release to pypi. Even if the quality is there, I had to find out how to organize my manifesto, how to use twine, how to industrialize it before being able to share my projects.

### version management

The version management in poetry is poorer than with setuptools. You must write the version number in the manifest, not elsewhere. `setuptools` has more options to retrieve this version number, either from a file or from an attribute in a module. This evolution makes all the same sense because the manifest is the ideal place to record a version number.

I use the version number of a python package to automatically create a tag in git and trigger a pipeline in the CI/CD. Here is the snippet, compatible with poetry, that I use to retrieve the version number of a package and then create a corresponding git tag.

```python
import fixtup

ROOT_DIR = os.path.realpath(os.path.join(__file__, "..", ".."))
VERSION = importlib.metadata.version(fixtup.__name__)
```

### multiple python projects in a mono repository

In enterprise repos, I use a mono-repository architecture with multiple python projects in a single repo. Above, I only have partial answers. `poetry` does not offer a workflow for managing multiple python projects. I use either a homemade utility [`alfred-cli`](https://pypi.org/project/alfred-cli/) to manage the mono-repository, bash scripts, or simply a makefile. No solution fully meets my requirements.

At the moment, [`alfred-cli`](https://pypi.org/project/alfred-cli/) is an open source experimental project I have written but its documentation and design is too messy.

## Moving to poetry

In 2022, with the standardization of the `pyproject.toml` manifest, `poetry` stands out as the most robust and easy-to-use package manager, even for a newcomer to the python universe. In all aspects, it is better than  `pipenv` with a developer experience that I find more complete and enjoyable.

|                                                | Poetry      | pipenv + setuptools | pipenv      | setuptools |
| ---------------------------------------------- | ----------- |-------------------- | ----------- | -----------|
| standard manifest                              | 4           | 4                   | 2           | 5          |
| manifest scaffholding                          | 5           | 1                   | 3           | 0          |
| virtual environment (.venv layout)             | 5           | 3                   | 3           | 0          |
| dependencies locking                           | 5           | 3                   | 3           | 0          |
| update locked dependencies                     | 5           | 2                   | 2           | 0          |
| source code organization (src layout)          | 4           | 4                   | (?)         | 4          |
| publication of library                         | 4           | 3                   | 0           | 4          |
| version management                             | 3           | 4                   | 3           | 4          |
| mono-repository                                | 1           | 1                   | 0           | 0          |

`poetry` comes from the experience accumulated by decades of wandering around package management in python. I regret leaving `setuptools`. It is a robust and powerful utility which in its time also made a huge difference. Its development is active and it masterfully ensures compatibility between several generations of manifest. 

I think this is the right direction to take for the next 1-2 years given the current ecosystem. I may have blind spots on this. Did you make the transition yourself ? Did you choose another package manager for your projects ? Share your experience in comments

* cover image by [Eli Digital Creative](https://pixabay.com/users/elifrancis-1160677/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1163695) from [Pixabay](https://pixabay.com//?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1163695)