Migrating a project to Poetry
Tags: python, packaging, poetry
Published: 2020-10-19 13:30:32

---
title: Migrating a project to Poetry
published: true
date: 2020-10-19 13:30:00 UTC
tags: python, packaging, poetry
canonical_url: https://browniebroke.com/blog/migrating-project-to-poetry/
---

![Poetry Banner](https://dev-to-uploads.s3.amazonaws.com/i/f264nwf29ysewscd6j8g.png)

[Poetry](https://python-poetry.org/) is a tool solving the problem of Python packaging. It was started back in February 2018 by Sébastien Eustace (also the author of [pendulum](https://pendulum.eustace.io/)). It has a beautiful website and a ambitious headline:

> Python packaging and dependency management made easy

It has been on my radar for a while, but I never gave it a proper go. I was happily using [pip-tools](https://github.com/jazzband/pip-tools), which was solving my main use case, while being a lot more lightweight and meant I could keep working with pip since its output is a good old `requirements.txt`. I heard about it a few times online, often next to [Pipenv](https://pipenv.pypa.io), but recently it looks like Poetry got a bit more traction.

Having a bit of time on my hands, a few weeks ago I decided to take a proper look at it and maybe migrate one of my projects, [Deezer Python](https://deezer-python.readthedocs.io).

## Starting point

Before the migration, here is how the package was managed:

- Project metadata were in `setup.cfg`, using [setuptools declarative config](https://setuptools.readthedocs.io/en/latest/setuptools.html#setup-cfg-only-projects).
- Development (and documentation) dependencies managed by [pip-tools](https://github.com/jazzband/pip-tools).
- Documentation hosted on Read the Docs (RTD).
- Releases automated with [Python Semantic Release](https://python-semantic-release.readthedocs.io) (PSR) on Github Actions.
- Development tools configured in `setup.cfg` (black, isort, pyupgrade, flake8).
- Using `tox` to help local testing.
- Using Github Actions for CI.

These are a couple of features that were impacted by migrating to Poetry, and I think it’s worth mentioning them for context. Since Poetry uses `pyproject.toml`, I was also hoping to move all my tools config from `setup.cfg` to that file.

## Migration

### Installing Poetry

The first step is to get the CLI. I initially installed it via Homebrew, but later realised that Poetry was setting some default values based on the Python version its installation uses. As the Homebrew Python can be updated without notice, I realised it was not the best option here, so I later reinstalled it via `pipx` using a Python installation managed by pyenv that wouldn’t be wiped without my knowledge:

```sh
pipx install \
 --python ~/.pyenv/versions/3.8.6/bin/python \
 poetry
```

### Project metadata

The first step was to migrate the project metadata from `setup.cfg` to `pyproject.toml`. Poetry comes with a handy interactive command `poetry init` which will create a minimal `pyproject.toml` for you. I already noticed a few pleasant surprises:

- The CLI was very nice to interact with
- The `author` and `author_email` from `setup.cfg` were merge into an array of `authors`, each with the format `Full Name <email@address.com>`.

I then went on to convert more settings into the new format manually, and the process was quite painless. Many settings have the same name and values, and when they are different, it’s mainly to simplify things. I guess it’s something a library like setuptools cannot easily afford to do due to backwards compatibility, but that a new opt-in tool like Poetry can.

### Dependencies

Adding dependencies and development dependencies was pretty simple, I just needed to run `poetry add [-D] ...` with the list of packages at the end.

In this process, I discovered that one of the development dependency, [`pyupgrade` is not compatible with Python 3.6.0](https://github.com/asottile/scratch/wiki/python-3-statement#360): Poetry would not let me set my own Python to `^3.6`. I initially changed my minimum version to `^3.6.1` as a quick fix, but I realised later that it impacted my package's trove classifiers, the ones generated by Poetry: my package wasn't listed as Python 3.6 compatible. Since this is just for a development dependency, there is a better way! The fix is to specify the dependency conditional to the Python version:

```toml
[tool.poetry.dependencies]
python = "^3.6"
...

[tool.poetry.dev-dependencies]
...
pyupgrade = { version = "^2.7.3", python = "^3.6.1" }
```

With this, the minimum Python version of my package and its classifiers are correct.

### Extra Dependencies

This package had an “extra require” dependency. There is a good [example in the documentation](https://python-poetry.org/docs/pyproject/#extras), these dependencies needs to be specified in the same section as normal dependencies, but as `optional`:

```toml
[tool.poetry.dependencies]
...
tornado = {version = "^6.0.4", optional = true}
```

And a dedicated `pyproject.toml` section maps the extras to an array of optional dependencies:

```toml
[tool.poetry.extras]
tornado = ["tornado"]
```

I got confused initially because I thought it was done via the `poetry add ... -E ...` command. However, it does something different, it’s for the extra of the dependency, not the extra of my own library the dependency should fall under.

For example Django has [2 possible extras](https://github.com/django/django/blob/0eee5c1b9c2e306aa2c2807daf146ee88676bc97/setup.cfg#L52-L54) at the moment, so one would run the following commands to use them:

```sh
poetry add Django -E argon2
```

It means “install Django with the `argon2` extra”. I thought it meant “install Django and put it under the `argon2` extra”.

### Docs dependencies

The dependencies to build the docs were specified in a `requirements.txt` in the `docs/` folder and RTD was configured to pick this up. I initially thought that I wouldn’t be able to remove that file, but it turns out [it’s possible to make it work](https://github.com/readthedocs/readthedocs.org/issues/4912).

Thanks to [PEP 517](https://www.python.org/dev/peps/pep-0517/), which [Poetry is compliant with](https://python-poetry.org/docs/pyproject/#poetry-and-pep-517), you can do `pip install .` in a Poetry package. This has been in pip since 19.0, and pip running on RTD is newer than this. However, this method wouldn’t install your development dependencies, so your docs dependencies cannot be specified as such. It works if you specify a `docs` extra, though:

```toml
# pyproject.toml
[tool.poetry.dependencies]
...
myst-parser = {version = "^0.12", optional = true}
sphinx = {version = "^3", optional = true}
sphinx-autobuild = {version = "^2020.9.1", optional = true}
sphinx-rtd-theme = {version = "^0.5", optional = true}

[tool.poetry.extras]
...
docs = [
    "myst-parser",
    "sphinx",
    "sphinx-autobuild",
    "sphinx-rtd-theme",
]
```

```yaml
# readthedocs.yml
version: 2
python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs
```

The downside of this, is that the extra is part of your package, so not ideal.

## Releases

I recently moved the automation of releases to [Python Semantic Release](https://python-semantic-release.readthedocs.io) which worked well for me, and this would have been a blocker if it wouldn’t work. These are the pieces I needed:

- Move its config to come from `pyproject.toml`.
- Package version is specified in `pyproject.toml` as well as in ` __init__.py`.
- Update build command to use Poetry instead of setuptools.

Here is the PSR config in `pyproject.toml` to achieve that:

```toml
[tool.semantic_release]
version_variable = [
    "deezer/ __init__.py: __version__",
    "pyproject.toml:version"
]
build_command = "pip install poetry && poetry build"
```

I was expecting to have to change more, but it all worked out of the box with just that.

## Linting and code formatting

All the tools I use for linting and code formatting were configured via `setup.cfg` and ideally I’d like to replace it by `pyproject.toml`. It was possible for almost everything, except for flake8 which has [an open issue](https://gitlab.com/pycqa/flake8/-/issues/428) for it.

I decided to move as much things as I could to `pyproject.toml`, and move flake8 config to `.flake8`.

With all the above, I was able to remove the `setup.cfg` as well as all the pip-tools files for dependencies.

## Code Coverage

Pytest had and [unexpected config section](https://docs.pytest.org/en/stable/customize.html#pyproject-toml) for `pyproject.toml`, I didn't pay much attention to it initially and put its config under `[tool.pytest]`, while the section should actually be `[tool.pytest.ini_options]`.

The consequence of this was that my config was ignored and I didn't run the tests with coverage enabled, which silenced another error in the coverage section `tool.coverage.run`, where source should be an array:

```toml{1,6}
[tool.pytest.ini_options]
addopts = "-v -Wdefault --cov=deezer"

[tool.coverage.run]
branch = true
source = ["deezer"]
```

With this, pytest was running with coverage, but for some reason, codecov wasn't picking up the report properly. I noticed that it used to find an `xml` file when it was working, so I edited the command on CI to generate it with `--cov-report=xml`:

```sh
poetry run pytest --cov-report=xml
```

With this changes I eventually got my code coverage reporting back, but it's something I missed in the original migration. Not directly related to Poetry, but interesting that Pytest decided to go with this section name.

## Tox

Poetry works nicely with Tox, I followed the [section in their FAQ](https://python-poetry.org/docs/faq/#is-tox-supported), and here is a overview of the changes:

```ini
[tox]
isolated_build = true
envlist = py36,py37,py38,py39,pypy3,docs,lint,bandit

[testenv]
whitelist_externals = poetry
commands =
    poetry install
    poetry run pytest
...
```

I replaced the `deps` section in each `testenv` by a `poetry install` into the list of commands to run, and prefixed all commands to be run in the isolated environment by `poetry run`.

## Github Actions

Poetry isn’t installed out of the box on Github Actions, one could either install it with a simple `run` step or use [a dedicated action](https://github.com/abatilo/actions-poetry) for it. I’ve opted for the dedicated action, thinking that Dependabot could keep it up to date for me.

The rest of the changes are pretty simple, it’s a matter or replacing `pip install` by `poetry install -E ...` and prefixing all commands by `poetry run`. My docs were tested and I was changing directory with `cd`, I took this opportunity to instead use `working-directory` key to the Github action step.

## Verdict

Did Poetry deliver on its ambitious tagline? I think so, I was really impressed by the developer experience of Poetry, its CLI is really nice, and I hit little issues on the way. Overall the migration was not too difficult, you can check the [pull request](https://github.com/browniebroke/deezer-python/pull/196) on Github. I feel like there are quite a few features I just scratched the surface (like multi-environments). 

I’m going to wait a bit to see how this works in the longer run, but I think I’ll migrate my other projects soon.