An Introduction to Poetry
Tags: python, tutorial, webdev
Published: 2020-04-01 01:13:06

*In this post, I am using `Windows 10`/`Windows Terminal`/`PowerShell 7` with `Python 3.8.1`.  Adjust examples according to your environment.*

*If you don't care about the logic and reasoning behind "why Poetry?", skip to Getting Started where we build a simple Flask app using Poetry as our package manager.*

* [History](#history)
* [Problems](#problems)
* [Installation](#installation)
* [Getting Started](#getting-started)
* [Conclusion](#conclusion)

<a id="history"></a>

## [§](#history) History [↑](#main-title)

Python has existed for almost 30 years now and many revolutions have occurred in the software arena since its inception.  Paradigms have shifted around OOP and functional, design patterns have come and gone, but one issue every language sees, and has varying success in tackling, is dependency management.

For the first half of its life thus far, Python did not have a dedicated way to handle packaging or dependencies.  It was the wild west: relying on what's in the standard library or figuring it out yourself.  It wasn't until Python 1.6 that the `distutils` package was added to the standard library in a first attempt to handle the problem in 2000.  Here are some more milestones [according to PyPA (Python Packaging Authority)][1]:

* 2003: PyPI was operational but wouldn't have "packages" hosted until 2005.
* 2004: `setuptools` is created and is the present packaging go-to.
* 2007: `virtualenv` package is created to segregate packages per-project.
* 2008: `pip` supersedes `easy_install` (from the `setuptools` package).
* 2011: PyPA is created to take over `pip` and `virtualenv` maintenance.
* 2013: PEP 425 and PEP 427 accepted which define the built-package (binary) format, `wheel`.
* 2016: PEP 518 accepted for the `pyproject.toml` format for static build dependency declarations.
* 2017: PEP 517 accepted for `setup.py`-independent (from `setuptools`) build system backends.

There is a lot of history when it comes to Python packaging and decisions, but I tried to highlight the milestones.  The last two are the most relevant to this post as they enabled the `Poetry` project to take off as it is today and move away from the history of `distutils`/`setuptools`.

<a id="problems"></a>

## [§](#problems) Problems [↑](#main-title)

When you start most Python projects, there isn't really guidance on *how* to start.  Most guides used to give you a `requirements.txt` file to define your dependencies that you install using `pip`:

```powershell
PS ~/> pip install --user --requirement requirements.txt
```

You would probably install those in your user site-packages (indicated by `--user` which could cause some chaos with changing versions, conflicting packages, etc.) and off you go.  This was hard to scale and deploy, however, which is why `virtualenv` came along to ease that development process by keeping your dependencies contained per-project and "activating" the environment when you're using it:

```powershell
PS ~/> python -m venv .venv
PS ~/> .venv/Scripts/Activate.ps1
```

Another problem with this approach is that the requirements file is rather simple.  It's typically a newline-separated plaintext file with a package name and some version indicator that a person writes or generates using:

```powershell
PS ~/> pip freeze > requirements.txt
```

But using this approach has its own problems.  It locks all the versions with exact specifiers and their dependencies:

```powershell
PS ~/> pip install flask==1.1.1
# ...
PS ~/> pip freeze
click==7.1.1
Flask==1.1.1
itsdangerous==1.1.0
Jinja2==2.11.1
MarkupSafe==1.1.1
Werkzeug==1.0.0
```

This creates a lot of manual work to audit your dependencies and bring them up-to-date when needed.

A route some people go with is using `setuptools` to manage their project as an installable package.  This is particularly effective when your project has tests and sub-packages, but adds another layer to the stinky onion.

`setuptools` uses a `setup.py` script to control the installation of your package.  This file has one requirement when called: to execute `setuptools.setup()`.

```python
import setuptools

setuptools.setup()  # barebones
```

Normally this function takes arguments to imperatively install the package, but a nice, declarative way to set these arguments is using a `setup.cfg` file.

```
# ...
[options]
install_requires =
    Flask==1.1.1
# ...
```

This file is also used by many packages for configuration, so it's a neat way to centralize your configuration, but it has the same problems as it uses the same format for dependencies.  More information on this method of setup can [be found here][2].

***

This leaves us with the following problems:

* How do we isolate our project?  `virtualenv` is nice, but can be forgotten
* How do we manage our dependencies and their versions?  Locking everything that gets installed is a management nightmare.

One last issue that deserves a little blurb: how do we install our own project/package and use it?  Depending on `PYTHONPATH` is fraught with complications especially when you start talking about testing your code and many a solution has been created to mangle it to fit a use-case perhaps unnecessarily.

<a id="installation"></a>

## [§](#installation) Installation [↑](#main-title)

Poetry's tagline is "Python packaging and dependency management made easy."  It accomplishes this by using features introduced by PEP 517 and PEP 518 introducing independent build system backends and new dependency declaration (away from `pip` and `setuptools`; they're still there, but abstracted away from the user).

What this looks like in action is something that resembles JavaScript's `yarn` tool.  There is the `pyproject.toml` similar to `package.json` and a `poetry.lock` like `yarn.lock`.  There is also configuration of the tool itself kept in `poetry.toml` (again, like `.yarnrc.yml`).

Poetry is available on PyPI, [but they recommend][3] using their alternative installer to avoid polluting your Python packages.

```powershell
# bash
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python

# PowerShell
(Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python

# python
pip install --user poetry
```

<a id="getting-started"></a>

## [§](#getting-started) Getting Started [↑](#main-title)

To the meat and potatoes- using the tool!  Here, the tool creates our project scaffold so the package is installable in a virtual environment:

```powershell
PS ~/> poetry new --name app --src poetry-intro
Created package app in poetry-intro
```

Now that we have a project, I go a step further to create the virtual environment local to the project for easier editor integration:

```powershell
PS ~/> Set-Location -Path poetry-intro
PS ~/poetry-intro/> poetry config --local virtualenvs.in-project true
```

With the configuration set, we're ready to initialize the environment, install our package, and add some dependencies:

```powershell
PS ~/poetry-intro/> poetry install
Creating virtualenv app in ~/poetry-intro/.venv
Updating dependencies
Resolving dependencies...

Writing lock file


Package operations: 11 installs, 0 updates, 0 removals

  - Installing pyparsing (2.4.6)
  - Installing six (1.14.0)
  - Installing atomicwrites (1.3.0)
  - Installing attrs (19.3.0)
  - Installing colorama (0.4.3)
  - Installing more-itertools (8.2.0)
  - Installing packaging (20.3)
  - Installing pluggy (0.13.1)
  - Installing py (1.8.1)
  - Installing wcwidth (0.1.9)
  - Installing pytest (5.4.1)
  - Installing app (0.1.0)
PS ~/poetry-intro/> poetry add flask
Using version ^1.1.1 for flask

Updating dependencies
Resolving dependencies...

Writing lock file


Package operations: 6 installs, 0 updates, 0 removals

  - Installing markupsafe (1.1.1)
  - Installing click (7.1.1)
  - Installing itsdangerous (1.1.0)
  - Installing jinja2 (2.11.1)
  - Installing werkzeug (1.0.1)
  - Installing flask (1.1.1)
```

If you're in an environment with private registries, add them [according to the docs][5] (I could not get them to work when specifying them in `poetry.toml`):

*`~/poetry-intro/pyproject.toml`*

```toml
[[tool.poetry.source]]
name = "private-repo"
url = "https://url"
default = true

[[tool.poetry.source]]
name = "private-repo2"
url = "https://url2"
```

The `default` tells Poetry to ignore the public `PyPI` registry.

Now we'll get to writing code:

*`~/poetry-intro/src/app/__init__.py`*

```python
from flask import Flask

app = Flask(__name__)


@app.route('/')
def index():
    return 'Hello, World!'
```

To execute our new app, we take advantage of the `run` command which executes commands in the context of the virtual environment:

```powershell
PS ~/poetry-intro/> $Env:FLASK_ENV = 'development'
PS ~/poetry-intro/> poetry run flask run
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 000-000-000
```

If you want to work in your environment for an extended period, you may want to use `poetry shell` to spawn a new shell according to `$SHELL` with the environment activated.

***

I usually have some linters and formatters to integrate into intellij/vscode for consistency, so let's add them here as developer dependencies:

```powershell
PS ~/poetry-intro/> poetry add --dev yapf flake8
Using version ^0.29.0 for yapf
Using version ^3.7.9 for flake8

Updating dependencies
Resolving dependencies...

Writing lock file


Package operations: 6 installs, 0 updates, 0 removals

  - Installing entrypoints (0.3)
  - Installing mccabe (0.6.1)
  - Installing pycodestyle (2.5.0)
  - Installing pyflakes (2.1.1)
  - Installing flake8 (3.7.9)
  - Installing yapf (0.29.0)
```

You may notice a lack of test runner, but that was already included in the `pyproject.toml` dependencies when using `poetry new`.  Speaking of which, no project is complete without some tests:

*`~/poetry-intro/tests/test_app.py`*

```python
from app import app

app.testing = True
client = app.test_client()


def test_index():
    response = client.get('/')
    assert b"Hello, World!" in response.data
```

```powershell
PS ~/poetry-intro/> poetry run pytest
======================== test session starts ========================
platform win32 -- Python 3.8.1, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: ~/poetry-intro
collected 1 item

tests\test_app.py .                                            [100%]

========================= 1 passed in 0.30s =========================
```

<a id="conclusion"></a>

## [§](#conclusion) Conclusion [↑](#main-title)

This post is starting to get a bit long, but I feel like I've covered the bases to get started.  There is a lot more to Poetry, such as building packages, deployment to registries, etc., but [the docs are a great location][6] to dig deeper.

In our sample project, we:
* showcased using Poetry for all interactions with the project which made forgetting our virtual environment an impossibility
* saw the lockfile in action which enables repeatable builds and avoids dependency nightmares
* avoided the overhead of `setuptools`/`setup.py` to install our package/module in our environment

I hope this project grows into further maturity and sees more adoption because it could be the catalyst to make `pip` better and/or become the standard like `yarn` has for many JavaScript devs.

[1]: https://www.pypa.io/en/latest/history/
[2]: https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files
[3]: https://python-poetry.org/docs/#installation
[4]: https://github.com/GoogleCloudPlatform/flask-talisman
[5]: https://python-poetry.org/docs/repositories/#using-a-private-repository
[6]: https://python-poetry.org/docs/