An introduction to packaging in Python
Tags: python, packaging, learning, pip
Published: 2024-03-08 21:54:43

Learning how to package your code is very useful for any python developer. It gives you a better understanding of how Python works and, above all, enables you to share your code with others or simply deploy it in a runtime environment.

So how does it work? Why not just share your code via git? Actually, it's more complex than it sounds. There's even a whole working group, the [Python Packaging Authority](https://github.com/pypa) (aka pypa), which has been working on this subject since [since 2011](https://www.pypa.io/en/latest/history/#before-2013) and it's a constantly evolving field.

In this article, we'll take a quick look at packaging in Python and present a simple method for packaging your code in 2023.

> **TL;DR**
> * Use the [pyproject.toml](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html) file following the [setuptools guide](https://setuptools.pypa.io/en/latest/userguide/quickstart.html).
> * You can simply use `pip install build && python -m build` to build your package.
> * Adopt src-layout.

## Packages in Python

First, a few definitions:

### Python module

A [_module_](https://docs.python.org/fr/3/tutorial/modules.html) python un is a text file with the extension `.py` containing python code.

It's called a _module_ because it "modularizes" the python source code into several `.py` files. We then use python's import functionality to import elements from another module.

For example, if you create a `count_lines.py` file containing a `count_lines_file` function, you can then import this function:

```python
# count_lines.py
def count_lines_file(filepath: str) -> int:
    """Count the number of lines in a file"""
    return sum(1 for _ in open(filepath))
```

```python
from count_lines import count_lines_file
count_lines_file("count_lines.py")
3
```

When you execute the instruction `from count_lines import count_lines_file`, python looks for the module in the current directory (where python was launched); then in the python installation directory `/usr/lib/python3.11`; then in the directory where python installs default packages: `/usr/lib/python3.11/site-packages`.

As soon as the module is found, it is executed in the python environment and the elements defined in it become available.

The list of directories in which python searches for modules is provided by the `sys` package:

```python
import sys
sys.path
["", "/usr/lib/python3.11/python311.zip", "/usr/lib/python3.11/python311", "/usr/lib/python3.11/site-packages"]
```

### Package python

A [_package_](https://docs.python.org/fr/3/tutorial/modules.html#packages) is a folder that groups together a set of python modules and facilitates access to them by creating a namespace: `from numpy.linalg import norm`.

To create a package, simply create a `__init__.py` file (which can be empty) in a folder.
The folder is then considered by python as a package.

Let's create a package for the `count_lines.py` module:

```
count_package
├── __init__.py
└── count_lines.py
```

We can now use "pointed" notation to import the `count_lines` module:

```python
from count_package.count_lines import count_lines_file
count_lines_file("count_lines.py")
3
```

When executing the instruction `from count_package.count_lines import count_lines_file`, python looks for a `count_package` folder containing a `__init__.py` file in the `sys.path` directories (the current directory and the default directories seen above).

If the package is found, the modules present in it can be accessed using the "pointed" notation.

## Distribute your package

To distribute your package, create a _distribution_. This is an archive containing the package to be distributed, which can then be installed using the _pip_ package manager.

There are two main distribution formats:

* The [_Source Distribution_](https://packaging.python.org/en/latest/glossary/#term-Source-Distribution-or-sdist) (sdist) format: this is an archive containing all source code and metadata.
* The [_Built Distribution_](https://packaging.python.org/en/latest/glossary/#term-Built-Distribution) format: this is a distribution format in which a number of things have been pre-compiled to facilitate installation on other environments. This is particularly useful for modules written in C / C++.

The `wheel` format is the reference _Built Distribution_ format. It is the format developed by the Python Packaging Authority and is widely used to distribute packages.

There are many tools available for creating distributions, but here we'll focus mainly on the tools created by the Python Packaging Authority, which have become indispensable: `setuptools`, `build` and `twine`.

### setuptools

[`setuptools`](https://setuptools.pypa.io/en/latest/index.html) is the tool used by the vast majority of projects to build their distributions.

Let's take our `count_package` example from earlier and see how to create a distribution with `setuptools`.

Our project tree might look something like this:

```
projet_genial
├── count_package
│   ├── __init__.py
│   └── count_lines.py
├── tests
│   └── test_count_lines.py
├── .gitignore
├── LICENSE.md
└── README.md
```

`setuptools` needs a configuration file to know what to include in the distribution and all the project metadata.

Unfortunatly there is three configuration files standards currently used (and you can use them all at the same time...) :

* The `setup.py` file, the oldest and most popular
([3.7 million results on github](https://github.com/search?q=filename%3Asetup.py))
* The `setup.cfg` file, the second most popular file ([0.4 million hits on github](https://github.com/search?q=filename%3Asetup.py))
* The `pyproject.toml` file, the latest arrival and now the official standard for all python python packaging tools. ([0.2 million hits on github](https://github.com/search?q=filename%3Asetup.py))

`pyproject.toml` is the [official standard](https://peps.python.org/pep-0621/), but is still in the minority compared to `setup.py`, so we'll be looking at all three file formats.

### setup.py

The `setup.py` file, as its extension suggests, is a python file. It has the following form:

```python
# setup.py
from setuptools import setup

setup(
    name='count_package',
    author='me',
    description='Package for counting the number of lines in files.'
    version='0.0.1',
    python_requires='>=3.7, <4',
    install_requires=[
        'pandas',
        'importlib-metadata; python_version >= "3.8"',
    ],
)
```

The very fact that it's a python file is both its strength and its weakness: it's possible to build the configuration dynamically in the code, but this makes it difficult to parse and interface with other external tools.

In addition, since this format is specific to `setuptools`, distributions of the _sdist_ type can only be installed if setuptools has been installed on the target environment and in a compatible version.

Sadly, since the vast majority of projects started using setuptools and `setup.py`, it became difficult to propose alternatives, so projects like [flit](https://pypi.org/project/flit/) had to be built "on top" of `setuptools`, so the `setup.py` doesn't encourage innovation.

Moreover, its use is often problematic.
To take the example from [this article](https://bernat.tech/posts/pep-517-518/#packaging-tool-diversity), you might be tempted to introduce a _if/else_ condition in your `setup.py` to manage a dependency needed in python 2.7 based on `sys.version`, but in doing so you would be introducing a vicious bug: the dependency will be included or not depending on the environment that compiles the distribution and not depending on the environment that is installing it.

It's also tempting [to import your own package from `setup.py`](https://bernat.tech/posts/growing-pain/#importing-the-built-package-from-within-setuppy) to manage the version. But by doing so, sdist distributions will crash at installation because the package you are trying to import is not yet present in the python environnment.

In short : **please do not use `setup.py` anymore**

And if you do, use it in a declarative way.

Using the file as a script: `python setup.py` is **depreciated**, as the documentation clearly explained in the setuptools documentation :

> It is important to remember, however, that running this file as a script (e.g. python setup.py sdist) is strongly
> discouraged, and that the majority of the command line interfaces are (or will be) deprecated (e.g. python setup.py
> install, python setup.py bdist_wininst, ...).
>
> We also recommend users to expose as much as possible configuration in a more declarative way via the pyproject.toml
> or setup.cfg, and keep the setup.py minimal with only the dynamic parts (or even omit it completely if applicable).
>
> See [Why you shouldn't invoke setup.py directly](https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html)
> for more background.

### setup.cfg

To address the issues mentioned above and make configuration more declarative, in 2016 pypa created the `setup.cfg` file format.

The example `setup.py` file above is equivalent to the following `setup.cfg` file:

```ini
# setup.cfg
[metadata]
name = count_package
version = 0.0.1
author = me
description = Package for counting the number of lines in files.

[options]
python_requires = >=3.7,<4
install_requires =
    pandas
    importlib-metadata; python_version >= "3.8"
```

This format has had many fans but has recently been superseded by the `pyproject.toml` format, which is now the official way of declaring python package configuration.

### pyproject.toml

In addition to adopting the declarative approach of `setup.cfg`, the `pyproject.toml` format introduces a number of new features.

It is now possible (and even mandatory) to specify the package builder.
It is also a means of centralizing the configuration of numerous development tools in an agnostic way, rather than multiplying configuration files such as `tox.ini`, `.coveragerc`, etc.

The `pyproject.toml` format includes a mandatory section to define the builder to be used to build the package :

```toml
# pyproject.toml
[build-system]
requires = [
  "setuptools>=60",
  "wheel>=0.30.0",
  "cython>=0.29.4",
]
build-backend = "setuptools.build_meta"
```

With `pyproject.toml` it is now possible to declare to `pip` the dependencies needed for the build!

It is then perfectly possible to specify the use of a builder other than setuptools, such as flit :

```toml
# pyproject.toml
[build-system]
requires = ["flit"]
build-backend = "flit.api:main"
```

This format is gradually becoming the preferred way of centralizing package configuration.
It is [the format preferred by setuptools](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html) and a number of third-party tools use it to store their configuration: [black](https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#what-on-earth-is-a-pyproject-toml-file), [pytest](https://docs.pytest.org/en/7.2.x/reference/customize.html), [isort](https://pycqa.github.io/isort/docs/configuration/config_files.html#pyprojecttoml-preferred-format), etc.

Here's a sample `pyproject.toml` file for our `count_package` example package:

```toml
# pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "count_package"
version = "0.0.1"
description = "Package for counting the number of lines in files."
name = "my_package"
authors = [
    {name = "me", email = "email@me.fr"},
]
requires-python = ">=3.8,<4"
dependencies = [
    "pandas",
    'importlib-metadata; python_version >= "3.8"',
]
dynamic = ["version"]
```

The [setuptools documentation](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html) gives more details on configuration in `pyproject.toml` format.

### Build a distribution of your package

Once you've written your configuration file (preferably `pyproject.toml`), all that's left to do is build your package.

The modern way to do this is to use the [`build`](https://pypi.org/project/build) package developed by pypa:

```bash
pip install --upgrade build
python -m build
```

`build` will first install the builder specified in your `pyproject.toml` file, then use it to build an sdist distribution and a wheel.

You can then use the [`twine`](https://twine.readthedocs.io/en/stable/) package to publish it on the official [pypi](https://pypi.org/) repository.

> For backward compatibility with older versions of packaging libraries, [you can create a minimal `setup.py` file](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html) in addition to the `pyproject.toml` file:
> ```python
> # setup.py
> from setuptools import setup
> setup()
> ```

### Version management

In the `pyproject.toml` example seen above, the package version is set manually. You therefore need to change it each time you want to publish a new version of your package.

I find the tool [`setuptools-scm`](https://github.com/pypa/setuptools_scm/) very useful for managing package versions using git or mercurial.

This is done very simply by adding the `pyproject.toml` dependency and specifying that the version is dynamic:

```toml
# pyproject.toml
[build-system]
requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2"]

[project]
# version = "0.0.1"  # Remove any existing version parameter.
dynamic = ["version"]

[tool.setuptools_scm]
write_to = "src/pkg/_version.py"
```

When building a package, `setuptools-scm` will search for the last tag with a valid version number and then deduce the package version number. By default, the version is built from three elements:

1. The last tag with a valid version number (example: `v1.2.3`)
2. The distance to this tag (number of revisions since this tag)
3. Working directory status (if there are any uncommitted changes)

Once the version number has been deduced, a `_version.py` file will be created inside the distribution at the specified location (e.g. `src/pkg/_version.py`), allowing the package version to be known from the distribution without the git history being present on the target environment.

> If you are in the habit of entering the version number of your packages yourself, please note that valid version formats are governed by [PEP 440](https://peps.python.org/pep-0440/#version-scheme).
> If you don't comply with these specifications, you're likely to run into problems when publishing or installing your packages.
> In particular, versions `v1.2.3-local` or `v1.2.3-dev` are invalid.


## The layout

When configuring your package, regardless of the method used (`setup.py`, `setup.cfg` or `pyproject.toml`), you must specify the packages **and subpackages** you wish to include in your distribution:

```toml
# pyproject.toml
[tool.setuptools]
packages = ["mypkg", "mypkg.subpkg1", "mypkg.subpkg2"]
```

Fortunately, `setuptools` has an automatic [discovery feature](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html) for your packages and subpackages. This is compatible with two classic project layouts:

flat-layout:

```
count_package
├── count_package
│   ├── __init__.py
│   └── count_lines.py
├── tests
│   └── test_count_lines.py
├── .gitignore
├── LICENSE.md
└── README.md
```

and layout with a src-layout folder:

```
projet_genial
├── src
|   ├── count_package
│   ├── __init__.py
│   └── count_lines.py
├── tests
│   └── test_count_lines.py
├── .gitignore
├── LICENSE.md
└── README.md
```

The difference may seem minimal, but personally I have a strong preference for src-layout because it prevents bad habits and forces you to understand how the package import and installation system
system works in Python.

In fact, when you develop your package, you test the functionalities you add to it as you go along.

To do this, we are very tempted to simply import our package from our :

```python
# test_file.py
from count_package import count_lines
```

This will work if you use a flat-layout and run your test module from the directory containing the `count_package` folder, because as we saw above python includes the current directory in the list of directories where it searches for modules.

However, this is a bad habit for two reasons:

* Firstly, if you use setup.py, which is located at the root of your project, it is able to import the `count_package` package it is supposed to install on a client in sdist mode, which can cause bugs if you're not careful.
* Secondly, **you're not really testing the package as it will be installed on others**! For example, you may not have thought to include data files in your configuration file, and your tests should crash as a result. But since these files are present in your working directory, you won't notice a thing.

For these reasons, I think it's best to opt for a src-layout and use an [editable installation](https://setuptools.pypa.io/en/latest/userguide/development_mode.html) with the command: `pip install -e .` for local development. This allows you to install the package by making a symbolic link with your code, so that any changes you make are immediately reflected in the installed package.

For an in-depth analysis of the benefits of src-layout, I refer you to [this article](https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure) (which dates back to 2014).

## Going further

If you want to learn more about the packaging eco-system I found the resources listed in this article very usefull : [🐍 Best resources on Python packaging 📖](https://dev.to/astrojuanlu/best-resources-on-python-packaging-fea)

In the meantime here is a non-exhaustive list of alternatives to setuptools you should consider :

* [Pipenv](https://pipenv.pypa.io/en/latest/): allows you to jointly manage your project's virtual environment and its dependencies.
dependencies. Adds a valuable feature: generation of `Pipfile.lock` files, which reference
reference exact versions of dependencies to enable identical reproduction of the development environment.
development environment.
![Gif montrant les fonctionnalités de pipenv](https://gist.githubusercontent.com/jlusk/855d611bbcfa2b159839db73d07f6ce9/raw/7f5743401809f7e630ee8ff458faa980e19924a0/pipenv.gif)
* [Poetry](https://python-poetry.org/): a powerful tool for managing virtual environments
environments, dependencies (and dependencies on dependencies), generate a `poetry.lock` file similar to
`Pipfile.lock`, publish your package, etc. However, it does not comply with certain PEP standards.
![Gif showing poetry features](https://raw.githubusercontent.com/python-poetry/poetry/master/assets/install.gif)
* [PDM](https://pdm-project.org/latest/): next-generation package manager for python. Unlike PerformanceEntry, it respects PEP standards.
* [Hatch](https://hatch.pypa.io/latest/): Pypa's new tool for managing python projects. It has many interesting features.
* [uv](https://github.com/astral-sh/uv): pip written in rust to make it 10 to 100 times faster.

### Bonus

* Try a next-gen manager [Hatch](https://hatch.pypa.io/latest/), [PDM](https://pdm-project.org/latest/) or [Poetry](https://python-poetry.org/).
* I'm sharing my simple [template `cookie-cutter`](https://github.com/h4c5/cookie-python-minimal) for your python projects.

### Let's be happy

Everything I've told you here may seem like a lot, and yet I've only skimmed the surface. In any case, I think we can count ourselves lucky when we see what the Sam & Max site wrote in 2018:

> First we have `distutils`, `setuptools`, `distribute`, and `distribute2` which were all at one time the "standards recommended for packaging a lib.
> Then came the days of `eggs`, `exe`, and other stuff that `easy_install` would go and find anywhere in the wild, blindly following links on PyPi. 
> Not to mention the stuff that had to be compile at every turn.
> Besides, nothing was encrypted when downloaded, and `pip` wasn't packaged with Python.
> Python, it was dying on stupid errors like badly managed encoding...
> On top of that, `virtualenv` was a separate thing, with lots of competitors, and linked system packages by default. 
> Not to mention that we didn't have `python -m`.
>
> In short, Python packaging was a real mess. Not to mention a shitty documentation.

It's a lot simpler these days.

## References

1. pypa's guide to python packaging:
[An Overview of Packaging for Python](https://packaging.python.org/en/latest/overview/)
2. An article on wheels:
[What Are Python Wheels and Why Should You Care?](https://realpython.com/python-wheels/#python-packaging-made-better-an-intro-to-python-wheels)
3. A series of three very enlightening articles on how Python packaging works, written by Bernát Gábor in
2019 :
    - [The state of Python Packaging](https://bernat.tech/posts/pep-517-and-python-packaging/)
    - Python Packaging - Past, Present, Future](https://bernat.tech/posts/pep-517-518/)
    - [Python packaging - Growing Pains](https://bernat.tech/posts/growing-pain/)
4. An article in praise of `setup.cfg` on the late Sam & Max site:
[about setup.cfg](https://sametmax2.com/vive-setup-cfg-et-mort-a-pyproject-toml/index.html)
5. Stackoverflow question:
[What is pyproject.toml file for](https://stackoverflow.com/questions/62983756/what-is-pyproject-toml-file-for)