Packaging Python Applications for Debian
Tags: devops, python, deployment, debian
Published: 2020-03-28 11:58:25

---
title: Packaging Python Applications for Debian
published: true
date: 2020-03-28 05:00:00 UTC
tags: devops, python, deployment, debian
series: Deploying Python Software to Production
description: Easily deploy any Python application in form of an ‘omnibus’ Debian package using `dh-virtualenv`.
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/sjw3gv69bhih8rj6r9ha.png
canonical_url: https://jhermann.github.io/blog/python/devops/2020/03/28/dh_virtualenv_howto.html
---

> *Easily deploy any Python application in form of an ‘omnibus’ Debian packaxge, i.e. one that contains all the application's dependencies, just like in a Java WAR. A basic understanding of Debian packaging, the Linux command prompt, and Python tooling is assumed.*


## Introduction

In this article, I'll show how to use `dh-virtualenv` to create self-contained Debian packages to deploy a Python application. The resulting package is very similar to a _executable JAR_ that you can start via `java -jar`, in that it contains all the moving parts except Python itself, without influencing or being influenced by version requirements of other applications. This also frees you from being restricted to the dependencies and their versions found on your target platforms, and makes porting to several different target environments easier.

The advantage of using a Debian package for deployment as opposed to the native Python tool chain is that you are less dependent on typical development tools and services, i.e. to deploy to QA or production environments you need neither Internet access nor any compiler suites (for extension packages). To achieve the same with direct use of `virtualenv` and `pip`, you'd need to have an in-house PyPI repository accessible from production networks, and also release any extension packages as wheels pre-built for the target platform. Removing and updating an application is also much easier with Debian packages.

To use `dh-virtualenv`, you just have to extend your existing application project with a `debian` subdirectory – project meta-data like `pip` requirements and so on will be leveraged to build the final package, i.e. common tasks are delegated to the standard Python eco-system.

Note that just like with any other form of omnibus packaging, you take over the responsibility to release security updates of the contained dependencies in a timely manner.


## How does it work?

`dh-virtualenv` is a _debhelper_ plugin that extends the normal Debian tool chain for package building with the ability to create a Python [virtualenv](https://virtualenv.pypa.io/) (an isolated Python environment), and then wrap that into the final Debian package.

Depending on the details of the application, you often also have to provide some kind of configuration of the software itself, and possibly some means to run it as a service. This can be done in several ways:

- add a `debian/«pkg».install` descriptor to add configuration files to the Debian package.
- provide a Puppet recipe or Ansible playbook that deploys the package and integrates it into the system.
- embed (default) configuration into the application's Python package (via the `include_package_data` option of `setuptools`).

All those can be combined, e.g. provide defaults via Python package data, and then add external configuration that only provides values specific to the concrete host installation.

A real-world example is the [devpi supervisor ERB template](https://github.com/jhermann/devpi-puppet/blob/master/templates/supervisord.conf) that serves both the purpose of passing configuration to the application process (via command line options), and also starting and controlling that process (i.e. handle demonization and automatic startup on boot).


## Installing the build tools

Unsurprisingly, you need to [install dh-virtualenv](http://dh-virtualenv.readthedocs.org/en/latest/tutorial.html#step-1-install-dh-virtualenv) to use it. Since it is architecture independant, you can choose to use a [recent release offered by Debian sid](https://packages.debian.org/sid/dh-virtualenv) whatever your build platform is.

If this is your first time to build a Debian package, you also need to add the basic tools for that:

```sh
sudo apt-get install build-essential debhelper devscripts equivs
```

Finally, to take advantage of the available template for easily adding an inital `debian` directory, [install the cookiecutter tool](https://github.com/Springerle/springerle.github.io#installing-the-cookiecutter-cli). Note that you can opt to [build packages in a Docker container](https://dockyard.readthedocs.io/en/latest/packaging-howto.html) instead, with only Docker as a requirement on your build host.


## Packaging an example project

To add the necessary `debian` directory with minimal effort, you can use the [dh-virtualenv-mold](https://github.com/Springerle/dh-virtualenv-mold#dh-virtualenv-mold) cookiecutter. The following commands basically repeat what the [integration test](https://github.com/Springerle/dh-virtualenv-mold/blob/master/test.sh) script of that project does, namely instantiate a Python project and then add debianization on top of it.

To provide common defaults to `cookiecutter`, it makes sense to have a `~/.cookiecutterrc` file similar to [the one I use](https://github.com/jhermann/ruby-slippers/blob/master/home/.cookiecutterrc).

Let's first create a sample project:

```sh
mkdir -p ~/tmp/dh-venv-blog
cd ~/tmp/dh-venv-blog
cookiecutter --no-input \
    "https://github.com/borntyping/cookiecutter-pypackage-minimal.git"
cd cookiecutter_pypackage_minimal/
python3 setup.py build
```

You can of course also use one of your own, then just check that out instead. Next, we add the `debian` directory:

```sh
cookiecutter --no-input \
    "https://github.com/Springerle/dh-virtualenv-mold.git"
dch -r "" # insert proper date & distro
```

Using `--no-input` causes the template's defaults to be accepted – it avoids answering all the template's prompts. After all, this is just a demo not requiring sensible inputs. Take the time to have a look at what's in the `debian` directory.

You're now able to build the package and if that succeeds, print the contained meta data:

```sh
dpkg-buildpackage -uc -us -b
dpkg-deb -I ../pyvenv-foobar_*.deb
```

The last command should show you something like this:

```ini
Package: pyvenv-foobar
Version: 0.1.0
Architecture: amd64
Maintainer: Jürgen Hermann <jh@web.de>
Installed-Size: 12877
Pre-Depends: dpkg (>= 1.16.1), python3, python3-venv
Section: contrib/python
Priority: extra
Homepage: https://github.com/jschmoe/foobar
Description: A Python package and its dependencies packaged up 
    as DEB in an isolated virtualenv.
```

Finally, install the new package via `dpkg -i`, or upload it to a repository and use it from there with `apt`.


## Real-world examples

These are examples of `dh-virtualenv` packaging for non-trivial applications:

- [devpi](https://github.com/jhermann/devpi-enterprisey/tree/master/debianized-devpi#readme)
- [nodeenv](https://github.com/ekalinin/nodeenv/tree/master/debian-upstream)
- [debianized-sentry](https://github.com/1and1/debianized-sentry#readme)
- [jupyterhub](https://github.com/1and1/debianized-jupyterhub#readme)

The last two show how to integrate Python web applications into `systemd`, instead of using `supervisor` like the `devpi` example. The `jupyterhub` one also demonstrates the integration of a Python project with server-side Javascript (in a NodeJS environment).