Deploying Python projects to air-gapped systems
Tags: python, tutorial, programming, learning
Published: 2023-10-27 09:44:00

## Motivation

A lot of times in the real world we have to deploy Python projects behind a company firewall, restricted systems or even an air-gapped system. This is most certainly a big hassle for most people.
I've found a pretty good setup which allows us to develop using modern tooling like [poetry](https://python-poetry.org/), but also deploy to any system without the need to pull any packages from the internet.

## Project setup

The project is setup as follows

```
airgap/
├── airgap/
│   ├── __init__.py
│   └── main.py
├── poetry.lock
├── pyproject.toml
├── README.md
```

and we're pulling some dependencies into our project (I've used `arrow` which is an awesome date and time manipulation library):

```toml
[tool.poetry]
name = "airgap"
version = "0.1.0"
description = "A project running on an air-gapped system."
authors = ["..."]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
arrow = "^1.3.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

```

We can have as many dependencies as we want, but for demonstration purposes I'll keep it short.

## Preparation before distribution

We'll need to generate a `requirements.txt` file for our next steps. Let's use poetry to do that:

```bash
poetry export -f requirements.txt -o requirements.txt
```

Command breakdown:

- `export` the command used by poetry to convert the `pyproject.toml` file
- `-f`: the format to use, only `contstraints.txt` and `requirements.txt` supported
- `-o`: the name of the output file

After running you'll get a `requirements.txt` similar to this one (except the hashes which I've omitted for brevity):

```
arrow==1.3.0 ; python_version >= "3.10" and python_version < "4.0"
python-dateutil==2.8.2 ; python_version >= "3.10" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.10" and python_version < "4.0"
types-python-dateutil==2.8.19.14 ; python_version >= "3.10" and python_version < "4.0"
```

Now comes the funky part. We'll create wheels from these dependencies and store them locally. We'll use `pip` to get the wheels:

```bash
pip wheel --no-deps --wheel-dir ./wheels -r requirements.txt
```

Command breakdown:

- `wheel`: the `pip` command used to generate wheels [docs](https://pip.pypa.io/en/stable/cli/pip_wheel/)
- `--no-deps`: do not install dependencies (we've already got them all in the `requirements.txt`)
- `--wheel-dir`: the directory to store the wheels (doesn't have to exist, it will be created for you)
- `-r`: which requirements file to use

Our `requirements.txt` includes hashes, which means that `pip wheel` will imply the `--require-hashes` option. In turn this will verify the packages when installing.

Now our directory `wheels` will hold four packages (the ones from the `requirements.txt`). The final part is our own package source. We'll use `poetry` to build the wheel and place it in the `wheels` directory.

```bash
poetry build && mv dist/*.whl ./wheels
```

Command breakdown:

- `build`: the poetry command which builds our project
- `mv`: move files
- `dist/*.whl`: source all files ending in `.whl` in the `dist/` directory
- `./wheels`: the destination for the sourced files

At this point we're done with the preparation. Just zip the `./wheels` directory and distribute it to your air-gapped server. We'll just create a zip of the files:

```bash
zip -r airgap-dist-0.1.0.zip ./wheels/*
```

Command breakdown:

- `-r`: compress the archive
- `iargap-dist-0.1.0.zip`: name of the compressed archive
- `./wheels/*`: the sources to put in the archive

## Deployment

Get your zip to your air-gapped system, unpack it and install. The best-practice I stick to is to use the `/opt` directory to deploy custom software. Let's put it there:

```bash
mkdir /opt/airgap
unzip airgap-dist-0.1.0.zip -d /opt/airgap
```

> Note: you might have to run the `mkdir` command using `sudo`. It's very dependant on your setup, user, OS version etc.

Just as an example you might need to do this:

```bash
sudo mkdir /opt/airgap
sudo chown youruser:yourgroup /opt/airgap
sudo chmod 755 /opt/airgap
```

Now in general I wouldn't advocate to install any project in the global Python site, but since we're striving for simplicity we'll do that. The next step is very easy, install our wheels:

```bash
pip install --no-cache /opt/airgap/wheels/*
```

## Bonus

Just in case you want to use `venv` you can do the following:

```bash
python -m venv /opt/airgap/venv
```

Command breakdown:

- `-m venv`: tell Python to use the module `venv` (should be installed by default)
- `/opt/airgap/venv`: the location where to create the virtual environment, note that the directory name `venv` is just a convention, but you can use any name you like.

Activate the newly created virtual environment:

```bash
source /opt/airgap/venv/bin/activate
```

And then do the install:

```bash
# verify you're using the venv pip
which pip
$ /opt/airgap/venv/bin/pip

# install your packages
pip install --no-cache /opt/airgap/wheels/*
```

> Note: to exit the virtual environment you can run `deactivate` in your terminal. It's a command automatically available as soon as you activate a virtual environment

## Conclusion

We've seen how to extract the dependencies from our new `pyproject.toml` file, build wheels and zip them up for distribution. Consequently how we can deploy our project to our air-gapped system. Happy coding!