Private Python Packages With devpi
Tags: python, tutorial, programming
Published: 2023-07-01 11:29:09

There are cases where you want the flexibility of installing a python package via pip without having it available to the open public. This article will focus on using [devpi](https://github.com/devpi/devpi) to provide a self-hosted pip compatible python package server. Ubuntu will be used for the OS as it's a fairly common Linux distribution and easily available on Windows Linux Subsystem. 

## Install

Before beginning we want to make sure that devpi will run as an actual user and not the root process for security purposes:

```
$ sudo useradd -m -s /bin/bash devpi
```

Now I'll switch to that user

```
$ sudo su - devpi
```

Next is to create a virtual environment so devpi and its packages don't clutter the global namespace (you may need to `apt-get install python3-virtualenv` for this to work):

```
$ virtualenv --python=python3.8 venv
created virtual environment CPython3.8.10.final.0-64 in 123ms
  creator CPython3Posix(dest=/home/devpi/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/devpi/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ src venv/bin/activate
```

Finally installation of the devpi server:

```
$ pip install devpi-server devpi-web
```

While `devpi-web` isn't a hard requirement it does enable better visualization of python packages and allows for `pip search` queries to work. Now we'll do install validation and initialize the server:

```
# devpi-server --version
6.9.0
# devpi-init
INFO  NOCTX Loading node info from /root/.devpi/server/.nodeinfo
INFO  NOCTX generated uuid: b7b5b93272124964b10bdb37cce942ad
INFO  NOCTX wrote nodeinfo to: /root/.devpi/server/.nodeinfo
INFO  NOCTX DB: Creating schema
INFO  [Wtx-1] setting password for user 'root'
INFO  [Wtx-1] created user 'root'
INFO  [Wtx-1] created root user
INFO  [Wtx-1] created root/pypi index
INFO  [Wtx-1] fswriter0: committed at 0
```

This is setting up the basic backend for the devpi server to work. It's also creating a root user for later administrative tasks. Next up we'll need to create files which provide several options to run the devpi server:

```
# devpi-gen-config
It is highly recommended to use a configuration file for devpi-server, see --configfile option.
wrote gen-config/crontab
wrote gen-config/net.devpi.plist
wrote gen-config/launchd-macos.txt
wrote gen-config/nginx-devpi.conf
wrote gen-config/nginx-devpi-caching.conf
wrote gen-config/supervisor-devpi.conf
wrote gen-config/supervisord.conf
wrote gen-config/devpi.service
wrote gen-config/windows-service.txt
```

A port can also be provided during execution if the default of 3141 doesn't suit you:

```
# devpi-gen-config --port 4040
```

As you can see there are several methods of running devpi server including cron, launchd (OSX service), nginx, Windows service, and [supervisord](http://supervisord.org/). It also has a `systemd` service file which we can use to manage the service easily as Ubuntu uses it for primary service management. First off though we're going to need a proxy script to ensure that devpi is running in the virtual environment:

`/home/devpi/start-devpi.sh`
```bash
#!/bin/bash
cd $HOME
source venv/bin/activate
devpi-server --restrict-modify=root
```

`--restrict-modify=root` makes it so that the root user is needed to make administrative changes such as index and user creation. Then modify it to be executable by systemd:

```
$ chmod u+x start-devpi.sh
```

Finally we'll adjust the systemd service file:

`gen-config/devpi.service`
```
[Unit]
Description=Devpi Server
Requires=network-online.target
After=network-online.target

[Service]
Restart=on-success
# ExecStart:
# - shall point to existing devpi-server executable
# - shall not use the deprecated `--start`. We want the devpi-server to start in foreground
ExecStart=/home/devpi/start-devpi.sh
# set User according to user which is able to run the devpi-server
User=devpi

[Install]
WantedBy=multi-user.target
```

Now exit out of the devpi user to return back to the normal user you work on Ubuntu with. Then go ahead and:

```
$ sudo cp /home/devpi/gen-config/devpi.service /etc/systemd/system/
$ sudo systemctl enable devpi.service
Created symlink /etc/systemd/system/multi-user.target.wants/devpi.service → /etc/systemd/system/devpi.service
$ sudo systemctl start devpi.service
```

Now the service is available on http://localhost:3141 by default unless the port was changed:

![view of the devpi web interface to validate installation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yroh1alixwvuuw131ysj.png).

## User Management

Before beginning we'll need to install the client which manages everything:

```
$ pip install devpi-client
```

Due to this being a local pip install it will end up in `$HOME/.local/bin` which is generally not in path. I'll modify `~/.profile` to add it:

```
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$HOME/.local/bin:$PATH"
fi
```

then refresh the session:

```
$ source ~/.profile
```

Now devpi needs to know how to communicate with the server. This is done via:

```
$ devpi use http://localhost:3141
using server: http://localhost:3141/ (not logged in)
no current index: type 'devpi use -l' to discover indices
/home/cwprogram/.config/pip/pip.conf: no config file exists
/home/cwprogram/.pydistutils.cfg: no config file exists
/home/cwprogram/.buildout/default.cfg: no config file exists
```

Where 3141 is the port you selected during setup if it's different. It also tries to update related pip config files if they exist, which is something we'll look at later. Now to setup the root password and add a new user:

```
$ devpi login root --password ''
$ devpi user -m root password=[something cool here]
$ devpi user -c cwprogram password=[redacted] email=[redacted]
user created: cwprogram
```

Finally we need to create an index that the user can utilize for uploading and downloading:

```
$ devpi index -c cwprogram/stable bases=root/pypi volatile=True
http://localhost:3141/cwprogram/stable?no_projects=:
  type=stage
  bases=root/pypi
  volatile=True
  acl_upload=cwprogram
  acl_toxresult_upload=:ANONYMOUS:
  mirror_whitelist=
  mirror_whitelist_inheritance=intersection
```

So what's happening here is that `cwprogram/stable` indicates this is a `stable` named index attributed to the `cwprogram` user. `bases=root/pypi` lets the system know to attempt PyPi if a package is not found. `volatile=True` will allow uploads to occur. Now that admin work is done we logoff as root:

```
$ devpi logoff
login information deleted
```

## Project Integration

The next part is going to be from the perspective of an end user. With this in mind I'll be switching to a Windows system for the next part. To facilitate logins that do not require having the password in plain sight I'll be utilizing the python [keyring](https://pypi.org/project/keyring/) project along with [client extensions for devpi](https://pypi.org/project/devpi-client-extensions/). Which allow it to integrate with keyring. As keyring is more of a frontend to several keychain managers, a backend has to be available. Documentation lists this as:

- macOS Keychain
- Freedesktop Secret Service supports many DE including GNOME (requires secretstorage)
- KDE4 & KDE5 KWallet (requires dbus)
- Windows Credential Locker

If the plan is to use this on a command line linux interface then [keyrings.cryptfile](https://pypi.org/project/keyrings.cryptfile) via an encrypted text file or [sagecipher](https://pypi.org/project/sagecipher) via ssh-agent can be used. I'll go ahead and install the devpi client along with the extensions and keyring to get started:

```
> pip install devpi-client devpi-client-extensions[keyring] keyring
> keyring set http://localhost:3141/ cwprogram
Password for 'cwprogram' in 'http://localhost:3141':
```

**Note:** The last `/` in the URL is necessary or keychain won't be picked up

Now if I go to login as the cwprogram user:

```
> devpi login cwprogram
Using cwprogram credentials from keyring
logged in 'cwprogram', credentials valid for 10.00 hours
```

Now I can also use `pip` to authenticate to this server. First I'll have a global setting to use the keyring provider:

```
> pip config set --global global.keyring-provider subprocess
```

Note that using `pip config set --site` with the same options can also be used if you would rather handle this on a per virtual environment project basis. We'll also need to setup keyring again due to `pip` expecting a different host format:

```
> keyring set localhost cwprogram
```

Now to show this works even with a virtual environment:

```
> virtualenv.exe --python=python3.9 venv_devpi
> .\venv_devpi\Scripts\activate
> devpi use cwprogram/stable
> pip install requests --index-url=http://localhost:3141/cwprogram/stable
Looking in indexes: http://localhost:3141/cwprogram/stable
Collecting requests
  Downloading http://localhost:3141/root/pypi/%2Bf/58c/d2187c01e70e6/requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 3.3 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2 (from requests)
  Downloading http://localhost:3141/root/pypi/%2Bf/830/d2948a5ec37c3/charset_normalizer-3.1.0-cp39-cp39-win_amd64.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.1/97.1 kB ? eta 0:00:00
Collecting idna<4,>=2.5 (from requests)
  Downloading http://localhost:3141/root/pypi/%2Bf/90b/77e79eaa3eba6/idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB ? eta 0:00:00
Collecting urllib3<3,>=1.21.1 (from requests)
  Downloading http://localhost:3141/root/pypi/%2Bf/48e/7fafa40319d35/urllib3-2.0.3-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.6/123.6 kB ? eta 0:00:00
Collecting certifi>=2017.4.17 (from requests)
  Downloading http://localhost:3141/root/pypi/%2Bf/c6c/2e98f5c7869ef/certifi-2023.5.7-py3-none-any.whl (156 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.0/157.0 kB ? eta 0:00:00
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2023.5.7 charset-normalizer-3.1.0 idna-3.4 requests-2.31.0 urllib3-2.0.3
```

As shown here the package install is successful. Another interesting feature is the pypi downloads are cached:

```
~/.devpi/server/+files/root/pypi/+f/48e/7fafa40319d35$ ls
urllib3-2.0.3-py3-none-any.whl
```

While the file structure might be different, we haven't done any uploads and yet urllib is on the devpi servers filesystem. As constantly using the `--index-url` option to point to the server is not ideal, `devpi use` can be ran with a `--set-cfg` option to write out local `pip` configuration options for the custom index to be used by default:

```
> devpi use --set-cfg cwprogram/stable
> pip install moto
Looking in indexes: http://localhost:3141/cwprogram/stable/+simple/
Collecting moto
  Downloading http://localhost:3141/root/pypi/%2Bf/6f4/0141ff2f3a309/moto-4.1.12-py2.py3-none-any.whl (3.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 48.2 MB/s eta 0:00:00
Collecting boto3>=1.9.201 (from moto)
...
```

Again it shows up on devpi's local filesystem:

```
$ find . -iname '*moto*'
./+files/root/pypi/+f/6f4/0141ff2f3a309/moto-4.1.12-py2.py3-none-any.whl
```

## Uploading

Now it's time to do the actual uploading part, as that's why most would want a private PyPi index right? To start out I'll clone [tomchen's example PyPi repo template](https://github.com/tomchen/example_pypi_package):

```
> git clone https://github.com/tomchen/example_pypi_package.git
```

Then I'll install python build (the version designation was there because otherwise I got a python version error back):

```
> pip3.9 install build
> python -m build --sdist --wheel .
> dir dist\
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          2023/06/30    22:45           8317 example_pypi_package-0.1.0-py3-none-any.whl
-a---          2023/06/30    22:45           9189 example_pypi_package-0.1.0.tar.gz
```

So there's an sdist (tar.gz) and wheel (.whl) file output from the above command. The devpi client provides a simple way to upload:

```
> devpi upload --sdist --wheel --from-dir dist\
file_upload of example_pypi_package-0.1.0-py3-none-any.whl to http://localhost:3141/cwprogram/stable/
file_upload of example_pypi_package-0.1.0.tar.gz to http://localhost:3141/cwprogram/stable/
> pip install example_pypi_package
Looking in indexes: http://localhost:3141/cwprogram/stable/+simple/
Collecting example_pypi_package
  Downloading http://localhost:3141/cwprogram/stable/%2Bf/318/c1017f3670e4c/example_pypi_package-0.1.0-py3-none-any.whl (8.3 kB)
Installing collected packages: example_pypi_package
Successfully installed example_pypi_package-0.1.0
```

The `pip` install from the private index works! Now to test to make sure the package itself works. I'll run this in a non-source code directory as well:

```
> python
Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from examplepy import Number
>>> n1 = Number(2)
>>> n2 = Number(3)
>>> n1.add(n2)
>>> n1.val()
5
```

[Twine](https://twine.readthedocs.io/) is also an option if you'd rather stick with the traditional PyPi upload system. It does require an additional duplication of the keyring entry and a configuration file addition as well. First I'll create a `$HOME\.pypirc` file (make sure this is Powershell so `$HOME` expands properly) with the following contents:

```
[distutils]
index-servers =
    devpi-stable

[devpi-stable]
repository = http://localhost:3141/cwprogram/stable/
username = cwprogram
```

keyring will need to be set for the value of `repository`:

```
> keyring set http://localhost:3141/cwprogram/stable/ cwprogram
```

From there simply do a twine upload as usual, with a designation to the `devpi-stable` repository:

```
> twine upload -r devpi-stable dist\*
Uploading distributions to http://localhost:3141/cwprogram/stable/
Uploading example_pypi_package-0.1.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 28.4/28.4 kB • 00:02 • ?
Uploading example_pypi_package-0.1.0.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 29.1/29.1 kB • 00:00 • ?
```

The packages also have a nice overview via the web interface:


![example view of the uploaded package in devpi web interace](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j5x8bxyt6sfmqjj9yo4z.png)

## Conclusion

This concludes a look at using `devpi` for a basic standalone python package server. It's nice for a local development box or small teams where the manual user entry is not so much of an issue. That said, it is possible to have LDAP as an authentication backend via [devpi-ldap](https://pypi.org/project/devpi-ldap/) though it will require a fair amount of technical skill to setup (along with running an LDAP server). In the next installment of this series I'll be looking at more user friendly options.