Manage your Python Project End-to-End with PDM
Tags: python, programming, packaging
Published: 2023-12-31 17:43:40

* [Installing PDM](#installing-pdm)
* [Starting a New Project](#starting-a-new-project)
* [Setting up a Virtual Environment](#setting-up-a-virtual-environment)
   - [Virtual Environment Auto-creation](#virtual-environment-autocreation)
   - [Creating a Virtual Environment Yourself](#creating-a-virtual-environment-yourself)
   - [Using a Pre-existing Virtual Environment](#using-a-preexisting-virtual-environment)
* [Managing Dependencies](#managing-dependencies)
   - [Syncing Changes](#syncing-changes)
   - [Adding Dependencies](#adding-dependencies)
   - [Removing Dependencies](#removing-dependencies)
   - [Updating Dependencies](#updating-dependencies)
* [PDM Scripts](#pdm-scripts)
* [Building and Publishing Distributions](#building-and-publishing-distributions)
* [A Note from the Author](#a-note-from-the-author)
* [References](#additional-references)

[PDM][_pdm] is a modern, PEP-compliant package and dependency manager, and a powerful development tool that makes managing your Python projects easier from starting to publishing.

## Installing PDM
In order to install PDM you will need a Python environment `>= 3.8`. 

> Note: Most modern systems will probably have a system environment that meets this requirement, but if yours does not or if you prefer not to install anything in your system environment (even  if it's just PDM) check out [asdf][_asdf] or [pyenv][_pyenv] to help install and manage additional Python environments.

The [PDM docs][_pdm_installation] prescribe a few different ways to install PDM, but the ones I have found to be the easiest are these from the [Other installation methods][_pdm_other_installation]:

**Using pip**

```shell
# as a global site-package
pip install pdm

# or as a user-specific site-package
pip install --user pdm
```

**As an asdf plugin**
If you are already using [asdf][_asdf] to manage additional Python environments (or if you just started to after reading the beginning of this section) there's also a PDM plugin for `asdf`:

```shell
asdf plugin add pdm
asdf install pdm latest
```

## Starting a New Project
Initializing a new project is super easy with `pdm init`. Here I start a new project called `"transient-fortitude"`:

```shell
# create my new project's root directory
mkdir transient-fortitude
cd transient-fortitude
# from my new project's root directory
pdm init
```

`pdm init` walks you through an interactive session and then initializes your project structure and metadata for you based on your answers to a series of questions like which interpreter you'd like to use for the project, and whether you'd like to create a virtual environment for the project.

Below is a sample of a session for `transient-fortitude`:

```text
$ pdm init
Creating a pyproject.toml for PDM...
Please enter the Python interpreter to use
0. /home/avlwx/.asdf/installs/python/3.12.0/bin/python (3.12)
1. /home/avlwx/.asdf/installs/python/3.12.0/bin/python3.12 (3.12)
# ... many more, omitted for brevity ...
Please select (0): 0
Would you like to create a virtualenv with /home/avlwx/.asdf/installs/python/3.12.0/bin/python? [y/n] (y): y
Virtualenv is created successfully at /home/avlwx/repos/github/transient-fortitude/.venv
Project name (transient-fortitude): 
Project version (0.1.0): 
Is the project a library that is installable?
If yes, we will need to ask a few more questions to include the build backend [y/n] (n): n
License(SPDX name) (MIT): 
Author name (Ryan Smith): 
Author email (<default from git user.email>): 
Python requires('*' to allow any) (==3.12.*): 
Project is initialized successfully
```

> Note: For the questions with no answer I elected to take the defaults, which PDM displays in parentheses at the end of a given prompt.

Here is the layout of my new project after running through `pdm init` (note: I shortened the full output here for brevity):

```text
$ tree -aL 2 -I '__pycache__'
.
├── .gitignore      # 1
├── .pdm-python     # 2
├── pyproject.toml  # 3
├── README.md       # 4
├── src             # 5
│   └── transient_fortitude
├── tests           # 6
│   └── __init__.py
└── .venv           # 7
```

To summarize, PDM created a project in the `src` layout style including:

1. A boilerplate `.gitignore` that I've found works for many Python projects. You can further customize this `.gitignore` to your needs.
2. Some PDM metadata for tracking a project's current Python interpreter. As a developer you rarely have to worry about this.
3. My project's `pyproject.toml`.
4. The beginning of a project `README`.
5. My project source code tree including an empty top-level package. The top-level package name is derived from the project name, and is normalized to a valid Python package name.
6. An empty `tests` package.
7. A new virtual environment for my project.

> Note: PDM can also create new projects from templates, but that is left as an exercise for the reader. For more information on creating a project from a template check out [Create Project From a Template][_pdm_from_template] in the PDM docs.

Now that I have my initial project structure, I can set up my project for version control:

```shell
# from the project root directory
git init -b main
git add *  # normally should be used with caution, but is helpful here
git commit -m "Initial project structure"
```

And that's it! My new project structure is set up and ready to go. 

## Setting Up a Virtual Environment
In [Starting a New Project](#starting-a-new-project) I talked about how to initialize a project using `pdm init` and showed  that by answering the prompt "Would you like to create a virtualenv with..." with `"y"` PDM creates a new virtual environment for me in my project's root directory.

While this is great for brand new projects it is still generally helpful to know how to use PDM to create new virtual environments in situations where you don't need to start from `pdm init`.

### Virtual Environment Auto-creation
Let's say you've just cloned an existing PDM-managed project that you don't have a local virtual environment for yet. In this case the first time you run `pdm install` (or `pdm sync`) in that project's root directory PDM will automatically create a new virtual environment for that project in `<project root>/.venv` and provision it with that project's dependencies. The interpreter that is used by default depends on what your default global Python interpreter is.

### Creating a Virtual Environment Yourself
Let's say that you want some extra control over the creation a project's virtual environment before provisioning it.  

In this case you can create the virtual environment yourself, and you have several options to do so depending on your needs.

**Unnamed, In-project Virtual Environment**
This is automatically created in `<project root>/.venv` by running:

```shell
pdm venv create
```

**Named Virtual Environment**
You can also give your new virtual environment a name with the `--name` flag:  

```shell
pdm venv create --name transient-fortitude
```

The location where new named virtual environments are managed is controlled by the PDM configuration setting `venv.location`. You can use `pdm config` to find out the value of this setting.

**With Pip Pre-installed**
To have `pip` pre-installed in your project virtual environment without having to add it as a dependency run:

```shell
pdm venv create --with-pip
```

You can configure PDM to automatically add `pip` to your new virtual environments by default and avoid using the `--with-pip` flag everytime by updating the PDM config setting `venv.with_pip` like so:

```shell
pdm config 'venv.with_pip' 'true'
```

As a PyCharm user I have found that having `pip` in my project virtual environment is helpful since PyCharm can have a hard time with project environments where `setuptools` is not installed.

**Matching a Python Version Specifier**
You can specify a Python version as an argument to `pdm venv create` and PDM will resolve a matching interpreter (if it can) and create your virtual environment with it.

```shell
# find a 3.11 interpreter
pdm venv create 3.11

# find specifically a 3.11.4 interpreter
pdm venv create 3.11.4
```

**With a Specific Python Interpreter**
Sometimes you want control over exactly which interpreter is used to create your virtual environment. 

Instead of just a version specifier, you can instead specify an absolute path to a specific interpreter and PDM will create your virtual environment with it.

```shell
pdm venv create /path/to/python
```

> For even more information on working with virtual environments check out [Working with Virtual Environments][_pdm_virtual_environments] in the PDM docs.

### Using a Pre-existing Virtual Environment
If you're used to creating virtual environments by hand using `venv` or `virtualenv` directly and prefer to do this (or if it's muscle memory to the point that you do it without thinking) that's okay! You don't have to go back and remove, then recreate your environment with PDM in order to use PDM for your project.

You can tell PDM which virtual environment to use for your project by running:

```shell
# -i flag to ignore whatever interpreter PDM might already remember
# -f flag to tell PDM to use the first matching interpreter
pdm use -if /path/to/venv/bin/python
# ... proceed to provisioning, etc. ...
```

## Managing Dependencies
In [Starting a New Project](#starting-a-new-project) and [Setting up a Virtual Environment](#setting-up-a-virtual-environment) I showed various ways to create a virtual environment for a project.

Once you have a virtual environment setup you will want to add, remove, and update your project's dependencies as the needs of your project evolve. 

### Syncing Changes
When collaborating on a project with other developers it will be necessary to synchronize your local virtual environment with remote changes to the project's dependency set maintained in your project's `pdm.lock` file.

To synchronize your working set with your project's lockfile, run:

```shell
pdm sync --clean
```

This will ensure any dependencies that have been added or upgraded get added or upgraded, while any dependencies that have been removed are removed.

> Note: without the `--clean` option PDM will not purge your working set of dependencies that have been removed from your project's lockfile.

### Adding Dependencies
To add new dependencies, run:

```shell
pdm add PACKAGE [PACKAGE ...]

# to allow pre-release versions use the option --pre/--prerelease
pdm add --pre PACKAGE [PACKAGE ...]
```

By default, `pdm add` will add your new dependencies to the `pyproject.toml`, perform a `lock` operation, and finally `sync` your working set by installing or updating any packages resolved during the `lock`. This may result in packages already in your project's lockfile being updated unexpectedly. If you'd like to try and prevent unexpected updates you can use the `--update-reuse` flag to tell PDM to reuse the versions of packages already pinned in the lockfile where possible.

### Removing Dependencies
To remove dependencies run:

```shell
pdm remove PACKAGE [PACKAGE ...]
```

### Updating Dependencies
To update your dependencies, run:

```shell
# update all dependencies, including transitive dependencies
pdm update --update-all

# update just specific packages
pdm update --update-reuse PACKAGE [PACKAGE ...]

# update specific packages and their dependency trees
pdm update --update-eager PACKAGE [PACKAGE ...]
```

## PDM Scripts
PDM provides a mechanism to execute arbitrary and user-defined scripts in an environment that is aware of the packages in your project. 

Just pass your command and its options to `pdm run` and PDM will execute that program for you in a dependency-aware environment without having to activate your virtual environment. For those who are familiar with `npm run`, `docker run`, and  `podman run` the concept is similar.

This allows you to do things like:
* Run your application entrypoint scripts without having to activate your virtual environment.
* Tinker in a Python interpreter where all of your project code and its dependencies are loaded.
* Define scripts that can be used both locally and in CI to execute tests or run static analysis tools against your code consistently for both developers and CI systems.

The one I find to be most powerful, though, is this:
> Define scripts that can be used both locally and in CI to execute tests or run static analysis tools against your code consistently for both developers and CI systems.

For example, you can add tools like `bandit` (security scanning), `black` (formatting), `flake8` (linting), `isort` (import management), and `mypy` (type-checking) as development dependencies for your project, then define scripts in your `pyproject.toml` file under the table `[tool.pdm.scripts]` (see example below) to run these tools against your code both locally and in CI! 

```toml
# example user-defined PDM Scripts for developer and CI tools
[tool.pdm.scripts]
check-formatting = { composite = [
    "isort --check --settings-path ./pyproject.toml src/ tests/",
    "black --check src/ tests/"
]}
format = { composite = [
    "isort --settings-path ./pyproject.toml src/ tests/",
    "black src/ tests/"
]}
lint = "flake8 --config .flake8 src/ tests/"
type-check = "mypy --config-file ./pyproject.toml src/"
security-scan = "bandit -rc pyproject.toml src/ tests/"
```

Using these scripts developers can run `pdm format` against their working copy before committing to version control so that it's counterpart `pdm check-formatting` doesn't fail in CI. 

As another example developers can run `pdm type-check` against their working copy before committing to version control, allowing them time to preemptively fix any issues with their type annotations so there are no surprises in CI.

The ability to standardize both CI and developer tools like this prevents issues with flaky CI, or drift between CI and developer tools! No more "it passed on my machine, but fails in CI" and project maintainers don't have to update developer tools in multiple places.

> Note: I first discussed using `pdm run` in order to execute scripts in a package-aware environment, and later omitted `run` in my examples. For user-defined scripts like I showed above PDM provides a convenient shortcut for users that allows you to execute either `pdm run format` or `pdm format` with the same result.

For more information on PDM Scripts check out [PDM Scripts][_pdm_scripts] in the PDM docs. 

## Building and Publishing Distributions
Once my project is in a state where I feel comfortable publishing it to PyPI for all the world to use I can use PDM to help me handle both building and publishing my distribution. 

To build your distribution(s), run:

```shell
# builds both a source and binary distribution
pdm build

# builds just a source distribution
pdm build --no-bdist

# builds just a binary distribution
pdm build --no-sdist
```

By default distributions are built under a directory called `dist/`. If you want to customize the output directory for your distributions use the `-d` flag:

```shell
# build under a directory called "build"
pdm build -d build
```

When you're ready to publish the artifacts you just built to PyPI, just run:

```shell
pdm publish --no-build
```

> Note: If you plan to use `pdm publish --no-build` with your distributions from `pdm build` you should note that `pdm publish` expects the distribution artifacts to be under `dist/`, and as of PDM `2.11.1` this cannot be overridden. Because of this, the `-d` flag for `pdm build` is primarily useful for scratch builds, or builds that are inputs to some middleware that processes artifacts further before publishing them.

If you want to build and publish both a source and binary distribution to PyPI (all the default stuff), you can condense the steps above in to one command:

```shell
pdm publish
```

### Publishing to a Private Registry
If instead you want to publish to a private registry, you can use the `--repository` option, along with `--username` and `--password` to do so:

```shell
pdm publish --no-build \
            --repository <your private registry URL> \
            --username <your registry username> \
            --password <your registry password>
```

> Warning: Make sure any command tracing for your shell is turned off (e.g. `set +x` in `bash`) otherwise you could leak your private registry password to your terminal!

For more information on adding registries and repositories for your project, and maintaining registry and repository secrets as local PDM configuration see [Configure the repositories for upload][_pdm_configure_repositories] in the PDM docs.

## A Note from the Author
This post is informed by my personal experience using PDM for managing my Python projects and is not intended to act as a complete reference for PDM's CLI. If you come across something I haven't mentioned here, it's not because it's not important it's only because I haven't come across it (yet). 

For PDM's complete CLI reference check out [CLI Reference][_pdm_cli] in the PDM docs.

## Additional References
* In a few places I talk about PDM's configuration settings, but never dive in to that in detail. For more information on configuring PDM check out [Configurations][_pdm_configurations] in the PDM docs. 
* This post is primarily informed by my personal experience using PDM for managing my Python projects. However, some of the language I use is informed by the [PDM docs][_pdm]  themselves to ensure consistency. A big shout out to the PDM Project for such an amazing tool!  

[//]: # (Link Table)

[_pdm]: https://pdm-project.org/latest/  
[_asdf]: https://asdf-vm.com/guide/getting-started.html  
[_pyenv]: https://github.com/pyenv/pyenv  
[_pdm_installation]: https://pdm-project.org/latest/#installation  
[_pdm_other_installation]: https://pdm-project.org/latest/#other-installation-methods  
[_pdm_from_template]: https://pdm-project.org/latest/usage/template/#create-project-from-a-template  
[_pdm_virtual_environments]: https://pdm-project.org/latest/usage/venv/  
[_pdm_configurations]: https://pdm-project.org/latest/reference/configuration/  
[_pdm_cli]: https://pdm-project.org/latest/reference/cli/  
[_pdm_scripts]: https://pdm-project.org/latest/usage/scripts/  
[_pdm_configure_repositories]: https://pdm-project.org/latest/usage/config/#configure-the-repositories-for-upload