Publishing a Python Package: What I Wish the Maze of Tutorials Covered
Tags: python, pip, pypi
Published: 2021-10-16 11:46:45

I've written a number of python packages over time for my own use really and I'm so snowed under just getting it done that every time I stopped to consider sharing one, so it could be `pip` installed, I searched on-line and immediately got lost among the tens, hundreds maybe, of guides, tutorials and options and what looked like enough material for a doctoral dissertation to wade through. And all of it looked scary talking about starting from scratch (not from an existing package), wanting me to add and write a dozen files and understand this and that, and worse there are old ways, new ways, alternate ways ... Aaargh.![Pull my hair out!](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7shtga13bm1getnbj4ca.png)

But hey, we're in a snap 3 day lockdown here (thanks COVID!) and I was tidying up some projects and tried again ... this time, experience in hand, (and less hair) I figured I would put my blinkers on and use _just_ the official tutorial:

https://packaging.python.org/tutorials/packaging-projects/

And I was mildly pleased. I guess my expectations had dropped but hey, it proved manageable and I did succeed in publishing a few packages. But even this tutorial left me wasting far too much time trying to work out how stuff works and I wanted to write it down ASAP, for my own sake, and well, here we are ... share it. 

So here's a better (IMHO) guide to packaging your Python Project and publishing it (better because it says what I wanted to know, and I'm making different mistakes and it has it's own shortcomings that I don't notice ;-).

# 1. Start with a package
Yep, I already have packages ... a good few of them. I want to publish them. Anyhow a package of course is just a folder with an `__init__.py` file in it, and if it's a small package (as many of mine are) that's _all_ they are. Nothing more, nothing less, than a folder with an `__init__.py` in it that provided some classes and/or functions.

Sometimes there might be a few more `.py` files in the folder beside the `__init__.py`. Minor detail, just what happens when it's a little too big to fit conveniently into the one file. 

Key thing here is we're not starting from scratch, but have a package.

# 2. Get the tools
You only need two tools as it happens and they are:

- `build` - which creates from your package by default a `.tar.gz` file and a `.whl` file which is what pip wants/needs for it's `install`ing career.

- `twine` - which publishes to pypi.org

and so for prep:

```py
pip install build twine
```

Not too bad. Step 2 was easy.

# 3. Prep the few extra files a package wants
Not many, don't fret.

- `README.md` - a simple markdown file with a welcome message and whatever you want to add. What is this package, how do you use it? I use [Typora](https://typora.io/) but you can write it in any text editor and it can be as brief or in depth as you like. It is what's shown on the pypi.org page for your package so it's also your ad if you like for your package.

- `LICENSE.md` - Not sure you need it but worth doing and easy as Py. I am beastly careless in this space and just love the [Hippocratic License](https://firstdonoharm.dev). Download the markdown version and save it as `LICENSE.md`

That's it! And to think it seemed so scary in past. 

# 4. But no! There's more - just a little more, not much.

The tutorial recommends that you lay out your folder like this (yes, I've simplified it a bit):

```
the-folder-I-keep-it-in/
├── LICENSE
├── README.md
├── pyproject.toml
├── setup.cfg
└── src/
    └── my_package/
        └── __init__.py
```
The things to note are:

- You don't need to put it in a `src` folder, but why not? If it ain't broke don't fix it. The `src` strategy means you can just drag and drop you package from where it was into src ... done. The stuff above it is the publishing kit ...

- `pyproject.toml` and `setup.cfg` just tell `build` and `twine` what to do. We'll come back to these shortly. The first is just a standard file to tell `build` that we're going to use `twine` in a roundabout way ;-) and the second one describes your package so `twine` can publish it (purists may argue with this neat division, but let them).

- The folder `the-folder-I-keep-it-in` can have any name you like. won't change a thing with the build or publish. I actually call it `my-package` (in this example). As to why, keep reading. It's just convenient that's all.

- The folder `my_package` should use underscores between words, yes, do it. There's a bizarre confusion in the Python world between `my_package` and `my-package`. 

## `my_package` and `my-package`? What, why, when?
This is described nowhere, and I had to work this out with a lot of trial an error and hair pulling alas. But here's what I got for you.

- `my_package`: just stick to this don't waver, never waver, use only this ;-). I kid ye not. Using `my-package` in either the folder under `src` or in `setup.cfg` will cause you grief during the build, publish, install and test. 

- Once it's published it will appear on pypi.org as `my-package` and people will install it with `pip install my-package`, but use it with `import my_package`. That's just the way it is, that's the convention, don't rock the boat, all you need to know is you don't have to lift a finger to make that happen, just stick with `my_package` in the `src` folder and in `setup.cfg`.

- But of course, `the-folder-I-keep-it-in`is irrelevant here and I call it `my-package` just because, because that's what the package is called. The only other exception is the github repo if you're using one (and I do), that too can be `my-package` and is in my case in fact later you'll se I can exploit that for a nice two line install script.

# 5. `pyproject.toml` and `setup.cfg`

`pyproject.toml` is easy. Just copy the standard. Put this in it:

```
[build-system]
requires = [
    "setuptools>=42",
    "wheel"
]
build-backend = "setuptools.build_meta"
```

and be done with it. Ask no more. It's `build` internals and unless you're super keen in digging deeper, let it rest, this just means when you run `python3 -m build` in your package folder, it knows what to do (if you don't have this file it will ask for one). What it does, is created a `dist` folder and drops two files in it. These are what `twine needs`

`setup.cfg` is not hard either and here's my minimalist take and the clarifications that I felt were missing elsewhere:

```
[metadata]
name = my_package
version = 0.1
author = my name
author_email = my email address
description = My little package
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/me/my-package
project_urls =
    Bug Tracker = https://github.com/me/my-package/issues
classifiers =
    Programming Language :: Python :: 3
    License :: Freely Distributable
    Operating System :: OS Independent
    Development Status :: 4 - Beta
    Framework :: Django :: 3.2
    Intended Audience :: System Administrators
    Topic :: Software Development :: Libraries :: Python Modules

[options]
install_requires =
    other_package1 >= 0.1.1
    other_package2 >= 2.0.1
package_dir =
    = src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src
```

And here's what I felt I should have known:

- `name` should use `my_package` not `my-package`. Just believe me. Things go weird if it says `my-package`. Experiment if you like, I wish I didn't need to and the tutorial was clear here.
- `install_requires` wants one indented line per requirement with relatively familiar syntax (similar to `pip freeze` - another one of those mysteriously named python commands that actually means `pip show-me-whats-installed`). This is completely missed in the tutorial. 
- `package_dir` is weird, yes, but forget it. Like `install_requires` it has a list of one liners beneath it, in this case just one. The one liners map package names to folders somehow in the internal complexities of setuptools - details most of don't care about or want to know about when publishing our simple one file package. The tutorial tells us that this line maps the "noname" package to the `src` folder, and that the "noname" package (that nothingness before the = sign) is a code name for the overarching root package, so the `src` folder becomes the mystical "root package". Do most of us actually care about this? What is a "root package"? anyhow. Nah, let's leave it for the boffins, and just accept this is the odd way of telling `build` and/or `twine` that our package is in the `src` folder.
- there's nothing missing after `find:`. No. That's just the syntax, live with it. Refer back to the intro, re: my sentiments on the unnecessary befuddling cryptic nature of Python package publication ... Ditto the `where = src`, just accept it.
- The classifiers are bit fiddly they have to come from the list of [allowed classifiers](https://pypi.org/classifiers/). And they bothersomley lack a clear way of saying you're using the Hippocratic License (which I just happen to love). 

# 6. The Importance and Catches with Testing

Publishing is as simple as:

```
python3 -m twine upload dist/*
```

**BUT**, it's committal. Once you've published there appears to be no way of undoing it **and** it consume the filenames you used (which means also the `version` you have in `setup.cfg` as these get built into the filenames in `dist`).

And so, testing first is critical. And pypi.org provide `testpypi` at https://test.pypi.org/ that you can publish to freely, as often as you need to get it right.

The main things that demand a retry are in my experience:

- You look at it on pypi and `README.md` has issues. Either typos, or code lines that are too long and render badly etc. Either way, you get see how it's going to presented on pypi and can adjust your README to look nice.

- Your test installing it with pip doesn't work. Which actually doens't happen now that I have a workflow, but happened a lot while Iw as trying to work all that `setup.cfg` syntax out that the tutorial deigns to gloss over.

To publish to the test site it's just small variant:

```
python3 -m twine upload --repository testpypi dist/*
```

## The catches

So testing is great. A lifesaver. But it caused me some modest grief too (the flip side of the same coin). 

Firstly you need to create an account on the site, and I did that but use [Bitwarden](https://bitwarden.com/) always, and generate large random passwords for me - a habit (that we should all have). 

`twine` when used as above prompts for username and password. Alas these long random passwords of mine are not easy to type, so I usually do a copy/paste but alas pasting the password does not work - I tried and tried.

Fortunately they can be provided on the command line as in:
```
python3 -m twine upload --repository testpypi -u $username -p $password dist/*
```
and I saved this in a file called `test-publish` that reads:
```bash
#!/bin/bash
source ~/.auth/pypi.auth
python3 -m twine upload --repository testpypi --verbose -u $username -p $password dist/*
```
Secondly, you can't republish. At all. You need to increment the `version` in `setup.cfg` and rebuild before you can republish. Slows things down some. Not least because of the time and energy spent searching online for ways and means to republish. Some [on-line sources](https://stackoverflow.com/a/52130985/4002633) suggest `--skip-existing` does the trick, but it doesn't - not for me and it's not clear what it does or what it's for and maybe I just misread that. C'est la vie.

Thirdly, the dependencies listed under `install_requires` in `setup.cfg` don't work, presumably because, when testing the required packages aren't on https://test.pypi.org/. But it took a bit of head scratching and try and try again to convince myself of that, as I was trying believe it or not to validate the syntax for just that setting as it's not described in the tutorial and sent me looking at that warren of other sources quickly again. I do wish that testpyi would look at pypi for requirements as a fallback so this test cycle could be complete.

# 7. A Standard Workflow

OK, so having gone through that all now, like most folk eventually do, I have a standard template (the last package I published). I now routinely use five tiny little two line shell scripts to make life easy for myself.

Basically a build script, and two publish and install scripts.

In order:

A script to build:
`build`:
```bash
#!/bin/bash
rm dist/*
python3 -m build
```

A script to test publishing:
`test-publish`:
```bash
#!/bin/bash
source ~/.auth/pypi.auth
python3 -m twine upload --repository testpypi  --verbose -u $username -p $password dist/*
```

A script to install the test publish (test installing) - noting that errors here about requirements that cannot be met are expected:
`test-install`:
```bash
#!/bin/bash
package=$(basename $(dirname $(readlink -f "$0")))
python -m pip install --index-url https://test.pypi.org/simple/ $package
```

A script to publish properly:
`publish`:
```bash
#!/bin/bash
source ~/.auth/pypi.auth
python3 -m twine upload --verbose -u $username -p $password dist/*
```

A script to install the package properly:
`install`:
```bash
#!/bin/bash
package=$(basename $(dirname $(readlink -f "$0")))
python -m pip install $package
```

A basic example of that together you can visit at:

https://github.com/bernd-wechner/django-model-admin-fields

and see here:

https://pypi.org/project/django-model-admin-fields/

I hope that helps someone save all the learning hassle, and publish something easily, by just adding 4 files to a folder (a `README.md` to write, a `LICENSE.md` to download, a `pyproject.toml` to copy, and `setup.cfg` to tune) and maybe 5 tiny little helper bash scripts and in no time a test and then publish cycle is underway.