Help, there's a snake in my pipeline
Tags: opensource, github, git, python
Published: 2020-11-30 04:06:48

*Not an Albertan pipeline, but a CI/CD pipeline.*

This week's lab for my Open Source Development class has us working on a CD pipeline ([as opposed to last week's](https://dev.to/chrispinkney/fake-it-till-you-make-it-46j0), which was the CI aspect) for our ever changing, semester long, [link checking program](https://github.com/chrispinkney/He-s-Dead-Jim).

> *For the uninitiated: A Continuous Delivery (CD) Pipeline deploys a new version of the software to the public for use (when a new version is available.) Releasing a new version can be done by hand, but having a CD pipeline ensures an automated process, so developers don't have to release new a version manually and can instead focus on development.* 

My program is written in Python, and as such the process of creating my CD pipeline for it will be described below from the perspective of the Python ecosystem.

Let's dive in!

___

For delivering new versions of my program, naturally the first step was to investigate and evaluate potential websites for hosting my deliverables. Since [PyPI](https://pypi.org/) is the goto standard for python packages, I decided to use that. The next step is obviously to [RTFM](https://packaging.python.org/tutorials/packaging-projects/).

Since we're automating our Python package, we have to ensure that the package has the right structure. Thankfully, my project knows its place, and was already structured properly. Here is a structure example from the above link, and my project's structure below it:

```
packaging_tutorial
├── LICENSE
├── README.md
├── example_pkg
│   └── __init__.py
├── tests
│   └── __init__.py
└── setup.py
```

```
He-s-Dead-Jim
├── LICENSE
├── README.md
├── src
│   └── __init__.py
│   └── hdj.py
│   └── hdj_fileio.py
│   └── hdj_linkchecker.py
│   └── hdj_util.py
├── tests
│   └── __init__.py
│   └── test_hdj_fileio.py
│   └── test_hdj_linkchecker.py 
└── setup.py
```

![](https://imgix.bustle.com/uploads/image/2019/3/21/8555ce94-b0fd-4848-8911-8fbd893143a0-tumblr_m2ykmqlrjf1qazx4ao6_250.gif)

The more keen readers (all 2 of you) would notice the `setup.py` file. `setup.py` is very important for a lot of reasons:

1. It keeps track of your project's dependencies, and without those your project will never run.
2. It's also used in the packaging process (see below) to determine what dependencies and files to keep track of.

Again, I already had a `setup.py` file but just recently deleted it to use the (much) easier `requirements.txt`. So I made it again (and actually learned how to use it this time):

```Python
import setuptools

with open("docs/README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="He's Dead, Jim",
    version="1.0.4",
    author="Chris Pinkney",
    author_email="hey@chrispinkney.com",
    description="A command-line tool for finding and reporting dead/broken links in a file or webpage.",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/chrispinkney/He-s-Dead-Jim",
    install_requires=[
        "argparse == 1.4.0",
        "requests == 2.24.0",
        "beautifulsoup4 == 4.9.1",
        "datetime == 4.3",
        "colorama == 0.4.4",
        "black == 20.8b1",
        "flake8 == 3.8.4",
        "pre-commit == 2.7.1",
        "pytest == 6.1.2",
        "pytest-cov == 2.10.1",
    ],
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    entry_points={
        "console_scripts": [
            "hdj = src.hdj:main_wrapper",
        ]
    },
    python_requires=">=3.6",
)
```

An important step is to get the `entry_points` dict just right. I had to [restructure](https://github.com/chrispinkney/He-s-Dead-Jim/commit/7abdbd8576337f7dce457900fd43138ee3c2a50b#diff-2589f96ddecfb638313e63270d22ee1e1062f60b89636f72f2872da394efdb73) a lot of my project to accommodate for `setup.py`, namely the main file, this had to happen because the project needed a proper entry point for use with `setup.py` and the main function already in place received command line arguments, making it not runnable when given no arguments. I'm sure there's a better way to do this but for now it works. I also had to suffix my import statements as the built version of my project was having troubles finding the proper files. Not sure why [but my hack fixed it](https://github.com/chrispinkney/He-s-Dead-Jim/commit/7abdbd8576337f7dce457900fd43138ee3c2a50b#diff-2589f96ddecfb638313e63270d22ee1e1062f60b89636f72f2872da394efdb73R1).

In order to test that my `setup.py` file works, I created virtual environments to see if all my dependencies were being installed properly:

1. `python -m venv hdj_env`
2. `.\hdj_env\Scripts\activate.bat`
3. `pip install .`
4. `pip list`

As your new shiny virtual environment does not come with ANY preinstalled dependencies, if your project's dependencies show up after running step 4, you know you did dun did it all good-like. With my `setup.py` file created and functioning properly, let's see if we can actually package our project.

In order to package my project, [I ran](https://www.youtube.com/watch?v=a7ZPSxHgKmw) `python setup.py sdist bdist_wheel` to generate the [Source Archive](https://packaging.python.org/glossary/#term-Source-Archive) (*.tar.gz*) file, and the [Build Distribution](https://packaging.python.org/glossary/#term-Built-Distribution) (*.whl*) file. We are generating these files in particular, because that is what PyPI accepts for upload. If our project can generate these files, we know that our automated CD pipeline can too.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/52d7nwjbid5apyu1k0fg.png)

Let's test manually uploading these files to PyPI to see if they will even be accepted.

PyPI uses a piece of software called `Twine` to upload things to their servers, naturally let's start there: `python -m pip install --user --upgrade twine`. With `Twine` installed, we now need to [create an account](https://pypi.org/account/register/) and an [API token](https://pypi.org/manage/account/) on PyPI. But now every time you upload, you'll be prompted for that API Token. That's annoying, so let's register our token locally on your computer. Simply create a file called `.pypirc` in `C:\Users\Chris` (aka `~/`) and put the following contents into it so twine won't prompt for username and password:

```
[distutils]
  index-servers =
    pypi

[pypi]
  username = __token__
  password = <your API key here>
```

> *Note that PyPI only allowed one version of your software on their servers, forever. Even if you delete that version from them, you must increment your versioning in order to push new source code to their servers, so if your pushes or CD pipeline fail, that might be why.*

And now the moment of truth, pushing our source code to PyPI. Remember that the archive and .whl file we generated earlier is located in the `/dist` directory, so let's point our command there:  `python -m twine upload dist/*`

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/rtj823ujj8qogi47t1qv.png)

Great success! Since my project is now uploaded, I can (kind of) be my own tester using a virtual environment! Let's create a virtual environment, activate it, and see if our project installs directly from PyPI: `python -m pip install He-s-Dead-Jim`

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/s3zi09d7tm7pqcjyq1wd.png)

Woohoo! [We're now live here](https://pypi.org/project/He-s-Dead-Jim/1.0.0/).

Okay, so what? We can upload files manually. That's all fine and dandy, let's take this one step further and add a CD pipeline to our GitHub repo to automate this process for us. The process is surprisingly easy, and similar to the process of creating a CI pipeline:  Generate a secret, create yet another YML file (heh) to register our GitHub actions, and push. Let's begin ([RTFM here.](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/))

Remember that API token I created earlier? Well, we need another one. This one is for GitHub (which is another user who will be submitting our project on behalf of us, so naturally we need an API token for them.) I generated a new one and added it to my repo's secrets (on GitHub: Your Repo > Settings > Secrets), and named it `PYPI_PASSWORD` (the secret name will be referenced later). Then I created the following workflow `.yml` file in my `.github/workflows/` folder:

```yml
name: CD Pipeline - Publish to PyPI and TestPyPI

on:
  push:  # Pattern matched against refs/tags
    tags:
      - '*'  # Push events to every tag

jobs:
  build-n-publish:
    name: Publish Python distributions to PyPI and TestPyPI
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
      - name: Set up Python 3.9
        uses: actions/setup-python@v1
        with:
          python-version: 3.9
      - name: Install pypa/build
        run: >-
          python -m
          pip install
          build
          --user
      - name: Build a binary wheel and a source tarball
        run: >-
          python -m
          build
          --sdist
          --wheel
          --outdir dist/
          .
      - name: Publish distribution to PyPI
        if: startsWith(github.ref, 'refs/tags')
        uses: pypa/gh-action-pypi-publish@master
        with:
          password: ${{ secrets.pypi_password }}
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/yz7n611252ps96owriv5.png)

Now that my project has a proper `setup.py` file configured, and can properly generate source code archives for automated CD upload, let's try adding a tag to our project and see if it uploads automatically. (*Reminder: We need a tag because our CD pipeline will do this process each and every time a new tag is pushed*):

> `git tag -a 1.0.4 -m "Big release!"`
>
> `git push --follow-tags`

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/hkkyc24y8r7lusm3xm7r.png)

Amazing. I can't believe all of this is free. Unreal.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/o5xpvp80qs4r4c9feesz.gif)

And that, my friends, is how I spent 9 hours avoiding studying for my finals which take place in 9 days.

___

With my project now live, I asked my friend Nilan to test my project out and see if it installs, runs, and works for him out of the box. The process went something like this:

> Me: Hey, I updated my project's README, can you test it for me?
> Me: Just follow the steps listed [here](https://github.com/chrispinkney/He-s-Dead-Jim/blob/master/docs/README.md#usage--installation).
> Nilan: wtf, I have to install Python?

><15 minutes later>

> Nilan: Wow I can't believe that just worked out of the box.
> Me: Cool eh?
> Nilan: This took you four months to build?
> Me: ...
> Nilan: Also fix up your readme, it's kind of confusing. Add some more what/where/why/how details, but otherwise your program downloaded and ran perfectly.
> Me: ... ok.

It was great to see that the program just worked off the bat from an install, and his feedback regarding the wording was really helpful, I think the document reads really nicely now.

___

For a very brief moment in time, I was number one in something:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/flughfpyal76q1kp8omn.png)

[also, adding emojis to a GitHub actions makes things really obnoxious, haha.](https://github.com/chrispinkney/He-s-Dead-Jim/commit/87ad65c23f4d5926537a958fec2a791542bd8f9b#diff-589b92445dc6f8ac68673fd9b852b4e366ab69127e791b0fd0c6b43a7a607ac2R1)

All I want for Christmas is to pass my Data Structures and Algorithms class.