Smooth Packaging: Flowing from Source to PyPi with GitLab Pipelines
Tags: python, automation, gitlab, pypi
Published: 2024-01-18 13:49:39

# Overview

In the process of releasing a tool called [piper-whistle](https://gitlab.com/think-biq/piper-whistle), making it available through [pip](https://pip.pypa.io/en/stable/installation/) was ultimately the goal. Ideally, the concepts should be documented in the automation code itself. Yet it seemed useful to have a closer look at the architecture, details and quirks. The setup and inner workings of these most convenient mechanizations, I'd wish to share with you in this article.

## Branch setup

The concept is based on three main stages the source code can be in, which are represented by the [three branches](https://gitlab.com/think-biq/piper-whistle/-/branches) through which the project is structured by.

  * Cutting edge - main
  * Simulation - staging
  * The real thing - release

### Cutting edge

![Working on the cutting edge.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/743d48cvarj1p81wsh0w.jpeg)

The state on the main branch is for development and most recent changes. When pushed, the main branch triggers a set of testing automation, to help detecting any new issues or regressions.

### Simulation

![Orbis Alius.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wtifojz0zeaci0y025yw.jpeg)

If a set of features is coming together and seem worthy of a new release, lose ends are tied up and the version number in `src/piper_whistle/version.py` is increased in respect to the severity of the changes. Now the change-set is merged into the staging branch, which when pushed, will trigger a set of automatons, simulating the publishing of the final package. After the simulated publishing a test install is performed to see how the released version would behave.

### The real thing

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ozhk0q057um13rwqmlan.jpeg)

If everything appears fine and dandy, the next step is the proper release. For this, the changes are merged into the release branch. A tag with the name of the current version is created and associated with the latest commit. When pushed, the appropriate pipeline targets get triggered and the release candidate is processed and uploaded to the live [python package index](https://pypi.org/project/piper-whistle/).

## Preparations

To get started, this setup needs access to several platforms, which are listed below. Please make sure to have them setup and ready, if you’d like to follow along. It also assumes that you wish to synchronize your repository between GitLab and GitHub.

  * A python package project [buildable with setuptools](https://setuptools.pypa.io/en/latest/setuptools.html)
  * Source checked into git repository with three branch setup
  * [Gitlab](https://gitlab.com/) account
  * [GitHub](https://github.com/) account
  * [PyPi](https://pypi.org/) account
  * [PyPi staging / test](https://test.pypi.org/) account
  * [CI/CD Variables](https://docs.gitlab.com/ee/ci/variables/index.html) set up

To prepare the environment, GitLab offers project specific CI/CD variable configurations. The following list names the variable names as used in the script and their qualifier. The settings can be reached by browsing your repository and selecting:

`Setting -> CI/CD -> Variables (click Expand)`

This will allow you to create all the environment variable necessary for this setup.

### GITHUB_API_KEY_FILE

[Personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens), which allows for communication with github repository API.

```
github_pat_...
```

### GITHUB_DEPLOY_DOMAIN

This will be used to construct the domain for ssh known host configuration.

```
github.com
```

### GITHUB_DEPLOY_KEY_FILE

Contains the [private key deploy key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys) generated through GitHub.

```
-----BEGIN OPENSSH PRIVATE KEY-----
...
-----END OPENSSH PRIVATE KEY-----
```

### GITHUB_REPO_LINK

The ssh capable link to your GitHub repository.

```
git@github.com:think-biq/piper-whistle.git
```

### GITHUB_USER_EMAIL

This should be set to the user email you want to be used for the push / sync.

```
your@login.email
```

### GITHUB_USER_NAME

The username you want to be used for the push / sync.

```
YourGithubUsername
```

### PYPI_CONFIG_FILE

The contents of the [PyPi test server](https://packaging.python.org/en/latest/specifications/pypirc/#using-a-pypi-token) deploy token.

```
[testpypi]
  username = __token__
  password = pypi-Your-PyPi-Test-Token
```

### PYPI_LIVE_CONFIG_FILE

The contents of the [PyPi live server](https://packaging.python.org/en/latest/specifications/pypirc/#using-a-pypi-token) deploy token.

```ini
[pypi]
  username = __token__
  password = pypi-Your-PyPi-Token
```

# The script

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ct7f5ypmym9apa65y7sp.jpeg)

Working with pipelines is done through defining different targets (or jobs), which will then be run at certain stages and under certain conditions. [The script](https://gitlab.com/think-biq/piper-whistle/-/blob/release/.gitlab-ci.yml?ref_type=heads) file is picked up if you name it *.gitlab-ci.yml* and place it at root level of your repository. Each target spins up a new [container image](https://docs.gitlab.com/ee/ci/yaml/#image) and then executes a [set of instructions](https://docs.gitlab.com/ee/ci/yaml/script.html). It is possible to define a base image for each target, define a global default one and combine both, leading to cascaded overrides (except when you disable it via the inherit attribute).

## Imagination

I created the base container [ue-plugin-staging](https://gitlab.com/think-biq/docker-ue-plugin-staging) to prepare releases of Unreal Engine plugins and found it useful for other automation purposes (hence the name). You may create your own, or use another one which provides the same set of tools.

```yml
image: thinkbiq/ue-plugin-staging:latest
```

## All in stages

Now we define the stages used through the pipeline. Which are essentially testing, building and releasing with extra utility layers.

```yml
stages:
  - pre-release-test
  - build
  - release
  - grace
  - post-release-test
  - finalize
```

## Journey to the testing grounds

The first major layer is testing. It is split in testing before any release happens and tests which are run after the release has been successfully concluded. Let’s look at the tests running before anything else is started.

### Smoke test

First in line of the pipeline is a functional test defined in the testing package of the `piper-whistle` source. This is what test-functional is responsible for. It checks functionality of the tool holistically by running the smoke / integration tests implemented through the class *CliCommandTests* defined in `src/testing/functionality.py` using [unittest](https://docs.python.org/3/library/unittest.html#basic-example).

```bash
python3 -m unittest src/testing/functional.py
```

The target is available for manual trigger on the main as well as staging branch and simply installs all necessary pip requirements and invokes the test through the [makefile](https://www.gnu.org/software/make/manual/make.html) target `run-tests`, which in turn invokes the unittest runner on functional.py.

```yml
test-functional:
  stage: pre-release-test
  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: true
    - if: $CI_COMMIT_REF_NAME == "staging"
      allow_failure: false
  script:
    - make pip-update-all
    - make run-tests
```

### Security audit

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mxn1bcjit3syjmta8748.jpeg)

Next up is making sure, none of the [dependencies](https://gitlab.com/think-biq/piper-whistle/-/blob/main/requirements.txt?ref_type=heads) used throughout the project brings with it any already identified security issue. The makefile target audit, invokes the handy tool [pip-audit](https://pypi.org/project/pip-audit/).

```bash
pip-audit --desc
pip-audit --fix --dry-run
```

With its help, the pipeline target test-security runs a quick scan on the packages installed and looks for known vulnerabilities and proposes fixes (if any).

```yml
test-security:
  stage: pre-release-test
  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: true
    - if: $CI_COMMIT_REF_NAME == "staging"
      allow_failure: false
  script:
    - make pip-update-all
    - make audit
```

It can be triggered manually on the main branch. When automatically triggered on branch staging, this needs to pass without any errors or warnings.

### Linting

There are multiple linting libraries and tools out there. For this scenario, I was looking for a reasonable balance of (proper) standards and ability to configure the process in regards to my customized (arguably foolish) code style. Some may be dismayed by this, which is why I take the liberty of quoting [Guido van Rossum](https://peps.python.org/pep-0008/) here:

> A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is the most important.

Adding more weight to ease of setup and configurability, the choice came down on [flake8](https://flake8.pycqa.org/en/latest/). It is easy to integrate, since its also available through pip and let’s you configure which standards you want to omit by simply stating them as a list via the --ignore switch. Moving to [ruff](https://github.com/astral-sh/ruff) appears [quite smooth](https://pypi.org/project/flake8-to-ruff/), so future updates may do so.

```yml
test-linting:
  stage: pre-release-test
  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      allow_failure: true
    - if: $CI_COMMIT_REF_NAME == "staging"
      allow_failure: false
  script:
    - make pip-update-all
    - make lint
```

## Turning the wheel of fortune

![Fortunam insanam esse et caecam et brutam perhibent philosophi.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ekaxxfvj4udv3q2ngnr.jpeg)

The target `build-wheel` - sole constituent of the build layer - installs all dependencies and resolves the [readme markdown template](https://gitlab.com/think-biq/piper-whistle/-/blob/release/etc/readme.md.tmpl?ref_type=heads) to its release form (using the tmplr tool), since the `readme.md` file is packaged with the [wheel file](https://pythonwheels.com/). This is followed by the generation of a compressed source file release ([tar.gz](https://linuxize.com/post/how-to-extract-unzip-tar-gz-file/)) and [building a wheel file](https://realpython.com/python-wheels/) (whl) using [build](https://build.pypa.io/en/stable/) and setuptools respectively.

```yml
build-wheel:
  stage: build
  rules:
    - if: $CI_COMMIT_REF_NAME == "release"
      allow_failure: false
    - if: $CI_COMMIT_REF_NAME == "staging"
      allow_failure: false
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
  script:
    - make pip-update-all
    - make readme-build
    - make release
    - export VERSION_TAG=$(etc/./query-tag-most-relevant-version.sh)
    - echo "VERSION_TAG=${VERSION_TAG}" > pypi.env
    - export WHEEL_FILE_NAME="piper_whistle-${VERSION_TAG}-py3-none-any.whl"
    - echo "WHEEL_FILE_NAME=${WHEEL_FILE_NAME}" >> pypi.env
    - export DEP_URL_BASE="https://pypi.debian.net/piper-whistle"
    - export WHEEL_FILE_URL="${DEP_URL_BASE}/${WHEEL_FILE_NAME}"
    - echo "WHEEL_FILE_URL=${WHEEL_FILE_URL}" >> pypi.env
    - echo "pypi.env:"
    - cat pypi.env
    - echo "Generating release page for ${VERSION_TAG} ..."
    - if ! etc/./query-tag-message.sh ${VERSION_TAG} > release-notes.md; then
        echo "Could not query tag ${VERSION_TAG}. Defaulting to empty.";
        echo "" > release-notes.md;
      fi
    - echo "release-notes.md:"
    - cat release-notes.md
  artifacts:
    paths:
      - build/release
      - release-notes.md
    reports:
      dotenv: pypi.env
```

Both of those are then [exposed as artifacts](https://docs.gitlab.com/ee/ci/jobs/job_artifacts.html), along side some [environment variables](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportsdotenv) as well as the release notes, which were obtained by invoking `query-tag-message.sh`. [This shell script](https://gitlab.com/think-biq/piper-whistle/-/blob/release/etc/query-tag-message.sh) queries the git log for the message attached to the current version tag. Or if run on a non-tagged commit, leaves it empty. The [version name is obtained](https://gitlab.com/think-biq/piper-whistle/-/blob/release/etc/query-tag-most-relevant-version.sh) via `query-tag-most-relevant-version.sh`, which either takes the name of the tag attached to the current commit, or queries the [whistle’s version module](https://gitlab.com/think-biq/piper-whistle/-/blob/main/src/piper_whistle/version.py?ref_type=heads). Targets wanting access to these artifacts, simply have to specify the [target name and indicate access to artifacts](https://docs.gitlab.com/ee/ci/yaml/#needsartifacts) through the needs attribute.

## Playtime Preparation: Setting stage for the package

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pm0scx6mslhz63t9uwg3.png)

The [PyPi team](https://psfmember.org/civicrm/contribute/transact/?reset=1&id=13) allows you to use a testing mirror of the live repository at [test.pypi.org](https://test.pypi.org/) to rehearse the release of your python package. I'm using [twine](https://pypi.org/project/twine/) to communicate with PyPi staging as well as live servers. The upload can be commenced by [invoking the script](https://gitlab.com/think-biq/piper-whistle/-/blob/release/etc/upload-pypi.sh?ref_type=heads) `upload-pypi.sh`, which looks something like:

```bash
twine upload --non-interactive --verbose \
        --config-file "${configpath}" -r ${repo} \
        --skip-existing \
        $*
```

The [format of the config](https://packaging.python.org/en/latest/specifications/pypirc/) file is defined by [distutils](https://docs.python.org/3.10/library/distutils.html). The pipeline script uses separate configuration files for test and live server of PyPi. You may as well put them in one file all together and only select the repository to use with the `-r` switch.

```ini
[testpypi]
  username = __token__
  password = pypi-ToKeN

[pypi]
  username = __token__
  password = pypi-ToKeN
```

### Being pushy

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h3f0h8nzloag9jku0m0u.png)

First in line is `release-pypi-test`, which fetches the artifacts produced by `build-wheel` and pushes them to the test server using the authentication configuration store in `PYPI_CONFIG_FILE`.

```yml
release-pypi-test:
  stage: release
  rules:
    - if: $CI_COMMIT_REF_NAME == "staging"
      allow_failure: false
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: true
  needs:
    - job: build-wheel
      artifacts: true
  script:
    - python3 -m pip install twine
    - etc/./upload-pypi.sh testpypi "${PYPI_CONFIG_FILE}" build/release/*
```

### Patience is a virtue

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cgdp0quh54t3rt10soj6.jpeg)

Since the PyPi package cache seem to need a few cycles to catch up with the updated index, `delay-before-checking-pypi-test` sits between the upload and following test target.

```yml
delay-before-checking-pypi-test:
  stage: grace
  rules:
    - if: $CI_COMMIT_REF_NAME == "staging"
  variables:
    DELAY_IN_SECONDS: 13
  script:
    - echo "Sleeping for ${DELAY_IN_SECONDS} seconds ..."
    - sleep ${DELAY_IN_SECONDS}
    - echo "Done."
```

### Let's go live! (well almost)

```bash
python3 -m pip install \
    --trusted-host test.pypi.org --trusted-host test-files.pythonhosted.org \
    --index-url https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple/ \
    piper_whistle==$(python3 -m src.piper_whistle.version)
```

After patiently waiting our turn, we check if pip is able to fetch the newly released wheel file via a carefully crafted `pip install` invocation. The *--index-url* argument specifies the test server index and the *--trusted-host* arguments ensures seamless binary downloads from the test servers. It turned out, that some dependencies where not available on the test server, which necessitated the `--extra-index-url` parameter pointing to the index of the live server.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vlrboggm0crbdy04zd6p.jpeg)

Finally, to ensure the current latest version of the staging branch is fetched from the test server and not the most recent version from the live server, the version module of piper-whistle is invoked to obtain the semantic version name. In the build script, this [command is contained](https://gitlab.com/think-biq/piper-whistle/-/blob/release/etc/install-test-pypi.sh?ref_type=heads) within the `install-test-pypi.sh` shell script for brevity. The target `test-release-pypi-test` is only available on the staging branch and is automatically triggered.

```yml
test-release-pypi-test:
  stage: post-release-test
  rules:
    - if: $CI_COMMIT_REF_NAME == "staging"
      allow_failure: true
  script:
    - python3 -m pip install -U pip
    - export PGK_NAME="piper_whistle==$(python3 -m src.piper_whistle.version)"
    - etc/./install-test-pypi.sh ${PGK_NAME}
    - piper_whistle -h
    - echo "Setting up search index ..."
    - piper_whistle refresh
    - echo "Checking if HU is supported ..."
    - if [ "hu_HU" = "$(piper_whistle guess hung)" ];
        then echo "Found hungarian language support";
        else exit 13;
      fi
    - echo "Checking if model at index 1 is available ..."
    - piper_whistle list -U -l hu_HU -i 1
    - echo "Installing model 1 ..."
    - piper_whistle install hu_HU 1
    - echo "Fetching path of model 1 ..."
    - export HU_VOICE_INFO=$(piper_whistle list -S -l hu_HU -i 1)
    - export VOICE_NAME="$(echo ${HU_VOICE_INFO} | awk '{ print $1 }')"
    - piper_whistle path ${VOICE_NAME}
```

After installing the package it simply goes through refresh, setup and downloads a voice to test basic functionality.

## Getting serious

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/anmsordmv1xffdl0wuf0.png)

### Publishing to the main index

Analogous to the test release target, `release-pypi` is virtually the same, except it is trigger on the release branch and uses the release server configuration in `PYPI_LIVE_CONFIG_FILE` instead.

```yml
release-pypi:
  stage: release
  rules:
    - if: $CI_COMMIT_REF_NAME == "release"
      allow_failure: false
  needs:
    - job: build-wheel
      artifacts: true
  script:
    - echo "Uploading ..."
    - ls -lav build/release
    - python3 -m pip install twine
    - etc/./upload-pypi.sh pypi "${PYPI_LIVE_CONFIG_FILE}" build/release/*
```

### Gitlab release on repository site

To allow people to [directly download the latest built](https://gitlab.com/think-biq/piper-whistle/-/releases) from piper-whistle's repository, the `release-gitlab` target makes use of GitLab's base image [release-cli](https://gitlab.com/gitlab-org/release-cli) exposing attributes for the creation of a new release page. It uses GitLab's release tool to process said attributes.

```yml
release-gitlab:
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  stage: release
  rules:
    - if: $CI_COMMIT_REF_NAME == "release"
      allow_failure: true
  needs:
    - job: build-wheel
      artifacts: true
  script:
    - echo Tag name = "${VERSION_TAG}"
    - echo Release name = "piper-whistle ${VERSION_TAG} wheel"
    - echo Release notes = "$(cat release-notes.md)"
    - echo Wheel file link = "${WHEEL_FILE_URL}"
  release:
    name: 'piper-whistle ${VERSION_TAG} wheel'
    description: 'release-notes.md'
    tag_name: '${VERSION_TAG}'
    ref: '$CI_COMMIT_SHA'
    assets:
      links:
        - name: '${WHEEL_FILE_NAME}'
          url: '${WHEEL_FILE_URL}'
```

Here we make use of the environment variables exposed in the `build-wheel` target, as well as use the release notes stored in `release-notes.md` for the description.

The wheel file download link is generated with the help of a [front-end to PyPi](https://pypi.debian.net/) provided by the [Debian Project](https://www.debian.org/). It may take several minutes until the front-end can find the package in the cache.

![Release 1.6.188 via GitLab's repository website.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8c4qo63ovya6pkmac5o5.png)

### Github release

With the help of [rota](https://gitlab.com/think-biq/piper-whistle/-/blob/release/src/tools/rota.py?ref_type=heads), we can make the new release available on the github hosted repository via the `github-release` command. It needs an access token stored in a file, the repository ID in the form of `{username}/{project}`, the version name and the release message.

```yml
release-github:
  stage: release
  rules:
    - if: $CI_COMMIT_REF_NAME == "release"
      allow_failure: true
  needs:
    - job: build-wheel
      artifacts: true
  script:
    - echo Tag name = "${VERSION_TAG}"
    - echo Release name = "piper-whistle ${VERSION_TAG} wheel"
    - echo Release notes = "$(cat release-notes.md)"
    - echo Wheel file link = "${WHEEL_FILE_URL}"
    - echo "Uploading release "${VERSION_TAG}" to github ..."
    - python3 -m src.tools.rota github-release
      --token-file "${GITHUB_API_KEY_FILE}"
      --attachment "build/release/${WHEEL_FILE_NAME}"
      think-biq/piper-whistle release "${VERSION_TAG}" "$(cat release-notes.md)"
```

It sends a request to the [github API](https://docs.github.com/en/rest/releases?apiVersion=2022-11-28) asking for a new release to be created. If that was successful, it retrieves the dedicated upload endpoint for the specific release and [uploads the file](https://docs.github.com/en/rest/releases/assets?apiVersion=2022-11-28#upload-a-release-asset) specified via *--attachment* (if any).

## Another page in the book

To keep the documentation [up to date and readily available](https://think-biq.gitlab.io/piper-whistle/), the target pages builds a web release of the [doc-strings](https://peps.python.org/pep-0257/) found throughout the source code via [doxygen](https://doxygen.nl/) and hosts it through [GitLab’s pages](https://docs.gitlab.com/ee/user/project/pages/) infrastructure.

```yml
pages:
  stage: finalize
  rules:
    - if: $CI_COMMIT_REF_NAME == "release"
      allow_failure: true
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: true
  script:
    - echo "Install requirements ..."
    - make pip-update-all
    - echo "Building documentation ..."
    - make docs-build
    - echo "Exposing as public ..."
    - mv docs/published public
    - echo "Result:"
    - ls -lav public
    - echo "Hosting ..."
  artifacts:
    paths:
      - public
```

### readthedocs.org

Additionally, a copy of most [recent documentation based off of the main branch](https://piper-whistle.readthedocs.io/en/latest/) is maintained on readthedocs.org servers.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2lucrywk6zu6sm5i1mp2.jpg)

When pushing to the main branch, readthedocs will scan the root path of the repository and look for a `.readthedocs.yaml` configuration file.

```yml
version: 2

build:
  os: ubuntu-20.04
  tools:
    python: "3.8"
  commands:
    - echo "Install dependencies ..."
    - make pip-update-all || echo 💥
    - echo "Building documentation using doxygen ..."
    - make docs-build || echo 💥
    - echo "Ensuring html directory ..."
    - mkdir -p "$(realpath ${READTHEDOCS_OUTPUT})/html"
    - echo "Moving page assets ..."
    - mv docs/published/* "$(realpath ${READTHEDOCS_OUTPUT})/html" || echo 🚚💥
    - echo "Per Aspera Ad Astra 🚀"
```

It’s very convenient, to have straight forward support for the static page setup which is produced by *doxygen*. We only need to make sure to move the generated artifacts to the right place. This would be `${READTHEDOCS_OUTPUT}/html`. Since I’ve encountered issues where the path was malformed (i.e. double /), `realpath` is used to normalize the path before handing it off to the `mv` command. The way to the stars seems a bit clearer now.

## The Harmonious Hustle: GitLab CI/CD's Dance with GitHub for Seamless Repository Sync

![In sync.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hdyn3jso9unba97v9r1r.png)

To increase the reach of piper-whistle and a greater change of someone contributing or reporting any issues, keeping the [project in sync with GitHub](https://github.com/think-biq/piper-whistle) seems like a reasonable ambition. Let's look at how me way do so with a pipeline target called `sync`.

```yml
sync:
  stage: finalize
  rules:
    - if: $CI_COMMIT_REF_NAME == "release"
      allow_failure: true
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: true
  script:
    - apt-get update -y
    - apt-get install -yqqf openssh-client sshpass --fix-missing
    - eval $(ssh-agent -s)
    - cat "${GITHUB_DEPLOY_KEY_FILE}" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan ${GITHUB_DEPLOY_DOMAIN} >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
    - git config --global user.email "${GITHUB_USER_EMAIL}"
    - git config --global user.name "${GITHUB_USER_NAME}"
    - git fetch --all
    - git checkout staging
    - git checkout release
    - git checkout main
    - git remote rm origin 2> /dev/null
    - git remote add origin ${GITHUB_REPO_LINK}
    - git push --all
    - git push --tags
```

To achieve this, the setup uses a container local ssh user setup, leveraging the [deploy key infrastructure](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys) offered by GitHub. After setting up the key and known_host configuration, changing the remote origin entry allows for pushing new commits as well as tags. Additionally, it may be appropriate to only offer this automation on the main branch and set it's trigger to manual, so not every commit grinds the pipeline budget.

You could just do this manually, by adding a separate remote for GitHub and pushing directly, but where would the fun be with this?

## Extra fun

For some extra spice, the current status of the release pipeline and readthedocs status is shown as badges on the top of the readme.

To obtain a badge for your projects pipeline, only taking into account non-skipped targets:

![release](https://gitlab.com/think-biq/piper-whistle/badges/release/pipeline.svg?ignore_skipped=true)

```yml
f"https://gitlab.com/{namespace}/{project}/badges/{branch}/pipeline.svg?ignore_skipped=true"
```

A badge showing the latest / most recent release tag, you may use:

![release](https://gitlab.com/think-biq/piper-whistle/-/badges/release.svg)

```yml
f"https://gitlab.com/{namespace}/{project}/-/badges/release.svg"
```

The readthedocs.org badge may be constructed via:

![release](https://user-content.gitlab-static.net/1febe753d6976b6ea4d05c8d04b5be704c426fc5/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f70697065722d77686973746c652f62616467652f3f76657273696f6e3d6c6174657374)

```yml
f"https://readthedocs.org/projects/{project_slug}/badge/?version={version}&style={style}"
```

# Conclusion

Thank you for reading and I wish you all the best on your automation adventures. To celebrate the mysteries before us, let’s share in spirit with these attributed sayings:

> “At last concluded that no creature was more miserable than man, for that all other creatures are content with those bounds that nature set them, only man endeavors to exceed them.” ― Desiderius Erasmus Roterodamus

![Traveling; north, east, south or west. Where's the wisdom, where's your zest.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d77t3iwmc48ey6t8x0k4.jpeg)

> “I have so many ideas that may perhaps be of some use in time if others more penetrating than I go deeply into them someday and join the beauty of their minds to the labour of mine.” ― Gottfried Wilhelm Leibniz

# Encore

So, you are still reading? Let’s do some math and statistics then. Using the following script to stalk through the repository and separating the code for the package in `piper_whistle` from all the rest of the scripts and configuration files, we can calculate a ratio of **lines of code to produce** to **lines of code in production**. Something like a build-infrastructure efficiency coefficient, or **biec** for short.

## The script: biec.sh

This approach is not omitting any comments or white-spaces, so keep that in mind.

```bash
count-lines-of-code-package () {
  # Counts only source files in main package.

  local files=$(find ./src/piper_whistle -iname "*.py" -type f)
  local count=$(for f in ${files}; do wc -l $f; done \
    | awk '{ print $1 }' \
    | awk '{ s+=$1 } END { printf "%.0f", s }')

  echo ${count}

  return 0
}

count-lines-of-code-tooling () {
  # Collects all source files relevant for tooling, building and configuring.

  local files=$(find . \
    -iname "*.py" -type f \
    -not -path "*/src/piper_whistle/*" \
    -not -path "*/build/lib/*" \
    -o -iname "*.sh" -o -iname "makefile" \
    -o -iname "pip.conf" -o -iname "*.cfg" \
    -o -iname "*.yaml" -o -iname "*.yml")
  local count=$(for f in ${files}; do wc -l $f; done \
    | awk '{ print $1 }' \
    | awk '{ s+=$1 } END { printf "%.0f", s }')

  echo ${count}

  return 0
}

main () {
  # Investigate the codebase and create a ratio of 'lines of code in production' to 'lines of code to produce'

  local root_path="${1}"
  local package_lines=0
  local tool_lines=0
  local ratio=0

  pushd "${root_path}" > /dev/null
    package_lines=$(count-lines-of-code-package)
    tool_lines=$(count-lines-of-code-tooling)
  popd > /dev/null
  ratio=$(echo "${tool_lines} / ${package_lines}.0" | bc -l)

  echo "package: ${package_lines}"
  echo "tooling: ${tool_lines}"
  echo "ratio: ${ratio:0:6}"

  return 0
}

main $*
```

Hint: Make sure to not place this script in the actual directory of the repository :)

## Analyzing

Invoking it via `biec.sh ~/Workspace/Remote/piper-whistle` on release [1.6.213](https://gitlab.com/think-biq/piper-whistle/-/tree/1.6.213?ref_type=tags) yields:

```yml
package: 2050
tooling: 4080
ratio: 1.9902
```

So I have virtually twice as much lines of code supporting the maintenance, testing and publishing of the actual `piper-whistle` python package. Looking at the files that were taken into account, the heaviest appears to be the doxygen documentation configuration file.

```bash
2 ./setup.cfg
70 ./makefile
36 ./etc/query-wheel-url-from-debian.sh
45 ./etc/query-tag-most-relevant-version.sh
12 ./etc/install-test-pypi.sh
20 ./etc/query-tag-message.sh
19 ./etc/upload-pypi.sh
8 ./src/__init__.py
1 ./src/testing/__init__.py
380 ./src/testing/functional.py
0 ./src/tools/__init__.py
217 ./src/tools/tmplr.py
363 ./src/tools/rota.py
19 ./.readthedocs.yaml
20 ./.github/workflows/publish.yml
58 ./setup.py
19 ./docs/makefile
2534 ./docs/docs.cfg
4 ./pip.conf
253 ./.gitlab-ci.yml
```

It alone is responsible for 2534 [LOC](https://en.wikipedia.org/wiki/Source_lines_of_code) (62.1%). When excluding this config, we have a more reasonable result of:

```yml
package: 2050
tooling: 1546
ratio: 0.7541
```

## Discussion

So roughly for every 4 lines of package code, there are 3 lines of supporting infrastructure code. I would be so bold as to state that the lines of code needed to be included when scaling the actual program does not grow as fast as the lines of code in the package. I would estimate about _10% growth_ of infrastructure code in relation to the product code. Under this assumption, we would reach a **biec** of **0.5** at about _3352 LOC_ for the product. As a reference, [_PyYAML_](https://pypi.org/project/PyYAML/) version 6.0.1 has _5890 LOC_. Recalculating for that line count, would result in a **biec** of about **0.33** for PyYAML.
Might be fun to test this on other (build-)setups and packages as well.

# Follow-up

If you feel inspired and want to get in touch, you may reach me on [twitter / X](https://twitter.com/blurryroots), [discord](https://discordapp.com/users/232489168477880320) or [LinkedIn](https://www.linkedin.com/in/blurryroots).