How to keep track of docstring coverage of Python packages on CI
Tags: actionshackathon21, python
Published: 2021-11-19 04:13:11

Months ago, a collaborator asked for a way to keep track of the docstring coverage of our Python package. We wanted to measure the current coverage of the docstrings in our code and prevent pull requests from decreasing that value. 

To solve this problem, a workflow needed to fulfill the following requirements:

- The workflow should fail if the coverage is lower than the score of the base branch (on `pull requests`) or the previous commit (on `push`), and files that made the job fail should be blamed.
- Pushing to main branch should update the current coverage of the project and display the result on a nice badge.

Then, I found the excellent [`docstr-coverage`](https://github.com/HunterMcGushion/docstr_coverage) package that suited most of our needs, and in combination with GitHub Actions, [shields.io](https://shields.io) and [jsonbin.org](https://jsonbin.org/) I came up with a solution.

## jsonbin

[jsonbin.org](https://jsonbin.org/) is a personal key/value JSON store as a service.

In this case, we use this service to create dynamic badges with the [shields.io endpoint](https://shields.io/endpoint). A small JSON dictionary is passed to the endpoint and the badge is created on-the-go. Thus, we avoid to push a new badge to the repository each time the coverage changes.

```json
{
  "schemaVersion": 1,
  "label": "docstr-cov",
  "message": "75%",
  "color": "green"
}
```

You will need to sign in to **jsonbin** with your GitHub account and store the token as a repository secret named `JSONBIN_APIKEY`.

## Configuration file

Place a file named `.docstr.yml` at the root of your repository.

```yaml
paths:
  - your_package
verbose: 2 # int (0-3)
skip_magic: True
skip_file_doc: True
skip_init: True
skip_class_def: False
skip_private: True
follow_links: True
percentage_only: False
```
 For more information, see the [package documentation](https://github.com/HunterMcGushion/docstr_coverage#config-file).


## The workflow

This workflow should work out the box. It's not necessary to tweak any environment variable, but you can choose different values for `RANGE`.

For example, a `RANGE` of `50..75` means that coverage below `50` will display a red badge, and above `75` a green one. It works exactly like Codecov!

```yaml
name: docstr-cov

on:
  push:
    branches:
    - main

  pull_request:
    branches:
    - main

env:
  RANGE: 50..75
  ENDPOINT: https://jsonbin.org/${{ github.repository_owner }}/${{ github.event.repository.name }}
  TOKEN: ${{ secrets.JSONBIN_APIKEY }}
```


At the job level, the steps are easy to follow:

1. Commits to compare (`HEAD` and `BASE`) are chosen depending on the event trigger: `HEAD` of the branch vs. previous commit on `push` events, or `HEAD` of the PR vs. base of the branch on `pull_request`.
2. Get coverage score on `BASE` and `HEAD`. Fails if HEAD score is lower.
3. Blame files (if failed).

```yaml
jobs:
  check:
    runs-on: ubuntu-latest
    steps:

      - uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.x

      - name: Install docstr-coverage
        run: pip install docstr-coverage

      - name: Get SHAs
        run: |
          if [[ ${{ github.event_name }} == 'push' ]]; then
            echo "BASE=$(git rev-parse HEAD^)" >> $GITHUB_ENV
            echo "HEAD=$(git rev-parse HEAD)" >> $GITHUB_ENV

          elif [[ ${{ github.event_name }} == 'pull_request' ]]; then
            echo "BASE=${{ github.event.pull_request.base.sha }}" >> $GITHUB_ENV
            echo "HEAD=${{ github.event.pull_request.head.sha }}" >> $GITHUB_ENV
          
          else
            echo "Unexpected event trigger"
            exit 1
          fi

      - name: Get base coverage
        run: |
          git checkout $BASE
          echo "BASE_COV=$(docstr-coverage -p)" >> $GITHUB_ENV

      - name: Test head coverage
        run: |
          git checkout $HEAD
          docstr-coverage --fail-under=$BASE_COV

      - name: Blame
        run: |
          git diff --name-only $(git merge-base $BASE $HEAD) | \
          xargs docstr-coverage --accept-empty
        if: failure()
```

Only on `push` events:

4. Get coverage (rounded)
5. Set the color label according to `RANGE`.
6. Post results to **jsonbin**.
7. Make the endpoint public.
8. Print the badge URL.

```yaml
      - name: Get new coverage
        run: echo "NEW_COV=$(printf "%.f" $(docstr-coverage -p))" >> $GITHUB_ENV
        if: always() && github.event_name == 'push'

      - name: Set label color
        run: |
          if [[ $NEW_COV -ge $(echo {${{ env.RANGE }}} | awk '{print $NF;}') ]]; then
            echo "COLOR=green" >> $GITHUB_ENV
          elif [[ $NEW_COV -lt $(echo {${{ env.RANGE }}} | awk '{print $1;}') ]]; then
            echo "COLOR=red" >> $GITHUB_ENV
          else
            echo "COLOR=orange" >> $GITHUB_ENV
          fi
        if: always() && github.event_name == 'push'

      - name: Post results
        run: |
          curl -X POST $ENDPOINT/badges/docstr-cov \
          -H "authorization: token $TOKEN" \
          -d "{ \"schemaVersion\": 1, \"label\": \"docstr-cov\", \
                \"message\": \"$NEW_COV%\", \"color\": \"$COLOR\" }"
        if: always() && github.event_name == 'push'

      - name: Set public endpoint
        run: |
          curl -X PUT $ENDPOINT/_perms -H "authorization: token $TOKEN"
        if: always() && github.event_name == 'push'

      - name: Show badge URL
        run: echo "https://img.shields.io/endpoint?url=$ENDPOINT/badges/docstr-cov"
        if: always() && github.event_name == 'push'
```


The badge should be updated dynamically on every push to `main`. Remember to add it to your README.md.

## Get the code

{% github epassaro/docstr-cov-workflow %}