PCCR - Perfect Code Coverage Reporting
Published: 2022-03-30 04:41:10

> [Cover photo](https://unsplash.com/photos/4EgyBDRZJXs/) by [Jonathan Cooper](https://unsplash.com/@theshuttervision)

---

If your project has perfect code coverage, and no one, not even you, knows about it, do doves cry?

We can never achieve *perfection*, but if we limit scope it is attainable. ;)

Perfect code coverage means 100%, including branches. NOTE: Easiest to accomplish when code has no branches!

Perfect code coverage _reporting_ would be to tell everyone about it, badge it up, and leave comments about it in PRs.

How do you get there?

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kh6470j7y6tj5jqgzsot.png)

---

I recently wrote a [gem](https://github.com/pboling/gitmoji-regex) (i.e. a Ruby library) that provides a canonical regex matching the set of unicode [Gitmoji](https://gitmoji.dev/) characters.

A simple library, with no branch statements, was easy to get coverage perfection, but a bit complicated to get reporting perfection...

My example utilizes Ruby, as my language of choice, but most of this, and particularly the Github Actions Coverage Workflow, will apply to any language.

---

# Benefits

## README.md

Self-promotion in open source projects is important.

In the [README.md](https://github.com/pboling/gitmoji-regex/blob/main/README.md) now there are glorious badges.

[![CodeClimate][⛳cclim-cov-img♻️]][⛳cclim-cov] [![CodeCov][🖇codecov-img♻️]][🖇codecov] [![Coveralls][🏘coveralls-img]][🏘coveralls]  [![CodeQL][🖐codeQL-img]][🖐codeQL] [![Code Coverage][🧮cov-wf-img]][🧮cov-wf]

[⛳cclim-cov]: https://codeclimate.com/github/pboling/gitmoji-regex/test_coverage
[⛳cclim-cov-img♻️]: https://api.codeclimate.com/v1/badges/f32e1d1148e8bad58197/test_coverage
[🖇codecov-img♻️]: https://codecov.io/gh/pboling/gitmoji-regex/branch/main/graph/badge.svg?token=EJCOr0hsPq
[🖇codecov]: https://codecov.io/gh/pboling/gitmoji-regex
[🏘coveralls]: https://coveralls.io/github/pboling/gitmoji-regex?branch=main
[🏘coveralls-img]: https://coveralls.io/repos/github/pboling/gitmoji-regex/badge.svg?branch=main
[🖐codeQL]: https://github.com/pboling/gitmoji-regex/security/code-scanning
[🖐codeQL-img]: https://github.com/pboling/gitmoji-regex/actions/workflows/codeql-analysis.yml/badge.svg
[🧮cov-wf]: https://github.com/pboling/gitmoji-regex/actions/workflows/coverage.yml
[🧮cov-wf-img]: https://github.com/pboling/gitmoji-regex/actions/workflows/coverage.yml/badge.svg

## Pull Request Automation

Additionally, PRs to the project should get a comment like the below from the `CodeCoverageSummary` [Github Action](https://github.com/irongut/CodeCoverageSummary)

![Code Coverage](https://img.shields.io/badge/Code%20Coverage-100%25-success?style=flat)

Package | Line Rate | Health
-------- | --------- | ------
gitmoji-regex | 100% | ✔
**Summary** | **100%** (50 / 50) | ✔

---

# Tools

- [CodeClimate](https://codeclimate.com/github/pboling/gitmoji-regex) SAAS
- [CodeCov](https://app.codecov.io/gh/pboling/gitmoji-regex) SAAS
- [Coveralls](https://coveralls.io/github/pboling/gitmoji-regex) SAAS
- [`irongut/CodeCoverageSummary`](https://github.com/marketplace/actions/code-coverage-summary) Github Action
- [`amancevice/setup-code-climate`](https://github.com/marketplace/actions/setup-code-climate) Github Action
- [Coverage-related Gems](https://github.com/pboling/gitmoji-regex/blob/main/Gemfile#L35) I use:
  - `gem "codecov", "~> 0.6"` - Note: `codecov/codecov-action` Github Action is an alternative for those not using Ruby. I left an example [in the coverage workflow](https://github.com/pboling/gitmoji-regex/blob/main/.github/workflows/coverage.yml#L107)
  - `gem "simplecov", "~> 0.21", require: false`
  - `gem "simplecov-cobertura"` provides XML format for Jenkins, or other tools
  - `gem 'simplecov-json'` provides JSON format for CodeClimate
  - `gem "simplecov-lcov", "~> 0.8", require: false` provides lcov format

---

# Implementation

## [Coverage Workflow](https://github.com/pboling/gitmoji-regex/blob/main/.github/workflows/coverage.yml)

This is the meat and potatoes that pulls everything together!

```yaml
name: Code Coverage

env:
  CI_CODECOV: true
  COVER_ALL: true

on:
  push:
    branches:
      - 'main'
      - '*-maintenance'
      - '*-dev'
      - '*-stable'
    tags:
      - '!*' # Do not execute on tags
  pull_request:
    branches:
      - '*'
  # Allow manually triggering the workflow.
  workflow_dispatch:

# Cancels all previous workflow runs for the same branch that have not yet completed.
concurrency:
  # The concurrency group contains the workflow name and the branch name.
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    name: Specs with Coverage - Ruby ${{ matrix.ruby }} ${{ matrix.name_extra || '' }}
    if: "!contains(github.event.commits[0].message, '[ci skip]') && !contains(github.event.commits[0].message, '[skip ci]')"
    strategy:
      fail-fast: false
      matrix:
        experimental: [false]
        rubygems:
          - latest
        bundler:
          - latest
        ruby:
          - "2.7"

    runs-on: ubuntu-latest
    continue-on-error: ${{ matrix.experimental || endsWith(matrix.ruby, 'head') }}
    steps:
      - uses: amancevice/setup-code-climate@v0
        name: CodeClimate Install
        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
        with:
          cc_test_reporter_id: ${{ secrets.CC_TEST_REPORTER_ID }}

      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Ruby & Bundle
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: ${{ matrix.ruby }}
          rubygems: ${{ matrix.rubygems }}
          bundler: ${{ matrix.bundler }}
          bundler-cache: true

      - name: CodeClimate Pre-build Notification
        run: cc-test-reporter before-build
        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
        continue-on-error: ${{ matrix.experimental != 'false' }}

      - name: Run tests
        run: bundle exec rake test

      - name: CodeClimate Post-build Notification
        run: cc-test-reporter after-build
        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
        continue-on-error: ${{ matrix.experimental != 'false' }}

      - name: Code Coverage Summary Report
        uses: irongut/CodeCoverageSummary@v1.2.0
        with:
          filename: ./coverage/coverage.xml
          badge: true
          fail_below_min: true
          format: markdown
          hide_branch_rate: true
          hide_complexity: true
          indicators: true
          output: both
          thresholds: '95 97'
        continue-on-error: ${{ matrix.experimental != 'false' }}

      - name: Add Coverage PR Comment
        uses: marocchino/sticky-pull-request-comment@v2
        if: matrix.ruby == '2.7' && always()
        with:
          recreate: true
          path: code-coverage-results.md
        continue-on-error: ${{ matrix.experimental != 'false' }}

      - name: Coveralls
        uses: coverallsapp/github-action@master
        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
        continue-on-error: ${{ matrix.experimental != 'false' }}

#      Using the codecov gem instead.
#      - name: CodeCov
#        uses: codecov/codecov-action@v2
#        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
#        with:
#          files: ./coverage/coverage.xml
#          flags: unittests
#          name: codecov-upload
#          fail_ci_if_error: true
#        continue-on-error: ${{ matrix.experimental != 'false' }}
```

## Simplecov

CONSTANTS NOTE: Some constants being used below, e.g. `RUN_COVERAGE` and `ALL_FORMATTERS`, are set in the [`spec/spec_helper.rb`](https://github.com/pboling/gitmoji-regex/blob/main/spec/spec_helper.rb#L12), which is run before the `.simplecov` config is loaded.  Others, e.g. `RUBY_ENGINE`, `RUBY_VERSION`, are standard Ruby.

You must [`require "simplecov"`](https://github.com/pboling/gitmoji-regex/blob/main/spec/spec_helper.rb#L23) before loading the code to be coverage checked.

```ruby
if RUN_COVERAGE
  require "simplecov" # Config file `.simplecov` is run immediately when simplecov loads
  require "codecov"
  require "simplecov-json"
  require "simplecov-lcov"
  require "simplecov-cobertura"
  # This will override any formatter set in .simplecov
  if ALL_FORMATTERS
    SimpleCov::Formatter::LcovFormatter.config do |c|
      c.report_with_single_file = true
      c.single_report_path = "coverage/lcov.info"
    end

    SimpleCov.formatters = [
      SimpleCov::Formatter::HTMLFormatter,
      SimpleCov::Formatter::CoberturaFormatter,
      SimpleCov::Formatter::LcovFormatter,
      SimpleCov::Formatter::JSONFormatter, # For CodeClimate
      SimpleCov::Formatter::Codecov # For CodeCov
    ]
  end
end

```

Configure it with a [`.simplecov`](https://github.com/pboling/gitmoji-regex/blob/main/.simplecov) config as below:

```ruby
# frozen_string_literal: true

# To get coverage
# On Local, default (HTML) output, it just works, coverage is turned on:
#   bundle exec rspec spec
# On Local, all output formats:
#   COVER_ALL=true bundle exec rspec spec
#
# On CI, all output formats, the ENV variables CI is always set,
#   and COVER_ALL, and CI_CODECOV, are set in the coverage.yml workflow only,
#   so coverage only runs in that workflow, and outputs all formats.
#

if RUN_COVERAGE
  SimpleCov.start do
    enable_coverage :branch
    primary_coverage :branch
    add_filter "test"
    track_files "**/*.rb"

    if ALL_FORMATTERS
      command_name "#{ENV["GITHUB_WORKFLOW"]} Job #{ENV["GITHUB_RUN_ID"]}:#{ENV["GITHUB_RUN_NUMBER"]}"
    else
      formatter SimpleCov::Formatter::HTMLFormatter
    end

    minimum_coverage(100)
  end
else
  puts "Not running coverage on #{RUBY_ENGINE} #{RUBY_VERSION}"
end
```

---

# End Result

[Github Actions Run](https://github.com/pboling/gitmoji-regex/runs/5748235810?check_suite_focus=true)

I have since re-implemented this pattern in other gems, including the [`oauth` gem](https://github.com/oauth-xx/oauth-ruby/), which uses MiniTest instead of RSpec, and which does *not* have 100% test coverage (yet).  Most of the implementation is the same.