Analyzing and enforcing .NET code coverage with coverlet
Tags: dotnet, csharp, testing, devops
Published: 2022-11-25 08:58:40

This post is part of a series:
[Part  1  -  Defining formatting rules in .NET with EditorConfig](https://blog.genezini.com/p/defining-formatting-rules-in-.net-with-editorconfig/)
[Part  2  -  Enforcing .NET code style rules at compile time](https://blog.genezini.com/p/enforcing-.net-code-style-rules-at-compile-time/)
Part  3  -  Analyzing and enforcing .NET code coverage with coverlet

## Introduction

Automated software tests are a requirement for ensuring we are delivering a product with quality to our users. It helps in finding bugs and requirements not fulfilled at development time, but also decreases the cost of maintenance by making the future changes to our code safer. Besides, the act of writing testable code alone increases the quality of the code we are writing because testable code has to be decoupled.

In this last post of this series, I'll show how to analyze and enforce a minimum code coverage in our applications, and how to use integration tests to increase our testing surface.

## What is code coverage?

Code coverage is a software metric that shows how much of our code is executed (covered) by our automated tests. It is shown as a percentage and can be calculated with different formulas, based on the number of lines or branches, for example. The higher the percentage, more of our code is being tested.

## Analyzing the code coverage of our application

In this example, we have an ASP.NET Core API with a simple use case class that checks an input number and returns a string telling if the number is even or odd:

```csharp
using CodeCoverageSample.Interfaces;

namespace CodeCoverageSample.UseCases;

public class IsEvenUseCase : IIsEvenUseCase
{
    public string IsEven(int number)
    {
        if (number % 2 == 0)
        {
            return "Number is even";
        }
        else
        {
            return "Number is odd";
        }
    }
}
```

For now, we have only one unit test for this use case class:

```csharp
using CodeCoverageSample.UseCases;

namespace CodeCoverageSample.UnitTests;

public class IsEvenUseCaseTests
{
    [Fact]
    public void EvenNumber_ReturnsEven()
    {
        //Arrange
        var isEvenUseCase = new IsEvenUseCase();

        //Act
        var result = isEvenUseCase.IsEven(2);

        //Assert
        Assert.Equal("Number is even", result);
    }
}
```

To analyze the code coverage of our application, first we need to install Coverlet's MSBuild integration using the `coverlet.msbuild` nuget package in our test project:

```bash
Install-Package coverlet.collector
```

Then, run the `dotnet test` command with the Coverlet options on the solution or project folder:

```bash
dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=../TestResults
```

We are using the following options:

- **CollectCoverage**: Inform dotnet test to use coverlet to collect the code coverage data;
- **CoverletOutputFormat**: The format of the report that coverlet will generate (opencover, cobertura, json). More [here](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/MSBuildIntegration.md);
- **CoverletOutput**: The path where the coverage report will be saved in. This path is relative to the test project;

This will print the code coverage result in a table and generate a report file named `TestResults.opencover.xml`:

![Code coverage results on the command line](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/CommandLine1.png)

> ⚠️ We can also run coverlet on the command line with the `coverlet.collector` nuget package, but it has limited options and doesn't print the results in the command line. More details [here](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md);

## Generating HTML reports

Coverlet generates the report in formats that are not easily readable by humans, so we need to generate an HTML report based on Coverlet report. To do it, we'll usa a tool called ReportGenerator.

### Installing ReportGenerator

ReportGenerator is installed as a .NET global tool.

To do this, we run the following command:

```bash
dotnet tool install --global dotnet-reportgenerator-globaltool --version 4.8.6
```

### Generating an HTML report of the opencover report

To generate an HTML report based on a Coverlet report, we run the following command:

```bash
reportgenerator "-reports:TestResults.opencover.xml" "-targetdir:coveragereport" -reporttypes:Html
```

We are using the following options:

- **reports**: The path to the coverage report;
- **targetdir**: The path where the HTML report will be saved in;
- **reporttypes**: The format the report will be generated in.

The command output will tell the relative path to the generated report: `coveragereport\index.html`.

![ReportGenerator output](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/Report0.png)

Opening the `coveragereport\index.html` file we can see the project Line and Branch coverage:

![HTML report summary](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/Report1.png)

Clicking on `CodeCoverageSample.UseCases.IsEvenUseCase` we can see details of the code coverage by method (in the table) and the line and branch coverage for the class:

![Details of a class on the HTML report](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/Report2.png)

## Line vs Branch coverage

But what is line coverage and branch coverage?

- **Line coverage**: Indicates the percentage of lines that are covered by the tests;
- **Branch coverage**: Indicates the percentage of logical paths that are covered by the tests (if, else, switch condition, etc).

In the example below, we can see that two lines in the `else` branch are not covered by the tests.

This will result in:
- 50% of branch coverage, because only the `if` branch is covered;
- 71.4% of line coverage, because only 5 of the 7 lines are covered.

![Example of a coverage report](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/LineBranchCoverage.png)

## Code coverage on Visual Studio

The is an extension called ***Run Coverlet Report*** that integrates Coverlet and ReportGenerator with Visual Studio.

1. First, we need to install the `coverlet.collector` nuget package in our test projects. Xunit template already has this package installed by default.

```bash
Install-Package coverlet.collector
```

2. Then, navigate to `Extensions > Manage extensions` and install the `Run Coverlet Report` extension.

![Manage Extensions screen](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/VisualStudioReport0.png)

3. Navigate to the new option `Tools > Run Code Coverage`. This will generate the ReportGenerator HTML report that will be open in Visual Studio.

![Code coverage report inside Visual Studio](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/VisualStudioReport1.png)

Also, after running the code coverage tool, Visual studio will read the coverlet report and show the coverage in our source file:

![Code inside Visual Studio showing code coverage](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/VisualStudioIntegration1.png)

## Improving our code coverage

### Fixing the unit tests

Now we will implement the `OddNumber_ReturnsOdd` method to test the logical path we didn't test before:

```csharp
using CodeCoverageSample.UseCases;

namespace CodeCoverageSample.UnitTests;

public class IsEvenUseCaseTests
{
    [Fact]
    public void EvenNumber_ReturnsEven()
    {
        //Arrange
        var isEvenUseCase = new IsEvenUseCase();

        //Act
        var result = isEvenUseCase.IsEven(2);

        //Assert
        Assert.Equal("Number is even", result);
    }

    [Fact]
    public void OddNumber_ReturnsOdd()
    {
        //Arrange
        var isEvenUseCase = new IsEvenUseCase();

        //Act
        var result = isEvenUseCase.IsEven(3);

        //Assert
        Assert.Equal("Number is odd", result);
    }
}
```

This will increase our average coverage to 50% of branch and 22.58% of line:

![Code coverage increased](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/CommandLine2.png)

And 100% for the `IsEvenUseCase` class:

![Code coverage showing all lives covered for the IsEvenUseCase class](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/Report3.png)

### Implementing integration tests

Integration tests using the `WebApplicationFactory` class (More [here](https://blog.genezini.com/p/integration-tests-without-api-dependencies-with-asp.net-core-and-wiremock.net/)) are also considered in the code coverage reports. Let's look at our `IsEvenController` and `Program` classes coverage:

![](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/IntegrationTests1.png)

![](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/IntegrationTests2.png)

Let's implement a simple integration test. It will just instantiate our API and make a call passing a number and validate the results:

```csharp
using Microsoft.AspNetCore.Mvc.Testing;
using System.Net;

namespace CodeCoverageSample.UnitTests.IntegrationTests;

public class IsEvenIntegrationTest : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly WebApplicationFactory<Program> _factory;

    public IsEvenIntegrationTest(WebApplicationFactory<Program> factory)
    {
        _factory = factory;
    }

    [Theory]
    [InlineData(2, "Number is even")]
    [InlineData(3, "Number is odd")]
    public async Task Number_ReturnsCorrectAndOk(int number, string expectedResult)
    {
        var HttpClient = _factory
            .CreateClient();

        //Act
        var HttpResponse = await HttpClient.GetAsync($"/iseven/{number}");

        //Assert
        Assert.Equal(HttpStatusCode.OK, HttpResponse.StatusCode);

        var ResponseStr = await HttpResponse.Content.ReadAsStringAsync();

        Assert.Equal(expectedResult, ResponseStr);
    }
}
```

Now we run the code coverage report again and the `IsEvenController` and `Program` classes are covered by the tests:

![](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/IntegrationTests3.png)

![](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/IntegrationTests4.png)

### Removing code from the code coverage analysis

If we want to remove a class or method from the code coverage analysis, we can decorate it with the `ExcludeFromCodeCoverage` attribute:

```csharp
using System.Diagnostics.CodeAnalysis;

namespace CodeCoverageSample;

[ExcludeFromCodeCoverage]
public class DoNotTestMe
{
    ...
}
```

> ℹ️ **We can also create custom attributes** to exclude from coverlet code coverage. Details [here](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/MSBuildIntegration.md#attributes).

### Ignoring auto-properties

Coverlet has the `SkipAutoProps` option to exclude the auto-properties from the coverage report.

For example, this class doesn't have any logic and doesn't need that the `get` and `set` methods of its properties be tested:

```csharp
namespace CodeCoverageSample;

public class NoLogicHere
{
    public int Id { get; set; }
    public int Name { get; set; }
}
```

Just set the `SkipAutoProps` to `true` when running the code coverage from the command line:

```bash
dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=TestResults -p:SkipAutoProps=true
```

> ⚠️ **Unfortunately, the Run Coverage Report extension** still doesn't allow us to configure the coverlet parameters. There is an open pull request with this feature awaiting for approval [here](https://github.com/the-dext/RunCoverletReport/pull/19).

## Enforcing a minimum code coverage on the build pipeline

Just like code style and code quality rules, that I talked about in my [previous post](https://blog.genezini.com/p/enforcing-.net-code-style-rules-at-compile-time/), we need to enforce a minimum code coverage in our build pipeline to maintain a level of quality in our code. Coverlet has the `Threshold` option that we can set to a minimum percentage and it will fail the tests if our code coverage is below this percentage:

```bash
dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=TestResults -p:SkipAutoProps=true -p:Threshold=80
```

We can also use the `ThresholdType` option to set the type of coverage to enforce. Not specifying will enforce all types of coverage (Line, Branch and Method). Details [here](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/MSBuildIntegration.md#threshold).

![Example of a failed test with Threshold=80](https://blog.genezini.com/p/analyzing-and-enforcing-.net-code-coverage-with-coverlet/images/ThresholdFail.png)

## Liked this post?

I post extra content in my personal blog. [Click here](https://blog.genezini.com/) to see.

## Follow me

- [On Feedly](https://feedly.com/i/subscription/feed/https://blog.genezini.com/index.xml)
- [RSS](https://blog.genezini.com/index.xml)
- [Linkedin](https://www.linkedin.com/in/danielgenezini/)
- [Twitter](https://twitter.com/GeneziniDaniel)

## References and Links

- https://blog.genezini.com/p/integration-tests-without-api-dependencies-with-asp.net-core-and-wiremock.net/
- https://blog.genezini.com/p/enforcing-.net-code-style-rules-at-compile-time/
- https://github.com/coverlet-coverage/coverlet
- https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/MSBuildIntegration.md
- https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md
- https://github.com/danielpalme/ReportGenerator
- https://marketplace.visualstudio.com/items?itemName=ChrisDexter.RunCoverletReport
- https://github.com/the-dext/RunCoverletReport/blob/master/README.md