JaCoCo doesn't directly support vacuously true 100% branch coverage
Tags: java, testing, codecoverage, metrics
Published: 2024-01-28 00:45:48

## Background

I'm an educator and share most of my thoughts just with my students. Once in a while, I have something to share that might help a wider audience and decided to try that here.

## Measuring code coverage with JaCoCo

[Code coverage](https://en.wikipedia.org/wiki/Code_coverage) is a metric that indicates how thoroughly we're testing. [JaCoCo](https://www.eclemma.org/jacoco) is a mature, actively developed code coverage tool for Java and other JVM-based languages. For [each type of coverage](https://www.eclemma.org/jacoco/trunk/doc/counters.html), such as lines, branches, etc., it keeps track of covered and missed items and generates a report with the corresponding coverage percentages.

## Minimal example in Java

Here is a minimal SUT (system under test) in Java:

```java
public class HelloWorld {
  public String getMessage() {
    return "hello world";
  }
}
```

Here is a JUnit assertion for it:

```java
assertEquals("hello world", new HelloWorld().getMessage());
```

And this is the resulting coverage report:

```text
[info] Test run started (JUnit Jupiter)
[info] Test hw.TestHelloWorld#getMessage() started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 0.136s
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[info]
[info] ------- Jacoco Coverage Report -------
[info]
[info] Lines: 100% (>= required 90.0%) covered, 0 of 2 missed, OK
[info] Instructions: 100% (>= required 80.0%) covered, 0 of 5 missed, OK
[info] Branches: 0% (< required 100.0%) covered, 0 of 0 missed, NOK
[info] Methods: 100% (>= required 100.0%) covered, 0 of 2 missed, OK
[info] Complexity: 100% (>= required 100.0%) covered, 0 of 2 missed, OK
[info] Class: 100% (>= required 100.0%) covered, 0 of 1 missed, OK
[info]
[info] Check /Users/laufer/Work/teaching/cs335/hello-java-sbt/target/scala-2.12/jacoco/report for detailed report
[info]
[error] java.lang.RuntimeException: Required coverage is not met
...
```

*According to JaCoCo, the required coverage threshold is not met because zero branches were covered.*

## How does JaCoCo calculate this?

In JaCoCo's [`CounterImpl`](https://github.com/jacoco/jacoco/blob/master/org.jacoco.core/src/org/jacoco/core/internal/analysis/CounterImpl.java#L181-L195) class, the coverage percentages are calculated as follows:

```java
public double getCoveredRatio() {
  return (double) covered / (missed + covered);
}
```

So, when there is nothing to be covered or missed, the ratio is `Double.NaN` ("not a number"), representing the division of 0 by 0 in this case.

## Discussion

If there are no branches to be covered, shouldn't coverage be automatically (vacuously) 100%? From a discrete math perspective, yes:

*If there is no work to be done, then the work is already 100% done.*

This corresponds to the understanding in mathematical logic that a universally quantified predicate over an empty set (of any element type) is always true (even if the predicate itself is always false):

```scala
scala> Set.empty[String].forall(x => false)
val res0: Boolean = true
```

It's easy to implement this behavior by adding a case distinction:

```java
public double getCoveredRatio() {
  if (missed == 0) {
    return 1;
  }
  return (double) covered / (missed + covered);
}
```

After working on this for a bit, I finally found the [pertinent closed issue from 2015](https://github.com/jacoco/jacoco/issues/305). The closing comment suggests that they are favoring a continuous math perspective based on the ratio calculation shown above, which returns `NaN` even if there are no items missed. They closed comments on this issue, and I don't think they'd consider reopening it.

*Arguably, this is a leaky abstraction that is inconsistent with the definition of code coverage from a discrete math perspective.*

## Why is this even relevant?

Doesn't all real-world code include branches, such as conditionals or switches? 

Well, not necessarily. Arguably, the fewer the better from a readability and maintenance point of view. Event-based systems, such as this [Android stopwatch](https://github.com/lucoodevcourse/stopwatch-android-java), might not need explicit branching in their control flow when using structural techniques instead, such as attaching listeners to different event sources and applying the [State pattern](https://en.wikipedia.org/wiki/State_pattern) to implement the underlying state-based behavior.

## Conclusion

This is obviously a rather minor issue but regularly throws off my students. The path of least resistance seems to be changing the JaCoCo branch coverage threshold in your build configuration: If your SUT happens to have no branches, set the branch coverage threshold to 0, otherwise set it to the desired percentage.

```scala
jacocoReportSettings := JacocoReportSettings()
  .withThresholds(
    JacocoThresholds(
      ...
      branch = 0,
      ...
```