Measuring performance using BenchmarkDotNet - Part 1
Tags: dotnet, performance, metrics, benchmark
Published: 2021-03-15 18:15:31

# Introduction
We all must build fast software, right? Right? It’s true that microservices tend to introduce latencies - stateless functions mean a whole lot more network calls, and you can wave goodbye to data locality. But a microservice is still dependent on its own code being fast, and at least fast enough. 

In the past we’ve relied on profilers, stopwatches, dedicated performance teams, and sometimes plain old complaints from the field. All of these methods require some form of measurement; unfortunately they tend to be “big picture” performance that lacks detail - and often without concrete scenarios. This gets very expensive very quickly.

Very often, you just want to measure the code’s performance without the baggage of dependencies. You might have a critical piece of code that _absolutely must_ meet certain performance criteria. Measuring such microcode cam obviously be done with profilers - dotTrace, ANTS to name just two.  The problem is they bring their own baggage as well, and worse can’t be easily relied upon in a CI pipeline. So how can you measure microcode performance in CI? Unit tests are a terrible idea, what else is there? Step forward BenchmarkDotNet.

---

# TL;DR

Measure your code’s performance with benchmarks at near zero cost and. All you need are:

* .NET 7 SDK
* VS/VSCode
* BenchmarkDotNet from Nuget

We’ll talk about how to write simple benchmarks, how to run them and how to interpret the results.

---

# What is BenchmarkDotNet?

[BenchmarkDotNet](https://benchmarkdotnet.org/index.html) does what it says on the tin: benchmark .net code. It’s available as a [Nuget packaged library](https://www.nuget.org/packages/BenchmarkDotNet/) for inclusion into your .net console applications. It is very [widely used](https://github.com/dotnet/BenchmarkDotNet#who-use-benchmarkdotnet) by all major players in the .net world, including the [dotnet core runtime project](https://github.com/dotnet/runtime) itself.

# What does a HelloWorld benchmark look like?

Let’s say you have a very basic Fibonacci implementation - and you want to measure its resource usage growth as more numbers are generated.

> By “resource usage” I mean time and memory consumed per method call. 

In other words, you'd want to know how it scales. Here's an implementation of "get the first N Fibonacci numbers":

```csharp
public static IEnumerable<int> GetFibonacci(this int count)
{
    var w = 1;
    var x = 1;

    yield return x;
    foreach (var _ in Enumerable.Range(1, count - 1))
    {
        var y = w + x;
        yield return y;
        w = x;
        x = y;
    }
}
```

> No prizes are sought for best efficiency here. Please **do not** take this as a reference implementation of Fibonacci!

---

To answer the scaling question, we would implement a benchmark, run it and analyse the results. Skipping forward a rendered benchmark report would look something like the below:

![alt text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ccoaxassb9omnz96k1ji.png)

---

## What do all the headers actually mean?

| The column | What it means |
| - | - |
|Method|The name of the code-under-test; a single benchmark may have several methods under test for, e.g. scenarios. This value is lifted directly from your benchmark code.|
|Count|An arbitrary parameter: in this case the number of Fibonacci numbers generated by the method under test.|
|Mean/Error/StdDev/StdError|Execution time statistics. Note that these can be given down to nanoseconds, depending on how fast your code is. Low is best.|
|Min/Q1/Median/Q3/Max|Quartile execution time statistics: note the time units. Low is best.|
|Ops/sec|The number of operations executed per second for the method/parameter combination. High is good.|
|Rank|The fastest performing method/parameter combination. Low is best.|
|Gen 0/1/2|The total number of collections per generation|
|Allocated|Total bytes allocated against all generations|

> Note the header information in the report! It’ll give details on the OS, CPU, .Net version, JIT method and GC configuration. Always benchmark like-for-like!

---

## OK… what do those numbers _really_ mean?

Let’s look at each iteration of ``Count``, and we’re using it here to get the first ``Count`` numbers of the Fibonacci sequence.

Where ``Count`` is 1 the mean execution time is 103.4 nanoseconds. That’s 0.1 microseconds, or 0.0001 milliseconds. I like that: nice and fast. 

Where ``Count`` is 13 (yes, the parameters themselves follow Fibonacci!) the mean time is 407.2 ns: four times what ``Count=1`` is, yet the Count is 13 times bigger. I’ll take that, for now. 

Where ``Count`` is 34 the mean time is 1,077.9 ns, or 1.077 microseconds, or just over 0.0001 milliseconds. That’s taking 2.6 times more time than ``Count = 13``. Let’s compare against ``Count = 1``: ``Count`` is 34 times bigger , yet takes 10 times the time. I’ll take that too. 

If we plot ``Count`` against the time ratio we see this:

![alt text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/maausg26euewio5iowek.png)
 
In other words, time used does not grow as ``Count`` grows. If it did, the lines would be parallel.

> So the benchmarks are showing that the implementation has reasonably acceptable scaling. It's not constant time, but it’s better than O(n) time: a pleasant surprise.

> If you're not satisfied with the performance results, simply make your changes, re-run the benchmarks & re-analyse. That's it.

---

## You haven’t mentioned the memory yet, have you?

Trust me, I’m getting to that. 

> Pay particular attention to memory usage. Garbage collections and memory allocations are as important as sheer speed!

* ``Count=1`` used 128 bytes.
* ``Count=13`` used 312 bytes
* ``Count=34`` used 744 bytes.

If we plot ``Count`` against the allocation growth ratios, we see this:

![alt text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t87hec7nhpqs0z0ultm8.png)
 

> This means the used memory isn’t constant either: the memory used for ``Count=34`` is greater than the memory used for ``Count=1``. Again it's better than O(n). To my mind this is OK, but not great: we need more investigation. It's probably incurred with yield return, but do we want to sacrifice the readability? Probably not, but in any case we’re getting new perspectives on our code. _This is a good thing_.

---

## What other rendered reports can you get?

You can output a markdown version of your report and many other formats; Markown output is GitHub inspired.

You can use the following attributes to output the many different types of rendered reports:

```csharp
[JsonExporterAttribute.Full()]
[CsvMeasurementsExporter()]
[CsvExporter(CsvSeparator.Comma)]
[HtmlExporter()]
[MarkdownExporterAttribute.GitHub()]
```

---

An example of the GitHub Markdown report:

![alt text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u8eh79e4i3zd4zoti91b.png)

Charting is supported through [the R project](https://www.r-project.org/). As R is a world in itself, I’m going to skip the subject. 

> If you want charts, consider importing the rendered JSON into Excel. The ``CsvExporter`` attribute will generate a CSV with the data you need.

---

## Full code example

What does the benchmark code look like using BenchmarkDotNet? It might surprise you to see how simple it is.

BenchmarkDotNet relies on declarative code over which it will reflect. Leaving aside the class attributes (more on those later), note the ``[Params]`` attribute over ``Count`` from the report above, likewise ``[Benchmark]`` and ``Fibonacci()``.

```csharp
using System.Linq;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Csv;

namespace benchmarkdotnetdemo
{
    [InProcess()]
    [MemoryDiagnoser]
    [RankColumn, MinColumn, MaxColumn, Q1Column, Q3Column, AllStatisticsColumn]
    [JsonExporterAttribute.Full, CsvMeasurementsExporter, CsvExporter(CsvSeparator.Comma), HtmlExporter, MarkdownExporterAttribute.GitHub]
    [GcServer(true)]
    public class FibonacciBenchmark
    {
        [Params(1, 2, 3, 5, 8, 13, 21, 34)]
        public int Count { get; set; }

        [Benchmark]
        public void Fibonacci()
        {
            var xs = Count.GetFibonacci().ToList();
        }
    }
}
```

> You’ll notice that the benchmarks have a return type of ``void``  and do not have any assertions. Remember: we’re not proving functional correctness here, we’re measuring resource usage.

---

# Show me the code!

I’ve created a simple BenchmarkDotNet implementation here:

{% github NewDayTechnology/benchmarking-performance-part-1 no-readme %}

There’s only the one C# project in there - ``benchmarkdotnetdemo.csproj`` - that contains the minimal files.

> BenchmarkDotNet will only work if the console project is built with a *Release* configuration, that is with code optimisations applied. Running in *Debug* will result in a *run-time error*.

### Setup

This is the ``Program.cs`` file, and like all C# console apps you need an entry point: 

```csharp
using System;
using BenchmarkDotNet.Running;

namespace benchmarkdotnetdemo
{
    class Program
    {
        static int Main(string[] args)
        {
            try
            {
                BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
                    .Run(args);
                
                return 0;
            }
            catch (Exception ex)
            {
                Console.ForegroundColor = ConsoleColor.Red;
                Console.WriteLine(ex.Message);
                Console.ResetColor();
                return 1;
            }
        }
    }
}
```

---

Aside from the standard method entry point let’s go over it bit by bit.

```csharp
using BenchmarkDotNet.Running;
```

For bootstrapping BenchmarkDotNet, this is the only import you need.

---

```csharp
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
```

This one-line-to-rule-them-all will perform all command line parsing, all help, all benchmark execution and all report generation. 

One point here is ``.FromAssembly(typeof(Program).Assembly)`` - this informs BenchmarkDotNet of its benchmark search scope. Benchmarks are internally discovered by reflection - you’ll see soon enough.

> NOTE: If you were to run the project without any command line arguments, BenchmarkDotNet will assume an interactive CLI. 

> ``.Run(args)`` returns a sequence of report objects comprised of the same data used for rendered reports: I’ve excluded them for simplicity. If you want to run benchmarks and fail CI builds if performance dips they are your first place to look.

---

## Create a new benchmark

There is a file called ``SimpleBenchmark.cs``. Let’s have a look.

```csharp
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Csv;

namespace benchmarkdotnetdemo
{
    [InProcess()]
    [MemoryDiagnoser]
    [RankColumn, MinColumn, MaxColumn, Q1Column, Q3Column, AllStatisticsColumn]
    [JsonExporterAttribute.Full, CsvMeasurementsExporter, CsvExporter(CsvSeparator.Comma), HtmlExporter, MarkdownExporterAttribute.GitHub]
    [GcServer(true)]
    public class SimpleBenchmark
    {
        [Benchmark]
        public void NoopTest() { }

        [Benchmark]
        public int AddTest() => int.MaxValue + int.MinValue;

        [Benchmark]
        public int MultiplyTest() => 11 * 3;
    }
}
```

---

#### FibonacciBenchmark.cs

Just for completeness: note the similar declarations as ``SimpleBenchmarks.cs``. In this case, we’re adding a ``[Params]`` parameter to support benchmark permutations. 

```csharp
using System.Linq;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Csv;

namespace benchmarkdotnetdemo
{
    [InProcess()]
    [MemoryDiagnoser]
    [RankColumn, MinColumn, MaxColumn, Q1Column, Q3Column, AllStatisticsColumn]
    [JsonExporterAttribute.Full, CsvMeasurementsExporter, CsvExporter(CsvSeparator.Comma), HtmlExporter, MarkdownExporterAttribute.GitHub]
    [GcServer(true)]
    public class FibonacciBenchmark
    {
        [Params(1, 2, 3, 5, 8, 13, 21, 34)]
        public int Count { get; set; }

        [Benchmark]
        public void Fibonacci()
        {
            var xs = Count.GetFibonacci().ToList();
        }
    }
}
```

---

### How are benchmarks executed?

Without going into too much detail, BenchmarkDotNet will attempt to run your benchmarks many many times over to settle on mean and median values. 

When you run the benchmarks you may first be confused by just how many iterations are involved, so let’s give a simplistic explanation. Modern OSs are preemptive multitaskers, CPUs have pipeline caches as well as instruction reordering features. .NET itself has the JIT compiler. This means that *no single execution of code can be relied upon to give a canonical result*.

> This is part of the reason why unit tests are terrible for benchmarking! They only run once and incur their own (unaccounted) overheads.

BenchmarkDotNet will run warm up iterations before it can take representative values. These show up as various stages: OverheadJitting  & WorkloadJitting, WorkloadPilot, OverheadWarmup, OverheadActual.

JIT comes at a cost: the first time any .NET code executes it must first be JIT compiled. The more complex the code the higher the JIT cost, usually showing as CPU and time costs. As we’re interested only in runtime performance, these steps eliminate JIT costs from measurements.

In the same vein  other warmup steps are run to eliminate other “once only” costs, for instance to warm up pipelining caches. 

![alt text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t434gva68vqgk4nfpd0m.png)

---

After these steps have completed, BenchmarkDotNet will iterate these operations to yield the final statistics; these are shown as WorkloadActual steps.

![alt text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kkkshojo5dsobe2ogndi.png)

---

> If you want more detail, please refer to [BenchmarkDotNet’s own documentation](https://benchmarkdotnet.org/articles/guides/how-it-works.html). In these code samples we’re using the default ``Throughput`` strategy for microbenchmarking.

---

### How long does it take?

It depends ;) Simple calculations, such as in the demo project, will run in under a minute. Adding permutations (such as with ``[Params]``)  will linearly increase the benchmarking time, as each parameter will be benchmarked in its own right.

With that in mind, it’s quite clear that resource hungry algorithms, benchmarked with a large variety of parameters, will take a considerable amount of time. 

> Don’t expect to parallelise BenchmarkDotNet: it runs benchmarks sequentially. Thread context switching is itself a cost and extremely difficult to compensate for.

---

# What have we learned?

* We’ve seen how to get BenchmarkDotNet 
* We’ve seen how to integrate it in a simple console application
* We’ve seen the minimum work needed to build benchmarks
* We’ve had a taste of the reports and inferences we can gain from BenchmarkDotNet

---

# Next Steps

How to incorporate into CI?

---

# More Information
* [What is benchmarking - Wiki](https://en.wikipedia.org/wiki/Benchmark_(computing))
* [BenchmarkDotNet](https://benchmarkdotnet.org/index.html)
* [BenchmarkDotNet on Github](https://github.com/dotnet/BenchmarkDotNet)
* [A real world use case of BenchmarkDotNet](https://stebet.net/benchmarking-and-performance-optimizations-in-c-using-benchmarkdotnet/)
* GitHub repository: {% github NewDayTechnology/benchmarking-performance-part-1 %}