An unexpected performance regression
Tags: rust, performance
Published: 2019-09-16 19:16:58

---
title: An unexpected performance regression
published: true
description: A short story on how compiler updates can cause unexpected performance regressions.
tags: rust, performance
---


Performance regressions are something that I find rather hard to track in an automated way. For the past years, I have been working on [a tool called `fd`](https://github.com/sharkdp/fd), which is aiming to be a *fast* and user-friendly (but not necessarily feature-complete) alternative to [`find`](https://www.gnu.org/software/findutils/).

As you would expect from a file-searching tool, `fd` is an I/O-heavy program whose performance is governed by external factors like filesystem speed, caching effects, as well as OS-specific aspects. To get reliable and meaningful timing results, I developed a [command-line benchmarking tool called `hyperfine`](https://github.com/sharkdp/hyperfine) which takes care of things like warmup runs (for hot-cache benchmarks) or cache-clearing preparation commands (for cold-cache benchmarks). It also performs an analysis across multiple runs and warns the user about outside interference by detecting statistical outliers¹.

But this is just a small part of the problem. The real challenge is to find a suitable collection of benchmarks that tests different aspects of your program across a wide range of environments. To get a feeling for the vast amount of factors that can influence the runtime of a program like `fd`, let me tell you about one particular performance regression that I found recently².

I keep a small collection of old `fd` executables around in order to quickly run specific benchmarks across different versions. I noticed a significant performance regression between `fd-7.0.0` and `fd-7.1.0` in one of the benchmarks:

![performance regression between 7.0 and 7.1](https://thepracticaldev.s3.amazonaws.com/i/ep14199yxpaswdcttop2.png)

I quickly looked at the commits between 7.0 and 7.1 to see if there were any changes that could have introduced this regression. I couldn't find any obvious candidates.

Next, I decided to perform a small binary search by re-compiling specific commits and running the benchmark. To my surprise, I wasn't able to reproduce the fast times that I had measured with the precompiled binaries of the old versions. Every single commit yielded slow results!

There was only one way this could have happened: the old binaries were faster because they were compiled with an *older version of the Rust compiler*. The version that came out shortly before the `fd-7.1.0` release was [Rust 1.28](https://blog.rust-lang.org/2018/08/02/Rust-1.28.html). It made a significant change to how Rust binaries were built: it dropped `jemalloc` as the default allocator.

To make sure that this was the root cause of the regression, I re-enabled `jemalloc` via the [jemallocator](https://crates.io/crates/jemallocator) crate. Sure enough, this brought the time back down:

![Runtime back to normal in v7.4.0](https://thepracticaldev.s3.amazonaws.com/i/9kkq8xll17pexngmmkl8.png)

Subsequently, I ran the whole "benchmark suite". I found a consistent speed up of up to 40% by switching from the system-allocator to jemalloc (see results below). The recently released [`fd-7.4.0`](https://github.com/sharkdp/fd/releases) now re-enables jemalloc as the allocator for `fd`.

Unfortunately, I still don't have a good solution for automatically keeping track of performance regressions - but I would be very interested in your feedback and ideas.

### Benchmark results

Simple pattern, warm cache:

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `fd-sysalloc '.*[0-9]\.jpg$'` | 252.5 ± 1.4 | 250.6 | 255.5 | 1.26 |
| `fd-jemalloc '.*[0-9]\.jpg$'` | 201.1 ± 2.4 | 197.6 | 207.0 | 1.00 |

Simple pattern, hidden and ignored files, warm cache:

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `fd-sysalloc -HI '.*[0-9]\.jpg$'` | 748.4 ± 6.1 | 739.9 | 755.0 | 1.42 |
| `fd-jemalloc -HI '.*[0-9]\.jpg$'` | 526.5 ± 4.9 | 520.2 | 536.6 | 1.00 |

File extension search, warm cache:

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `fd-sysalloc -HI -e jpg ''` | 758.4 ± 23.1 | 745.7 | 823.0 | 1.40 |
| `fd-jemalloc -HI -e jpg ''` | 542.6 ± 2.7 | 538.3 | 546.1 | 1.00 |

File-type search, warm cache:

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `fd-sysalloc -HI --type l ''` | 722.5 ± 3.9 | 716.2 | 729.5 | 1.37 |
| `fd-jemalloc -HI --type l ''` | 526.1 ± 6.8 | 517.6 | 539.1 | 1.00 |

Simple pattern, cold cache:

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `fd-sysalloc -HI '.*[0-9]\.jpg$'` | 5.728 ± 0.005 | 5.723 | 5.733 | 1.04 |
| `fd-jemalloc -HI '.*[0-9]\.jpg$'` | 5.532 ± 0.009 | 5.521 | 5.539 | 1.00 |



<small>¹ For example, I need to close Dropbox and Spotify before running `fd` benchmarks as they have a significant influence on the runtime.</small>


<small>² As stated in the beginning, I don't have a good way to automatically track this. So it took me some time to spot this regression :-(</small>