Decreasing latency noise and maximizing performance during end-to-end benchmarking
Tags: hasura, performance, powermanagement, intel
Published: 2020-04-15 03:17:28

---
title: Decreasing latency noise and maximizing performance during end-to-end benchmarking
published: true 
date: 2020-04-14 16:31:59 UTC
tags: hasura, performance, powermanagement, intel 
canonical_url: https://hasura.io/blog/decreasing-latency-noise-and-maximizing-performance-during-end-to-end-benchmarking/
---

![](https://hasura.io/blog/content/images/2020/04/Hasura_Perf_1.jpg)

> I was taken by surprise to find that benchmarking graphql-engine at 100 RPS was such a light load that power management was skewing my measurements dramatically (and worse, misleading me when e.g. having the browser open (i.e. more load) actually caused a dramatic performance improvement, causing me to think I'd made a performance breakthrough!).

While benchmarking some tweaks to graphql-engine we noticed some confusing and misleading/inconsistent results which led us down a bit of a rabbit hole. This is a summary of what we've learned so far, what is left to try, and what we've tried which didn't seem to have much effect.

We're starting to explore two interrelated things:

- understanding how to maximize performance in a running graphql-engine generally; this is a type of noise reduction as well for the purposes of benchmarking, but is also obviously beneficial for best-practices to share with users or to help them measure performance
- to understand how to run graphql-engine such that latency is consistent (but possibly much slower), so that we can be confident that a change we've made was actually beneficial or not

Some of the results here probably apply to production deployments of graphql-engine (and other services) as well; but we don't have concrete recommendations there yet. If you have success with any of these techniques in production, let us know!

## Things that were very effective

**Background** : modern intel processors have extremely sophisticated power management that modifies the clock frequency and powers up and down subsystems dynamically and constantly (many times per second). There are knobs on modern linux for some amount of control over how all this behaves: the `intel_idle` driver allows some control over c-states (processor idle states), while `intel_pstate` deals with [p-states](https://software.intel.com/en-us/blogs/2008/05/29/what-exactly-is-a-p-state-pt-1). See the references at the bottom for more.

I was taken by surprise to find that benchmarking graphql-engine at 100 RPS was such a light load that power management was skewing my measurements dramatically (and worse, misleading me when e.g. having the browser open (i.e. more load) actually caused a dramatic performance improvement, causing me to think I'd made a performance breakthrough!).

We can use  `turbostat` (from the `cpupower` package) to look at power management states of the machine during an execution of some load, with:

```
$ turbostat --interval 0.1 sleep 120
```

For my laptop (with Core i7-3667U), without tweaks, running the 100 RPS benchmark:

<!--kg-card-begin: html-->
```
    Core CPU Avg\_MHz Busy% Bzy\_MHz TSC\_MHz IRQ SMI CPU%c1 CPU%c3 CPU%c6 CPU%c7 CoreTmp PkgTmp GFX%rc6 GFXMHz Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt - - 110 9.44 1167 2494 0 0 14.44 4.30 0.01 71.81 57 57 0.00 0 4.58 1.66 3.58 35.50 4.14 1.16 0.31 0 0 113 9.80 1161 2494 0 0 13.99 3.98 0.00 72.23 56 57 0.00 0 4.58 1.66 3.58 35.50 4.14 1.16 0.31 0 1 103 8.92 1160 2494 0 0 14.87 1 2 113 9.85 1151 2494 0 0 14.12 4.62 0.02 71.40 57 1 3 110 9.18 1196 2494 0 0 14.79 Thread Stats Avg Stdev Max +/- Stdev Latency 2.44ms 833.69us 13.82ms 78.94% Req/Sec 26.36 67.81 400.00 85.22% 6004 requests in 1.00m, 2.35MB read
```
<!--kg-card-end: html-->

...we can see the processor was busy (in C-state 0) less than 10% of the time, while it spent over 70% of the time in the deep C-7 state, which is slow to wake and do useful work.

### "performance" p-state governor

`intel_pstate` offers two "governors" or modes. Switching it to "performance", with...

```
$ sudo cpupower frequency-set -g performance

```

...significantly reduces latency as the CPU is (waves hands) more ready to ramp up and do work when it comes:

<!--kg-card-begin: html-->
```
    Core CPU Avg\_MHz Busy% Bzy\_MHz TSC\_MHz IRQ SMI CPU%c1 CPU%c3 CPU%c6 CPU%c7 CoreTmp PkgTmp GFX%rc6 GFXMHz Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt - - 99 4.49 2206 2494 0 0 13.02 5.00 0.01 77.47 55 55 0.00 0 5.35 2.06 3.97 48.43 4.40 1.67 0.13 0 0 104 5.01 2073 2494 0 0 12.72 5.05 0.02 77.20 54 55 0.00 0 5.35 2.06 3.97 48.43 4.40 1.67 0.13 0 1 93 3.96 2361 2494 0 0 13.77 1 2 103 4.99 2071 2494 0 0 12.31 4.95 0.00 77.74 55 1 3 95 4.00 2386 2494 0 0 13.30 Thread Stats Avg Stdev Max +/- Stdev Latency 1.52ms 1.14ms 33.86ms 97.87% Req/Sec 26.48 67.00 333.00 85.14% 6004 requests in 1.00m, 2.35MB read 
```
<!--kg-card-end: html-->

Notice `Bzy_MHz` is close to the advertised clock speed here.

### Prohibit deep sleep states

Finally we can keep the processor from entering deep sleep states with:

```
$ sudo cpupower idle-set -D10  

```

(Note: you can re-enable all idle states with : `sudo cpupower idle-set -E`)

<!--kg-card-begin: html-->
```
    Core CPU Avg\_MHz Busy% Bzy\_MHz TSC\_MHz IRQ SMI CPU%c1 CPU%c3 CPU%c6 CPU%c7 CoreTmp PkgTmp GFX%rc6 GFXMHz Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt CorWatt GFXWatt - - 68 2.28 3000 2494 0 0 97.72 0.00 0.00 0.00 64 64 0.00 0 0.00 0.00 0.00 0.00 8.74 5.86 0.11 0 0 71 2.36 3000 2494 0 0 97.64 0.00 0.00 0.00 60 64 0.00 0 0.00 0.00 0.00 0.00 8.74 5.86 0.11 0 1 67 2.25 3000 2494 0 0 97.75 1 2 73 2.43 3000 2494 0 0 97.57 0.00 0.00 0.00 64 1 3 63 2.09 3000 2494 0 0 97.91 Thread Stats Avg Stdev Max +/- Stdev Latency 1.25ms 404.96us 6.43ms 68.94% Req/Sec 25.66 63.97 333.00 84.95% 6004 requests in 1.00m, 2.35MB read
```
<!--kg-card-end: html-->

Here we see a smaller but significant reduction in latency. We can see that we're spending nearly all our time in the shallower C1 state, ready to wake up quickly.

We can graph latency improvements from these two changes. Here we're measuring from instrumented graphql-engine:

![plot](https://user-images.githubusercontent.com/210815/69687240-38a69e00-1090-11ea-800b-8ea34be1e526.png)

**FYI** Here are all AWS instances under $3/hr that allow setting both cstate/pstate:

```
                       CPUS ECUs RAM
  $1.591 c4.8xlarge 36 132 60 GiB
  $1.872 h1.8xlarge 32 99 128 GiB
  $2.00 m4.10xlarge 40 124.5 160 GiB
  $2.128 r4.8xlarge 32 99 244 GiB
  $2.496 i3.8xlarge 32 99 244 GiB

```

## Things that seemed to have little or dubious effect 

...but might when e.g. graphql-engine is carefully isolated on CPUs:

- `echo 0 | sudo tee /proc/sys/kernel/randomize_va_space`
- `+RTS -qa -qm` and/or pinning the parent haskell process with `taskset -cp $(ps -F <pid> | awk '{print $7}' | tail -n1) <pid>`  
I'm still trying [to understand what's happening here](https://stackoverflow.com/questions/59057716/understanding-the-child-process-of-a-threaded-ghc-haskell-program), but launching a new OS thread (at ~5 us latency) is unlikely to be a bottleneck, however maybe it's associated with some other type of blocking
- `echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled` : this resulted in 15% regression for small return payloads, and 15% improvement for large ones. Worth revisiting.

## Things that are likely to be effective but not explored

carefully isolating graphql-engine on particular CPUs. See also link dump below.  
Copied from LLVM benchmarking docs:

> Use [https://github.com/lpechacek/cpuset](https://github.com/lpechacek/cpuset) to reserve cpus for just the  
> program you are benchmarking. If using perf, leave at least 2 cores  
> so that perf runs in one and your program in another::

> cset shield -c N1,N2 -k on

> This will move all threads out of N1 and N2. The `-k on` means  
> that even kernel threads are moved out.

> Disable the SMT pair of the cpus you will use for the benchmark. The  
> pair of cpu N can be found in  
> `/sys/devices/system/cpu/cpuN/topology/thread_siblings_list` and  
> disabled with::

```
cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | \
awk -F, '{print $2}' | \
sort -n | \
uniq | \
( while read X ; do echo $X ; echo 0 | sudo tee /sys/devices/system/cpu/cpu$X/online ; done )

```

> Run the program with::

```
cset shield --exec -- perf stat -r 10 <cmd>

```

> This will run the command after `--` in the isolated cpus. The  
> particular perf command runs the `<cmd>` 10 times and reports  
> statistics.

### ...and not practical for actual deployment but potential for more stable (and slower) latencies

- disable turboboost: `echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo`
- consider using `tmpfs` for our benchmarking postgres instance so we never touch disk

## Things that seemed to be a waste of time

fiddling with `/sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq` and `.../intel_pstate/min_perf_pct`. It's not clear if this did anything. It certainly didn't produce a fixed CPU frequency.  From [https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt](https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt) :

> For contemporary Intel processors, the frequency is controlled by the processor itself and the P-State exposed to software is related to performance levels.  The idea that frequency can be set to a single frequency is fictional for Intel Core processors. Even if the scaling driver selects a single P-State, the actual frequency the processor will run at is selected by the processor itself.

# References dump

- Controlling placement of processes and isolating them: [https://webcache.googleusercontent.com/search?q=cache:SmEke2ayDOAJ:https://access.redhat.com/solutions/2884991+&cd=1&hl=en&ct=clnk&gl=us](https://webcache.googleusercontent.com/search?q=cache:SmEke2ayDOAJ:https://access.redhat.com/solutions/2884991+&cd=1&hl=en&ct=clnk&gl=us)
- solid info on p/c-states [https://metebalci.com/blog/a-minimum-complete-tutorial-of-cpu-power-management-c-states-and-p-states/](https://metebalci.com/blog/a-minimum-complete-tutorial-of-cpu-power-management-c-states-and-p-states/)
- Kernel pstate driver docs: [https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel\_pstate.html](https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_pstate.html)  
and [https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt](https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt)
- restricting to c0 doesn't play nice with HT !:  
[https://www.codeblueprint.co.uk/2017/03/06/intel\_idle-max\_cstate-considered-harmful.html](https://www.codeblueprint.co.uk/2017/03/06/intel_idle-max_cstate-considered-harmful.html)  
[https://software.intel.com/en-us/forums/software-tuning-performance-optimization-platform-monitoring/topic/535130](https://software.intel.com/en-us/forums/software-tuning-performance-optimization-platform-monitoring/topic/535130)
- controlling cstates : [https://wiki.ntb.ch/infoportal/\_media/embedded\_systems/ethercat/controlling\_processor\_c-state\_usage\_in\_linux\_v1.1\_nov2013.pdf](https://wiki.ntb.ch/infoportal/_media/embedded_systems/ethercat/controlling_processor_c-state_usage_in_linux_v1.1_nov2013.pdf)
- constellation of kernel cargo cult crankery:  
[https://www.codethink.co.uk/articles/2018/configuring-linux-to-stabilise-latency/](https://www.codethink.co.uk/articles/2018/configuring-linux-to-stabilise-latency/)  
[https://gitlab.com/CodethinkLabs/determinism/wikis/Kernel Tunings](https://gitlab.com/CodethinkLabs/determinism/wikis/Kernel%20Tunings)  
[http://epickrram.blogspot.com/2015/09/reducing-system-jitter.html](http://epickrram.blogspot.com/2015/09/reducing-system-jitter.html)  
[http://epickrram.blogspot.com/2015/11/reducing-system-jitter-part-2.html](http://epickrram.blogspot.com/2015/11/reducing-system-jitter-part-2.html)

Enjoyed this article? Join us on [Discord](discord.gg/hasura) for more discussions on Hasura & GraphQL! 

Sign up for our [newsletter](http://eepurl.com/dBUfJ5) to know when we publish new articles.