Cables profiling returns: GC.compact & jemalloc
Tags: ruby, profiling, benchmarks
Published: 2020-08-28 10:49:03

> This post introduces a new series—_Unfinished plays_. I have many drafts and incomplete posts I do not plan to finish in the nearest future. Since they still could be interesting/helpful to the community, I decided to release them almost as-is. This is the first one.

In the [first part][part-1], we compared Action Cable memory usage with different VM configurations: `MALLOC_ARENA_MAX`, `malloc_trim`. I promised to continue researching in this direction and evaluate `GC.compact` and jemalloc (via [Fullstaq Ruby][fullstaq]) as well. So, here we are.

## The Benchmark

This time I decided to analyze a little bit different scenario: "A sudden attack during a constant pressure".

"Constant pressure" means a uniform load: clients are connecting, communicating, and disconnecting during the benchmark, but the total number of concurrent users stays about the same.

I have a specific tool for writing such scenarios called [wsdirector][]. It allows you to define a scenario in YML format and run it using a different scale factor.

"A sudden attack" emulates a situation when the number of concurrent connections unexpectedly spikes and then returns to normal. We do that by performing the "WebSocket shootout" scenario from the [first part][part-1].

The source code of the benchmark is available here: https://github.com/anycable/simple-cable-app (see [simiulate.yml](https://github.com/anycable/simple-cable-app/blob/master/features/simulate.yml) and [simulate.rb](https://github.com/anycable/simple-cable-app/blob/master/benchmarks/simulate.rb)).

The server is running within a Docker container. During the benchmark, we capture the container's memory usage and generate a chart in the end (with the help of the awesome [unicode_plot][] library, see [monitor_docker.rb](https://github.com/anycable/simple-cable-app/blob/master/benchmarks/monitor_docker.rb)).

The exact command I used for the benchmarks below is as follows:

```sh
# first, start the app
dip up rails

# or for Fullstaq
dip up rails-fs

# or for Fullstaq with malloc_trim
dip up rails-fs-trim

# pressure
TOTAL=20 SAMPLE=50 N=4 SCALE=200 ruby benchmarks/simulate.rb

# spike (runs twice during the pressure)
cat benchmarks/broadcast.opts | xargs websocket-bench broadcast
```

Below you can find the results I got on my machine (Windows 10 + WSL2, AMD Ryzen 3200G 3.6GHz, 16GB RAM). But first, let's talk about the heap compaction for a bit.

## `GC.compact`

GC compaction has been finally released in Ruby 2.7 (thanks to [Aaron Patterson][tenderlove]). To learn more about this feature, watch one of the latest Aaron's talks, for example, this one from RubyConf 2019:

{% youtube 1F3gXYhQsAY %}

First, I tried to visualize the effect of the `GC.compact` after running a simulation with 1k connected clients:

![No compaction](https://dev-to-uploads.s3.amazonaws.com/i/xnv4sihbiym80jlb73a0.jpg)
<figcaption>Without compaction</figcaption>

![After compaction](https://dev-to-uploads.s3.amazonaws.com/i/6r9j8jpihpiz5q1nrw7v.jpg)
<figcaption>After GC.compact</figcaption>

Awesome! Compaction works! Or does it 🤔

I tried to run the _pressure_ scenario with added `GC.compact` calls after each _wave_ (while other _waves_ are active, i.e., we continue accepting connections, broadcasting messages, etc.) and, unfortunately, found myself in the [segmentation fault](https://github.com/anycable/simple-cable-app/issues/6) situation. Likely, the problem is with C extensions (we have at least Puma and nio4r).

Calling `GC.compact` while there are no active Action Cable clients works fine. So, I had to update the scenario a bit and add a "stop-the-world"-like feature to perform compaction in isolation.

## Fullstaq Ruby & jemalloc

[Jemalloc][] is an alternative memory allocator which could be used instead of `malloc` (used by default in MRI).

> Here are a couple of articles to learn more about jemalloc for Rails applications: [one](https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/) and [two](https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html).

We're going to use a [Fullstaq Ruby][fullstaq] distribution with jemalloc built-in (via [Docker images provided by Evil Martians](https://github.com/evilmartians/fullstaq-ruby-docker)).

## Benchmark results

**NOTE:** Since this is an unfinished play, I'm not providing any explanations/bikeshedding here—just plain results. Feel free to start a discussion in the comments!

We have 6 different setups:

- **baseline**: MRI 2.7.1
- **compact**: MRI 2.7.1 w/ compaction
- **jemalloc**: Fullstaq Ruby 2.7.1 w/ jemalloc
- **jemalloc_compact**: Fullstaq Ruby 2.7.1 w/ jemalloc and compaction
- **trim**: Fullstaq Ruby 2.7.1 w/ malloc_trim
- **trim_compact**: Fullstaq Ruby 2.7.1 w/ malloc_trim and compaction.

Below you can find the results of the benchmarks in different combinations (to better demonstrate the difference).

![w/o compaction](https://dev-to-uploads.s3.amazonaws.com/i/houa6t8t16i7xhxl3fg7.png)
<figcaption>Without compaction</figcaption>

![w/ compaction](https://dev-to-uploads.s3.amazonaws.com/i/xngte18a40zpvwzralpz.png)
<figcaption>With compaction</figcaption>

![Ruby 2.7 w/ and w/o compaction](https://dev-to-uploads.s3.amazonaws.com/i/jh3rqyt86sbjbz0h3ngb.png)
<figcaption>Ruby 2.7.1</figcaption>

![Fullstaq jemalloc w/ and w/o compaction](https://dev-to-uploads.s3.amazonaws.com/i/bt5pduf9cro3gu6cwtrs.png)
<figcaption>FullstaqRuby 2.7.1 with jemalloc</figcaption>

![Fullstaq Ruby 2.7.1 + malloc_trim w/ and w/o compaction](https://dev-to-uploads.s3.amazonaws.com/i/x6u7vt385bsh7ay6lwzs.png)
<figcaption>FullstaqRuby 2.7.1 with malloc_trim</figcaption>

![All together](https://dev-to-uploads.s3.amazonaws.com/i/3h8u15h97ev58rps6dcj.png)
<figcaption>All results combined</figcaption>

[part-1]: https://dev.to/evilmartians/cables-vs-malloctrim-or-yet-another-ruby-memory-usage-benchmark-3emo
[fullstaq]: https://fullstaqruby.org
[unicode_plot]: https://github.com/red-data-tools/unicode_plot.rb
[tenderlove]: https://twitter.com/tenderlove
[Jemalloc]: http://jemalloc.net
[wsdirector]: https://github.com/palkan/wsdirector