Cables vs. malloc_trim, or yet another Ruby memory usage benchmark
Tags: ruby, benchmarks, rails
Published: 2019-03-19 17:00:05

---
title: Cables vs. malloc_trim, or yet another Ruby memory usage benchmark
published: true
canonical_url: https://evilmartians.com/chronicles/cables-vs-malloc_trim-or-yet-another-ruby-memory-usage-benchmark
description: Benchmarking Action Cable memory usage.
tags: ruby, benchmarks, rails
cover_image: https://thepracticaldev.s3.amazonaws.com/i/be51rfmuaat7g2g2mseu.png
---

## Single line, big promise

Memory usage has always been one of the most discussed performance-related topics in the Ruby community.

Last week it got even more attention after [Hongli Lai](https://twitter.com/honglilai) published his excellent ["What causes Ruby memory bloat?"](https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html) post.

The author has proposed a very simple patch to Ruby's (MRI, to be precise) garbage collector: to add a call to [`malloc_trim`](https://linux.die.net/man/3/malloc_trim) at the very end of the full GC cycle and release _some_ allocated memory back to the kernel.

**NOTE**: to make the previous paragraph not sound all Greek to you, read the post by Hongli Lai—he did a really great job explaining complex things; I cannot do better, so I leave you with the link.

The day after the publication, [Noah Gibbs](https://twitter.com/codefolio) did some RRB ([Rails Ruby Benchmark](https://github.com/noahgibbs/rails_ruby_bench)) benchmarks and found that the performance improvement (requests timing) is **just about %1**. Nothing to write home about, right?

On the other hand, like Noah noticed, **"1 line of changed code for a 1% speedup is hard to complain about"**.

So, I decided to measure the impact of the patch on [Action Cable](https://guides.rubyonrails.org/action_cable_overview.html) that is somewhat infamous for its speed and memory usage (check out [AnyCable](https://anycable.io) for a speedier replacement).

I've been doing Action Cable benchmarks [since 2016](https://evilmartians.com/chronicles/anycable-actioncable-on-steroids), when I started my work on [AnyCable](https://anycable.io), so I already [had a setup](https://github.com/anycable/anycable/blob/master/benchmarks/HowTo.md) for that.


I usually do Action Cable benchmarks when some _new things_ appear: either new web servers (e.g., see [the benchmarks](https://github.com/anycable/anycable/blob/master/benchmarks/2018-10-27.md) for [Iodine](https://github.com/boazsegev/iodine#why-not-eventmachine) and [Falcon](https://github.com/socketry/falcon)) or novel Ruby _features_ (like the `malloc_trim` patch or the upcoming [`GC.compact`](https://bugs.ruby-lang.org/issues/15626)) that I plan to try out next time.

The framework itself hasn't changed a lot since its early days (yeah, Rails 6 will ship with [Action Cable testing support](https://github.com/rails/rails/pull/33659), but it has nothing to do with performance). It's mature and stable. And it still eats tons of memory.

**NOTE**: see the ["High-speed Ruby cables" talk](https://www.youtube.com/watch?v=8XRcOZXOzV4) (or [watch the slides](https://speakerdeck.com/palkan/rubyconf-2018-high-speed-cables-for-ruby)) from the latest RubyConf to learn more about Action Cable's (and other _cables_) scalability issues.

## ... and the benchmark

![benchmark is running](https://thepracticaldev.s3.amazonaws.com/i/srjesjc4yp2x2qaqunn9.gif)
<figcaption>The benchmark is running slow</figcaption>

The benchmark is based on Hashrocket's [WebSocket shootout](https://hashrocket.com/blog/posts/websocket-shootout) (more precisely, its _broadcast_ part).

We measure the broadcasting round-trip time (RTT), i.e., how much time does it take for the server to re-transmit the message to all the connected clients–the less the time, the better the _real-time-ness_ of the server.

![RTT](https://thepracticaldev.s3.amazonaws.com/i/6opzfexmn2mspgfbedmi.png)
<figcaption>Broadcasting diagram</figcaption>

As a side-effect, we also capture the application memory usage during the benchmark using [psrecord](https://github.com/astrofrog/psrecord) utility.

I'm using AWS EC2 c3.2xlarge (8 vCPU, 15 GiB RAM) instances for both server and [benchmark client](https://github.com/Arkweid/websocket-bench) and running Puma with the default number of threads (`0:16`) and 8 workers.

The benchmark configuration:
- 10 steps by 1k connections (i.e., 10k connections in the end)
- at every step, 100 random clients initiate broadcasting (with 8x concurrency)
- the size of the message payload is 200B.

I compared three different configurations (all use Rails 6.0.0.beta3):
- Ruby 2.6.2 (_baseline_)
- Ruby 2.6.2 + `MALLOC_ARENA_MAX=2` ([one of the oldest _spells_](https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html) to improve Ruby application memory usage)
- Ruby 2.7-dev (`trunk`) with `malloc_trim` patch.

Let's talk about expectations a bit:
- we expect the memory usage to be lower with either `MALLOC_ARENA_MAX=2` or `malloc_trim` patch
- we also would like to see some positive changes in the performance (lower RTT).

### 🏁 Ready, Steady, Run!

Here is the memory profile for the baseline:

![Action Cable memory usage, Ruby 2.6.2, default VM settings](https://thepracticaldev.s3.amazonaws.com/i/l0l61zo5rsdkb2ajq1pa.png)
<figcaption>Memory (Ruby 2.6.2, default VM settings)</figcaption>

**NOTE**: ignore red lines (should be CPU), it doesn't seem accurate when running multiple child processes (and I don't know how to turn it off 🙂).

We have **~1.8GB** of memory at the end (i.e., after connecting 10k clients and sending 1k messages).

What about RTT?

![Action Cable RTT, Ruby 2.6.2](https://thepracticaldev.s3.amazonaws.com/i/dujv4bxaml4i9rnml46o.png)
<figcaption>RTT (Ruby 2.6.2, default VM settings)</figcaption>

The 95th [percentile](https://en.wikipedia.org/wiki/Percentile) is about **8s for 6k** and **12s for 10k clients**. Let's remember these numbers.

**NOTE**: Why is it not [_monotonic_](http://mathworld.wolfram.com/MonotonicSequence.html)? Mostly due to [_outliers_](https://en.wikipedia.org/wiki/Outlier) (see the grey "Max" line). What causes this? We can only guess, and we will, so keep on reading. 

### Action Cable + `MALLOC_ARENA_MAX=2`

First, memory usage:

![Action Cable memory usage, Ruby 2.6.2, MALLOC_ARENA_MAX=2](https://thepracticaldev.s3.amazonaws.com/i/shlb4prye0eiannv0xi8.png)
<figcaption>Memory (Ruby 2.6.2, MALLOC_ARENA_MAX=2)</figcaption>

Wow! Memory is growing much smoother and hardly reaches **1.3GB**. Amazing, isn't it?

Why haven't I tried this simple trick before?

OK. Let's take a look at the RTT chart:

![Action Cable RTT, Ruby 2.6.2, MALLOC_ARENA_MAX=2](https://thepracticaldev.s3.amazonaws.com/i/prdr398pexhcc9zg737n.png)
<figcaption>RTT (Ruby 2.6.2, MALLOC_ARENA_MAX=2)</figcaption>

The 95th percentile shows **~7s for 6k** and **~11s for 10k clients**.

That means that **both memory usage and performance is better with the less number of malloc arenas**.

Note that the RTT is (almost) _monotonic_. That makes me guess that outliers in the first experiment could also be caused by heavy memory usage (longer GCs runs? allocations?).

### Action Cable + `malloc_trim(0)`

Finally, let's try the patch (that inspired me to run all these benchmarks in the first place):

![Action Cable memory usage, Ruby 2.7-dev, malloc_trim patch](https://thepracticaldev.s3.amazonaws.com/i/mhdvx7e630zvphpyzg2r.png)
<figcaption>Ruby 2.7-dev, malloc_trim patch</figcaption>

Memory usage is about **1.4GB**–not bad, right? But take a look at the blue line–it looks like a handsaw! Memory goes up and **down** (not just up as usually), `malloc_trim` does the trick!

Let's take a look at the performance:

![Action Cable RTT, Ruby 2.7-dev, malloc_trim patch](https://thepracticaldev.s3.amazonaws.com/i/ipzmcmqm1mo0q074ey26.png)
<figcaption>RTT (Ruby 2.7-dev, malloc_trim patch)</figcaption>

We have **~8s for 6k** and **~14s for 10k** as the 95th percentile values.

What? Performance decreased? Maybe, that's because `malloc_trim` ["can be painfully slow"](https://marek.vavrusa.com/memory/)?

Don't panic though. It needs more research. And more time for getting more results.

This benchmark is of _integration_ type, we can only use it to generate hypotheses, for instance: "Although `malloc_trim` positively affects memory usage, it might negatively impact overall performance."

## So, what should I do with Action Cable?

First, if you don't experience any issues with it–do nothing.

If you have some concerns regarding memory usage, first give `MALLOC_ARENA_MAX=2` a fair try.

Didn't help? Well, [AnyCable](https://anycable.io) is the next candidate.

**UPD** (03/22/2019): [ksec](https://dev.to/ksec) [pointed out](https://dev.to/ksec/comment/9i8e) that the benchmarks for AnyCable are missing. So, let's add them.

Here is what we have for AnyCable, by the way:

![AnyCable memory usage](https://thepracticaldev.s3.amazonaws.com/i/4bvvue3tmyj5cfrmbsms.png)
<figcaption>AnyCable memory usage</figcaption>


![AnyCable RTT](https://thepracticaldev.s3.amazonaws.com/i/0h0ynw4su5xt01bjk8yj.png)
<figcaption>AnyCable RTT</figcaption>


P.S. Help me out in improving AnyCable by taking a [small questionnaire](https://evilmartians.typeform.com/to/W35uRm) — does not matter if you are an avid user or have just heard about this speedy drop-in replacement for Action Cable in production, I'd like to know what you think!

---

The next time I plan to experiment with `jemalloc` and [`GC.compact`](https://bugs.ruby-lang.org/issues/15626). [Stay tuned!](https://twitter.com/palkan_tula)

---

Read more dev articles on [https://evilmartians.com/chronicles](https://evilmartians.com/chronicles)!