How we improved our Elixir build speed by 5x
Tags: softwareengineering, elixir, ci
Published: 2020-03-02 16:14:55

---
title: How we improved our Elixir build speed by 5x
published: true
date: 2019-10-08 20:53:00 UTC
tags: Software Engineering,Elixir,CI
canonical_url: https://www.christianblavier.com/how-we-improved-our-elixir-build-speed-by-5x/
---

 ![How we improved our Elixir build speed by 5x](https://www.christianblavier.com/content/images/2019/12/1__NyTuevjXFDbns3htQrN3g.jpeg)

_I’m really happy and proud to say that since this story was first posted, I received great feedback and relay throughout the Elixir community! The last advice given in this article has indeed been taken as an inspiration by_ [_José Valim_](https://medium.com/u/f31378845318?source=post_page-----d45393c6700f----------------------) _to add a_ [_test partitioning feature_](https://github.com/elixir-lang/elixir/pull/9422) _to Elixir_ 🤗

* * *

# Some context

I have been using Elixir + Phoenix on a daily basis for 3 years now, and almost since day one, my team and I have set up continuous integration (CI) and continuous delivery (CD) processes.

Since we’re deploying our app on Heroku, we’ve been using:

- **SemaphoreCI as our CI tool** (because [SemaphoreCI](https://semaphoreci.com/) is available as a Heroku add-on, simple as that!)
- **Heroku native pipelines to perform CD** whenever CI is successful on our Git master branch

Although our CD process has always been very reliable, our CI process gradually started to under-perform, becoming both slower and unreliable. Like boiling frogs, we did not really pay attention until very recently when **we had to wait almost for 1 hour before being able to deploy our code to staging** : each CI run would take 15 minutes, and random test failures forced us to re-run each build a few times before getting the green light. This drove us really crazy, and led us to take action!

In this article, we’ll talk about the actions we took in order to **improve an unreliable** — randomly failing — **15 minutes long CI process** to a **solid and steady 3 minutes build.**

# #1 — Improve worst tests

First action is to find out the culprits: which tests are performing really slowly and which are failing randomly?

Running `mix test --trace` is very useful as it gives you detailed execution time, per test. But my favorite is `mix test --slowest 10` which prints the list of the 10 worst performing tests. I have no general purpose advice to give about fixing these tests, but the slowest are usually quite easy to improve.

Regarding randomly failing tests, two things can really help:

- a simple **bash loop** to run a single test over and over. If it returns anything else than 0, the test is unreliable.
<!--kg-card-begin: markdown-->
```
(for i in {1..10}; do mix test test/lib/my_app/features/my_browser_test.exs:10 > /dev/null; echo $?; done) | paste -s -d+ - | bc

```
<!--kg-card-end: markdown-->
- a `wait_until` function that helps to fix unreliable browser tests (we're using [Hound](https://github.com/HashNuke/hound) with _chromedriver_). Here is the [gist](https://gist.github.com/cblavier/1acccfd914e4cd6a51af0cf500dff6f8).

A last ugly trick 🙈 I am ashamed of and I urge you not to use is to give another chance to your failing tests! Running your tests with `mix test || mix test --failed` will automatically rerun any failed test and still give you a 0 exit code if it passes on the second time.

![How we improved our Elixir build speed by 5x](https://www.christianblavier.com/content/images/2019/12/cat.gif)

_Pro tip: I strongly encourage you to read_ `mix help test` _output: hidden gems in there!_

# #2 — Cache all things

At this stage, the build became way more reliable (99% success rate) but still really slow (~ 12 min).

My first attempt to improve this was to see if the grass was greener at GitHub’s with their brand new [GitHub Actions](https://github.com/features/actions), and if it would result in an improved build duration. Although promising, I did not manage to run any of our _chromedriver_ tests with their product, and execution time was anyway quite similar.

I then decided to go back to Semaphore and migrate from the classic plan to their new [SemaphoreCI 2.0](https://semaphoreci.com/). The new Semaphore (like GitHub Actions _btw_) requires you to describe your CI pipeline in a YAML file committed along your source code. In this YAML file you can run docker containers, arbitrary shell commands or specific semaphore commands, including the very useful [cache commands](https://docs.semaphoreci.com/article/149-caching).

By using `cache store` and `cache restore` at strategic locations you can make most of your builds way faster:

- by not re-downloading Elixir dependencies
- not re-compiling all Elixir code, only updated code
- not re-downloading NPM/Yarn dependencies

_Pro tip: have a look at fail\_fast and auto\_cancel Semaphore settings. Real time savers!_

# #3 —Leverage on parallelism

Caching really helped, and at this step we could have stopped improving the process as our build time already **went down to 6 minutes.**

But reading SemaphoreCI documentation is insightful, and their concepts of blocks and jobs that can run either sequentially or in parallel suggest there is still room for improvement!

Our build was really straight-forward then:

![How we improved our Elixir build speed by 5x](https://www.christianblavier.com/content/images/2019/12/semaphore1.png)

The first improvement is to run in parallel what can be done in parallel:

- `mix credo` (the Elixir code analysis tool) and `mix test` can be run at the same time. With a fail-fast strategy that stops running tests whenever _credo_ fails, it can be a real time saver!
- JS & CSS assets can be compiled at the same time than Elixir code: it can save up to one minute.

The final step is to split our tests in different batches to also run them in parallel. I wrote this [custom test runner](https://gist.github.com/cblavier/4e6fa2939145bfb56a02b2bd413fec60) (which simply wraps `mix test` shell command) that splits all test files in N batches (by hashing their filename) and only runs a single batch out of N:

<!--kg-card-begin: markdown-->
```
# Run all tests in 4 random batches
$> mix n_test 1 4
$> mix n_test 2 4
$> mix n_test 3 4
$> mix n_test 4 4

```
<!--kg-card-end: markdown-->

You can then leverage on SemaphoreCI [parallelism](https://docs.semaphoreci.com/article/50-pipeline-yaml#parallelism) to run a single task N times with an incremental index passed as an environment variable.

Our CI finally looks like this, running at a solid and steady 3mn pace!

![How we improved our Elixir build speed by 5x](https://www.christianblavier.com/content/images/2019/12/semaphore2.png)

_Pro tip: save time at learning Semaphore by reading our_ [_build pipeline_](https://gist.github.com/cblavier/db7fb947aa8158449608557faafb1216) _setup_

# Wrapping up

It has been known [for years](https://martinfowler.com/articles/continuousIntegration.html) how important Continuous Integration is, for any size of software team. But it should be a source of trust, [happiness](https://rubyonrails.org/doctrine/#optimize-for-programmer-happiness) and increased productivity, never a source of frustration!

So start paying attention, and maybe some of the guidelines in this post will help you at improving your CI build!

_NB - I was not paid by SemaphoreCI to write this article, so feel free to experiment with any other CI software, as long as it offers you caching & parallelism._