How I reduced the runtime of an RSpec test suite by 15%
Tags: ruby, rails, testing, tdd
Published: 2021-01-11 11:03:15

Some time ago I worked on a project whose goal was to improve the speed of automated tests in a Ruby on Rails application. In this post, I'd like to share my learnings as well as some tips that might help you increase the speed of your project's test suite as well. In my case, the runtime of the test suite decreased by 15%.

Note: all code examples in this article are related to Ruby on Rails, RSpec, and FactoryBot, but some of the general principles can be carried over to other testing frameworks and programming languages.

### Step 1: Find the slowest tests in your test suite

I started out by making a list of the slowest tests in our suite and analyzing each of them. To find the top 10 slowest tests in your suite, run the following command from your project's root (replace 10 with the number of slow tests you wish to get):

```ruby
rspec --profile 10
```
### Step 2: Analyze each of the slow tests

With each test that appears slow relative to other tests, it's important to determine whether the test is slow for legitimate reasons or it is slow because the underlying code is buggy or inefficient. What follows is the rubric I used to analyze the slow tests.

_The test slow, but it's acceptable if:_

* It doesn’t make unnecessary calls to external services, i.e. all necessary external requests are stubbed.

* All objects that are not under test have been replaced with mocks or stubs.

* It writes objects into the database, but this is necessary for the tests to work, i.e. `:create` can’t be replaced with `:build_stubbed` or `:build`.

* Factories used for creating objects don’t create unnecessary Active Record associations. 

* It doesn’t test private methods.

* It doesn’t test logic that belongs to other classes.

* It doesn’t test logic that has already been tested by someone else (an external library or a gem).

* If this is a request/integration spec, it doesn’t test edge cases or cases that have already been tested on the unit level.

* More broadly, the test tests behavior, not implementation.

The slow tests I analyzed faired well under all of the above-mentioned points, except for the test data generation rubric where it was hard to assess the test without further investigation. So my hypothesis was that some of the tests were so slow because they created too much test data that wasn't necessary for the test to work properly. In general, unnecessary test data creation is one of the most common reasons why a test suite gets slow: writing to a database is one of the slowest operations a test can perform (this, and calling external APIs).

### Step 3: Check how many test objects are created during a test run

✅ Add this snippet to `spec_helper.rb`:
```ruby
# spec/spec_helper.rb

config.before(:each, :monitor_database_record_creation) do |example|
  ActiveSupport::Notifications.subscribe("factory_girl.run_factory") do |name, start, finish, id, payload|
    $stderr.puts "FactoryGirl: #{payload[:strategy]}(:#{payload[:name]})"
  end
end
```
✅ Add a meta tag `:monitor_database_record_creation` to the test example or test group that you suspect of creating too many objects:

```ruby
describe '#recipe_complete?' do
  it 'returns true if a recipe is complete', :monitor_database_record_creation do
    # test body
  end
end
```
✅ Run the test.
The console output will tell you how many objects were created for this particular test along with which strategy was used to create them:

```ruby
FactoryGirl: create(:recipe)                                                                                                                                                                                        FactoryGirl: create(:step)
FactoryGirl: create(:step)
FactoryGirl: create(:ingredient)
FactoryGirl: create(:step)
```
❓ At this stage, you might wonder why so many objects are created for this test example or, more specifically, should the `step` object be created twice for this test example to work. Often such duplicate objects happen to be nothing but "mystery guests", i.e. unnecessary—and hard to spot—objects that were written to the database but weren't used by the test. 

In many cases, you should be able to refactor your test and get rid of these "guests" that slow down your test suit.

---

Next, I'll go over the main culprits of unnecessary test data creation and describe how you can deal with them.

### Culprit #1: Using `:create` where `:build` would do the job

![Screenshot 2021-01-10 at 14.59.43](https://dev-to-uploads.s3.amazonaws.com/i/3dxz0qc9te78fup6fr5b.png)
 
 



It's likely that your specs predominantly rely on `:create` strategy. In many of these specs, you might be able to safely replace `:create` with `:build`. These are the cases where the test doesn’t assume the object has actually been written to the database, which is often true for model tests.

In my case, I replaced `:create` with `:build` in the slowest tests whenever it made sense, and also grepped through several of the unit tests for the most often used models in the application. Such models tend to become responsibility magnets and accumulate many methods, and, consequently, many tests.

Word of caution: some blog posts recommend doing a global `find and replace` throughout your whole project and replacing all instances of `:create` with `:build`. I don't recommend doing that: you are likely to end up with numerous failing tests 🤯 Fix the specs one-by-one. It will take longer, but you will be confident of the end result.
 
### Culprit #2: Relying on default `:create` strategy for creating associated objects in factories

You are likely to have several factories where you provide associations as well, and this is where things might get tricky.

By default, even if you call the parent factory with `:build`, the subordinate factory will still be called with `:create`. This means that in your tests, you will always write associated objects to the database even when it isn't necessary for the test.

```ruby
factory :step do
  association :recipe
end

FactoryBot.build(:step)

(0.1ms) begin transation
Recipe Create (0.5ms) INSERT INTO "recipes" DEFAULT VALUES
(0.6ms) commit transaction
```
To avoid this, you can explicitly use the `:build` strategy in the factories whenever possible.
```ruby
factory :step do
  association :recipe, strategy: :build
end
```
### Culprit #3: Not providing associated objects in factory callbacks *explicitly* 

Say we have a factory trait for creating a `recipe` that has two `steps`:

``` ruby 
# spec/factories/recipe_factory.rb
FactoryBot.define do
  factory :recipe do
    # more code
    trait(:with_two_steps) do
      after(:create) do |record|
        record.steps << create_pair(:step,
          account_id: record.account_id,
          body: Faker::Food.description
        )
      end
    end
  end
end
```
The issue with `with_two_steps` trait is that `after(:create)` callback doesn't explicitly specify the `recipe` for the `step` object that is being created. When `step` factory is called from this callback, another `recipe` object is always created, unbeknown to anyone. Why?

Let's have a look at how the `step` factory is designed. For each new `step` an associated `recipe` object is created:

```ruby
# spec/factories/step_factory.rb
FactoryBot.define do
  factory :step do
    account_id: Faker::Number.number
      # more code
    association(:recipe) # this always creates a recipe
  end 
end
```
This is what causes that extra `recipe` to be created in the example above. By explicitly setting the `recipe` object in the `:with_two_steps` trait definition, you can avoid writing an unnecessary extra `recipe` into the database: 

```ruby  
# spec/factories/recipe_factory.rb
FactoryBot.define do
  factory :recipe do
  # more code
    trait(:with_two_steps) do
      after(:create) do |record|
        record.steps << create_pair(:step,
          account_id: record.account_id,
          body: Faker::Food.description,
          recipe: record # set the recipe explicitly
        )
      end
    end
  end
end
```

This might seem like a minor win, but if this trait is called from multiple tests, a fix like this can remove dozens of unnecessary writes to the database. If multiple factories are buggy in similar ways, we can be talking about hundreds of unnecessary writes.

This might also seem like a convoluted tutorial example, but sadly, bugs related to associations in factories are very common, incredibly tricky to spot, and can be even more convoluted in real-life projects.

A good rule of thumb here is this: whenever possible, avoid defining associations in factory definitions. Create the associated objects test by test, as needed. You’ll end up with much more manageable test data. If this is not possible, make sure you are not creating more data than is strictly necessary.

### Culprit #4: Using `let!` incorrectly
It might seem convenient to define all the test data you need in your test examples on top of your test file in this way:

```ruby
let!(recipe_1) { ... }
let!(recipe_2) { ... }
let!(step_1) { ... }
let!(step_2) { ... }

it 'test example that uses recipe_1 and recipe_2 objects' do
end

it 'test example that uses just recipe_1 object' do
end

it 'test example that uses step_1 and step_2 objects' do
end

```
This looks clean and easy to read. However, due to the nature of how `let!` [works](https://relishapp.com/rspec/rspec-core/v/2-5/docs/helper-methods/let-and-let), a new instance of **all** of these test objects will be created before each test example run, even for test cases that don’t require all (**or any!**) of those objects to exist. In a big test group, this innocent mistake might lead to **dozens** of unnecessary writes to the database. 

To fix this, see if it's possible to create separate contexts for related text examples:

```ruby

context 'tests that use recipe_1 and recipe_2 objects' do
  let!(recipe_1) { ... }
  let!(recipe_2) { ... }

  it 'test example that uses recipe_1 and recipe_2 objects' do
  end
  # more test examples
end

context 'tests that use step_1 and step_2 objects' do
  let!(step_1) { ... }
  let!(step_2) { ... }

  it 'test example that uses step_1 and step_2 objects' do
  end
  # more test examples
end

```
***

All of these tips come down to the simple idea of being mindful of the objects your tests generate and never creating more than the bare minimum of data that is necessary for your test to work properly. It's a mindset shift that can take a while to adopt, but that is likely to pay off in the future. After all, a slow test suite kills your team's productivity and makes certain programming approaches like Test-driven development ~~impossible~~ incredibly painful. 

I hope this tutorial will help you speed up your project's tests. If you are aware of other common issues that can slow down a test suite, please share them in the comments below.

Happy testing!