CircleCI Workflows to rerun flaky RSpec examples
Tags: ruby, rspec, circleci
Published: 2019-07-10 05:45:34

---
title: CircleCI Workflows to rerun flaky RSpec examples
published: true
description: 
tags: ruby,rspec,circleci
---

Here is the overview. The point is separating jobs to run all test and run failed test. That allows you to rerun the failed test only. It could save the time because You don't need to rerun whole test suite, You can rerun just failed ones.


```ruby
# spec/spec_helper.rb
RSpec.configure do |config|
  config.example_status_persistence_file_path = "spec/examples.txt"
end
```

```yaml
# .circleci/config.yml
version: 2.1

executors:
  ruby:
    docker:
      - image: circleci/ruby

jobs:
  test:
    executor: ruby
    steps:
      - checkout
      - run: gem install rspec
      - run: rspec --failure-exit-code=0
      - persist_to_workspace:
          root: .
          paths:
            - '*'

  test_again:
    executor: ruby
    steps:
      - attach_workspace:
          at: ~/project
      - run: gem install rspec
      - run: rspec --only-failures

workflows:
  test:
    jobs:
      - test
      - test_again:
          requires: [test]
```


The `test` job run whole test suites, that fail sometimes because of flaky test. But it's ok. I'll rerun the failed tests in the `test_again` job.

RSpec has [`example_status_persistence_file_path`](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures) configuration to store the test result.
In this `.circleci/config.yml`, I use [`persist_to_workspace`](https://circleci.com/docs/2.0/configuration-reference/#persist_to_workspace) to save the test result of the `test` job and pass them to the `test_again` job.


`test_again` job `requires` the `test` job, which means `rspec` exit successfully whether test passed or not to run the `test_again` job.

> A list of jobs that must succeed for the job to start
https://circleci.com/docs/2.0/configuration-reference/#requires


To do this, pass `--failure-exit-code=0` option to `rspec` command.

c.f. [\`\-\-failure\-exit\-code\` option \(exit status\) \- Command line \- RSpec Core \- RSpec \- Relish](https://relishapp.com/rspec/rspec-core/v/3-8/docs/command-line/failure-exit-code-option-exit-status)

Then the `test_again` job receives the test result of the `test` job by [`attach_workspace`](https://circleci.com/docs/2.0/configuration-reference/#attach_workspace) and run `rspec` with `--only-failures` option to run failed test only..

c.f. [Only Failures \- Command line \- RSpec Core \- RSpec \- Relish](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures)

## Sample project

I create an example repository

https://github.com/hanachin/rspec-soumen

It has randomly failing test.

```ruby
using Module.new {
  refine(Integer) do
    def 冷やしそうめんがうまい?
      self >= 25
    end
  end
}

RSpec.describe '冷やしそうめん' do
  it { is_expected.to be }

  context '気候変動' do
    let(:temperature) { rand(10..40) }

    subject { temperature.冷やしそうめんがうまい? }

    it { is_expected.to be }
  end
end
```

Then the test failed in `test` job and `test_again` job.

![](https://thepracticaldev.s3.amazonaws.com/i/vu3zjm49bb47nv32ev7d.png)
https://circleci.com/workflow-run/2cced3b3-59c4-4e3e-9f53-33172388b74b

You can see the `test` job runs whole test suite, and it failed but marked as success.

![](https://thepracticaldev.s3.amazonaws.com/i/u06lpfqil6ujyopcwfxu.png)
https://circleci.com/gh/hanachin/rspec-soumen/6


And the `test_again` job runs failed test only.


![](https://thepracticaldev.s3.amazonaws.com/i/x53529a2dmbb4db8f793.png)
https://circleci.com/gh/hanachin/rspec-soumen/7

You can rerun workflows from failed job.
That time the `test_again` job passed.

![](https://thepracticaldev.s3.amazonaws.com/i/hmhd13irngjfvqx3ijum.png)
![](https://thepracticaldev.s3.amazonaws.com/i/08a7j90z8qst51mb43nb.png)
https://circleci.com/workflow-run/09fb370d-bbdd-4648-acba-0bebe5a7bddf

## Conclusion

I separate the test jobs to run all test and run failed test.

Result:

- If tests failed again in `test_again`, I don't need to rerun whole test suite, I can rerun just failed ones.

- I can rerun failed tests without any changes to the test code like `rspec-retry` or extending `Capybara.default_wait_time` or something like that.

- I don't need to investigate flaky test, just rerun the failed tests.
  I know fixing the flaky test is good but it may take tons of times,
  I avoid it.

- My failed tests rerun automatically in the `test_again` job once.
  I can chain the `test_again_again`, `test_again_again_again` jobs as well 😘

- I can take statistics of flaky tests in `test_again` job, maybe.

Cheers🍻