Reasoning about slow tests
Tags: testing, devops
Published: 2020-08-12 06:58:09

## Introduction

<a href="https://twitter.com/dpaulmerrill">Paul Merrill</a> asked a very interesting question on Twitter last week. 

{% twitter 1290641404058906624 %}

I decided that as a new Software Test Engineer, I wanted to push myself and try and answer that question, pulling from my experience as a marketer to answer the last part of the problem. 

About that...

Paul says this in one of his replies.

{% twitter 1290643543720112132 %}

And then we get into a whole discussion about business priorities.

## But why is it important that tests are (as) fast as they can (realistically) be?

- The whole development process isn't slowed down waiting for the tests to complete.
- Running tests isn't skipped because they're too slow. 
- Developers aren't tempted to context switch to other tasks while they wait, so they should be able to get feedback on whether it is ok within 5 minutes. (A full suite of regression integration tests may take longer.)

### Answering the question the first time round

First, I did what the applicants were currently doing, which was to immediately list a set of reasons why the tests could have sub-optimal speed, which I will actually get into later.

Then I realised what I was doing, and took a step back.

I looked through the other responses to Paul's question and his response to that to gauge what content he was looking for in an answer. Some that I found useful in structuring my response are below.

<a href="https://bughuntersam.com/">Sam Connelly</a>
{% twitter 1290885698208165888 %}
<a href="https://www.gesqa.com/">Graham Ellis</a> 
{% twitter 1290647108215812097 %}
<a href="https://twitter.com/BenOconis">Ben Oconis</a>
{% twitter 1290653907375607817 %}

Then, I put together a framework of how to answer the question, with some realism from how I approached the problem initially.

{% twitter 1290655943181639680 %}

In this blog post, I want to flesh out that framework, but this time, I ask questions first. Some of the questions I would ask are addressed in the below tweet. 

{% twitter 1290664199077728262 %}

Let's get started.

### **What** is slow?

Does this apply only to functional tests, or to non-functional tests as well?

How many test cases? Is it one? Is it a stack of them? Is it all of them?

Does it apply to presentation layer, service, integration, unit?

Are there just too many tests? Do they all need to be run when they’re run? 

Is it the tests that are slow, or are they slow because the code is slow, or has testability issues? 

It could also be the configuration, for example, the test fixture, or the machine (the test environment).

### How do you **define** slow?

Is there a performance benchmark set for the speed of the tests?

What is the purpose of the tests? Are they smoke tests? Are they regression tests?

How much slower is it than the benchmark?

### How do you find out **that** a test case is slow?

Is there historical data from previous runs of the test?

Is it easily sortable by time taken per test?

What code profiling tools are available? 

### How do you find out **why** a test case is slow?

By looking at the code for the tests, the code that it's testing, the environment it’s running in, and various configuration settings and scripts.

If there are multiple test cases that are slow, what do they have in common? Do they have something interesting in common? 

Here are some of the reasons test cases can be slower than expected, with more to be added as I understand them.

#### All tests

Is the test data created smartly? How is it created and cleaned up? Is it cleaned up? Are there race conditions? Are there non-atomised tests? You might find performance tests FASTER than they should be if you're using already-created data.

#### UI tests

1. If your E2E tests are using xpaths when they could be using id, name, or even CSS selectors, it's the tests that are slow. I find using accessibility ids that are very unlikely to change is more likely to make UI test code last, without false negatives over time. 

2. Perhaps there are tests run as E2E tests when they would be better as unit, integration, or API tests, which run faster than E2E tests.

3. Are the tests using generic wait statements where they could be using conditional waits? Are there unnecessary waits in the tests?

4. Are you using your UI to build up state? It's faster to hit the DB & API. 

5. Is the specific action/event itself taking longer than the benchmark to load?

#### DB tests

1. Are there long-running queries? Are there inefficient queries?

### When slow tests aren't a problem

From my currently limited perspective, when total test time still comes in under 2 minutes for smoke tests. This of course does not apply to life-to-death software like medical admin, medical devices, or vehicle AI.

From a business perspective, they're not a problem if it's not causing blocks for the engineering team yet.

### How do you fix slow-running tests?

The first question to ask after your troubleshooting is complete is to determine if it's a problem that can be fixed, or that can't be fixed.

This really depends on business priorities, budget constraints, and resource capacity. 

How long would it take to refactor or debug?

Is it significantly longer than the collective time that will be spent waiting for the tests to run? 

If not, is there an educational benefit?

Has the irritation from customers from slow loading or inaccessible software the potential to become a reputational/revenue-hitting issue?

Past a certain point of optimisation, I think the team as a whole would have to find a point where you all agree to move on. Or to put it a better way, set a goal for how testing in the organisation should work, and aim to reach that goal. 

The idea is that the tests as a whole run fast enough to help meet business requirements and achieve a satisfying user experience.

### When slow tests are a problem we can fix

Liquibase has a great article on <a href="https://www.liquibase.com/blog/making-automated-tests-faster">how to make automated tests faster</a>, where it's the tests themselves that are an issue.

For everything else, you'd need to talk to the development team about adding it onto their agenda. As a baby test automation engineer, I am assuming any changes would either factor into refactoring time, or bug-fixing time, depending on the severity of the issue.