Anti-patterns of automated software testing
Tags: goodpractices, testing, antipattern
Published: 2020-06-05 22:14:09

---
title: Anti-patterns of automated software testing
published: true
date: 2020-04-16 17:08:30 UTC
tags: goodpractices,testing,antipattern
canonical_url: https://medium.com/swlh/anti-patterns-of-automated-software-testing-b396283a4cb6
---

#### Sometimes it’s easier to learn from ‘_what not to do’_ rather than from ‘_what to do’_. Anti-patter_ns represent c_ommon bad practices, [code smells](https://en.wikipedia.org/wiki/Code_smell) and pitfalls, in this case, when creating automated tests. You should learn them so that you can spot them.

![](https://cdn-images-1.medium.com/max/1024/1*lgpPDW08K5GQnD1IW12MiQ.jpeg)

> Creating patterns to describe something creates nomenclature. Once you have a name for something, it’s easier to recognize when you see it again.  
> [The Productive Programmer, Neal Ford](https://www.goodreads.com/book/show/3411606-the-productive-programmer)

📝 _The list applies regardless of the language/runtime._

### No clear structure within the test

Not following an expectable structure in each test makes it hard to maintain it. **Tests are living code documentation** , so everyone should be able to quickly read/change them.

[Arrange, Act, Assert](http://wiki.c2.com/?ArrangeActAssert) is a pattern for structuring code in test methods. Basically, you **organize your code in phases for the sake of clarity**. It can be mapped to _Setup, Exercise, Verify, Teardown_ in the [four-phase test](https://thoughtbot.com/blog/four-phase-test) pattern and to [_Given, When, Then_](https://www.agilealliance.org/glossary/gwt/) in [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development):

1. **Arrange** /_Given_/_Setup_: initialization of the SUT ([_system under test_](https://en.wikipedia.org/wiki/System_under_test)_,_ whatever you’re testing) and other dependencies like mocks and fakes;
2. **Act** /_When_/_Exercise_: the call to the method being tested — ideally a single-liner;
3. **Assert** /_Then_/_Verify_: assertions of the output or verifications of calls made;
4. (Optional) Teardown: if there are resources to be freed.

![](https://cdn-images-1.medium.com/max/1024/1*RGFt7NBhOLdnnxKJX24jtw.png)

As a consequence, I usually **have 3 phases visually separated by empty lines**. If you reply with: “But I have lots of code in my test! I need more than 2 empty lines!”, I’d say “that’s likely a design smell”: you might be testing a [_god object_](https://en.wikipedia.org/wiki/God_object) which you need to split, you might need to create a “local [DSL](https://en.wikipedia.org/wiki/Domain-specific_language)” — high-level helper methods, or you could consider extracting the common setup and teardown elsewhere.

{% gist https://gist.github.com/lsoaresvw/811dd9acbfa3ca2c3bf57c02370a41bf %}

#### TL;DR

👎 _Having no consistent test structure, harming readability;_  
👍 _Applying Arrange/Act/Assert consistently, visually separated by newlines._

### Refactoring when adding a feature

Adding a new feature bears some risk and stress, even when you do TDD. So, why would you make things worse by fixing _this_ or improving _that_ simultaneously? This can lead to an explosion of untested cases and hard-to-track down bugs, but the worse is the anxiety it brings.

![](https://cdn-images-1.medium.com/max/220/0*ANqaoJ-2KuuqQ6ul)

So the bottom line is: **relax and focus on the feature at hand**. It’s true that you should leave the code in a better state than you found it — [_opportunistic refactoring_](https://martinfowler.com/bliki/OpportunisticRefactoring.html)), but please do it **before** (if it helps the feature) or **after** the feature. If it’s after, consider writing the refactoring ideas in a sheet of paper, as TODOs in the code, in tasks/stories in your tracker, to reduce the “fear of forgetting”.

#### TL;DR

👎 _Improving things you step into, while you create a feature;_  
👍 _Doing the refactor before the feature or noting it down for later._

### Changing implementation to make tests possible

I recently read [_Test-Driven Development by Example by Kent Beck_](https://www.goodreads.com/book/show/387190.Test_Driven_Development) and I was a bit disappointed that he was teaching why and how to test private methods (using Java reflection). Later, I think he made up his mind:

{% tweet 1258954349314625538 %}

I never faced a situation where I needed to test private methods; it always feels like a big red flag.

In “_unit test_”, the word unit represents a piece of a public interface. Private methods are an implementation detail — auxiliaries of public methods — and are to be tested indirectly through them! There’s probably no exception to this rule. If you find it hard to test all private method possibilities, then maybe you have a _god object_ that you should split into multiple components.

“Ah.. but I can make the method public!”. That’s even worse… **you should not change the code implementation just for making testing possible**. Testing private methods is just an example, but there’s much more. This usually means you’re doing something wrong.

#### TL;DR

👎 _Changing the implementation solely for the sake of making testing possible;_  
👍 _Be hinted to redesign the implementation code because the tests are hard to write._

### The test is hard to write

A common reason for a _hard-to-write-test_ is that you’re new to the problem and need to get more familiar with it. In that case, try explaining the problem to someone (or even [to a duck](https://en.wikipedia.org/wiki/Rubber_duck_debugging)) or take a break to get your ideas organized.

If you’re using TDD, you can try the _starter test_ (a really basic one) or the _explanatory test_ (the one you’d use to explain to someone).

![](https://cdn-images-1.medium.com/max/1024/0*Bve2haCED7kVOFW8)

However, if the test ends up being too complicated, too long (e.g. lots of setup code), or it feels like a hack, probably there’s a better way. A good way of finding if your code is too complicated is the need to add comments to explain it: **code should speak for itself**.

Generally speaking, **tests that are hard to write or understand are a symptom of bad code design.** Tests are a client of the implementation code — when tests of a component are hard to write, you can extrapolate how hard is to use that component.

> If it’s hard to write a test, it’s a signal that you have a design problem, not a testing problem. Loosely coupled, highly cohesive code is easy to test.  
> [Extreme Programming Explained, Kent Beck](https://www.goodreads.com/book/show/67833.Extreme_Programming_Explained)

In that case, you should consider if the implementation code needs a refactor to make it easier to test and maintain. The solution may be _divide and conquer_, as your [SUT](https://en.wikipedia.org/wiki/System_under_test) may be a [_god object_](https://en.wikipedia.org/wiki/God_object). It may also be wise to try to use more adequate [_software design patterns_](https://en.wikipedia.org/wiki/Software_design_pattern).

#### TL;DR

👎 _Making hacks that need an explanation or having complicated setups, so that a scenario can be tested;_  
👍 _Considering a code redesign when a test is hard to write._

### Optimizing DRY

This is the most common mistake I encounter. People see a repeated value and they will immediately create a variable for it. That’s not an early optimization: it’s just making documentation (tests) harder to read.

**In tests, some repetition is bearable for the sake of clarity.** This is one of the few differences between test code and implementation code. **Test code** ≠ **implementation code. Tests should more** [**DAMP (Descriptive And Meaningful Phrases)**](https://testing.googleblog.com/2019/12/testing-on-toilet-tests-too-dry-make.html) **than** [**DRY**](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) **(Don’t Repeat Yourself).**

On a related note, I hate when I have to scroll up and down or open multiple files to understand a single test. A typical case is a folder full of sample data that is used everywhere. Sometimes it’s fine to leave repeated setup and teardown in all tests in all cases. **Tests should be self-evident**. Tests are documentation. Therefore, I’d prefer to read tests _top-down_, like in a user manual (imagine if you had to swap pages multiple times when reading your washing machine manual…). This is also a callout to **avoid using globals in tests** (i.e., class or file globals).

![](https://cdn-images-1.medium.com/max/1024/1*KoAhqPj_pJEh63J4Uve4Lw.png)<figcaption>A very hard to read test; the more dependency arrows, the harder to read</figcaption>

#### TL;DR

👎 _Pursuing_ [_DRY_](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)_ness by creating variables and functions in tests;_  
👍 _Starting with everything local and evident; break the rule only if it brings more value._

### Very similar test cases

_Copy-pasting_ of more than one line of code should generally raise an alarm in your head. Well… it’s OK to do it just to make it work, but you should fix it later. It’s worse if you copy an entire test where only the initial values vary. This calls out loud to resort to _table-driven testing,_ also known as [_data-driven testing_](https://en.wikipedia.org/wiki/Data-driven_testing). **Table-driven testing allows running the same test multiple times, one per set of inputs and expected output** (or other consequences).

```
   Input | Output    
   -----------------
       1 | 1         
       2 | 2         
       3 | Fizz      
       5 | Buzz      
      15 | Fizz Buzz
```

📝 _It’s not my goal to teach you here about it, so I’ll leave a few links:_

- [_JUnit parameterized tests_](https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests) _(Java, Kotlin);_
- [_NUnit parameterized tests_](https://github.com/nunit/docs/wiki/Parameterized-Tests)_ (C#);_
- [_Jest each method_](https://jestjs.io/docs/en/api#testeachtablename-fn-timeout) _(JavaScript, TypeScript);_
- [_RSpec shared\_examples_](https://relishapp.com/rspec/rspec-core/docs/example-groups/shared-examples) _(Ruby);_
- [_Table-driven testing in Go, Python, and PHP_](https://medium.com/@jmdalmeida/table-driven-testing-b07ea4be21f4)_._

Bottom line: **use table-driven testing if you keep duplicating the same test with different data**. As advantages, you’ll have:

- Less code because you don’t repeat very similar tests;
- Easy comparison of test cases;
- Quick ability to add or remove test cases;
- Better documentation, because positive, negative, and edge cases are explicit.

⚠️ _In table-driven testing,_ **_do not mix the happy path with the error scenarios_** _. These should be kept apart for clarity reasons._

Finally, consider the trade-off you’re making, given that the tests become less [DAMP](https://stackoverflow.com/questions/6453235/what-does-damp-not-dry-mean-when-talking-about-unit-tests) when applying this technique.

#### TL;DR

👎 _Copy-pasting many similar chunks of test code to add new test cases;_  
👍 _Learning how to do table-driven testing with your language/library._

### Testing everything

According to the [law of diminishing returns](https://en.wikipedia.org/wiki/Diminishing_returns), there’s a point where the cost to add tests does not compensate for its benefits anymore.

![](https://cdn-images-1.medium.com/max/301/1*9gw9caDNRCWuXo-gGSMGpQ.png)

Applied to automated testing, **not every possibility needs to be tested**. Some situations can be left untested:

- some are too rare or exotic;
- others will quickly arise when using the software (e.g. missing env. var.)
- many won’t have a big impact if they fail (e.g. CSS styling, animations).

How to determine if a certain test is worth it? You should roughly calculate its [benefit-cost ratio](https://en.wikipedia.org/wiki/Benefit%E2%80%93cost_ratio): is **_benefit/cost_** _≥ 1_? Within _benefit_ consider the permanent prevention against the problem ([_testing as a safety net_](https://www.sasworkshops.com/unit-testing-as-a-safety-net/)). For the _cost_, take into account the effort to simulate the test scenario and the resources (lines of code to maintain, time to run) it will require from thereon.

#### TL;DR

👎 _Religiously trying to test everything;_  
👍 _Analyzing the cost-ratio test by test, to decide if it’s worth it._

### Other smells and anti-patterns

- **Using the word “and” when describing a test** : this hints that you should split the test in two, each asserting/verifying different things (check code example above);
- **Testing lots of combinations** ([_combinatorial explosion_](https://www.freecodecamp.org/news/combinatorics-handle-with-care-ed808b48e5dd/)): this is a smell that you have multiple components into one; the solution is to recognize them and slowly split them apart.
- **Treating test code as a second class citizen** : test code is your safety net so take good care of it; most good code practices are transversal to any source code (e.g. avoid globals);
- **Not discarding a** [**spike**](https://en.wikipedia.org/wiki/Spike_(software_development)) **’s code, especially when doing TDD** : a spike is a learning tool; its code should not be kept and evolve;
- **Testing business rules though UI** : business rules should be tested regardless of the UI being used. Of course, end-to-end testing is an exception;
- **Quantity without diversity** : test cases should be different somehow — do not add cases for the sake of quantity; the same applies to test scenarios: they should vary;
- [**Mocking what you don’t own**](https://github.com/testdouble/contributing-tests/wiki/Don't-mock-what-you-don't-own): avoid leakage of libraries into your software (runtime or external); there are alternatives like wrappers and mock servers;
- **Conditionals in tests** : having _“ifs”_ in tests is wrong; it’s proof that you need another test scenario;
- **Overvaluing test coverage** : coverage shouldn’t be something you pursue just for the sake of it because you may be near 100% and still have crappy tests. Anyway, with TDD, coverage is almost an irrelevant concept.
- **Early optimization** : in TDD, always reduce the “distance” between a red (now) and a green state (future). Do the “_quick and dirty_” implementation even if it means to _return a constant_ or do an ugly or slow implementation. Make it pass, then make it beautiful. The reason is not only psychological — it’s dopamine-freeing to see a green state — but it’s also about learning sooner. You learn, from the start, more about the problem and the solution (e.g. code design) if you make it pass first.

### Final remarks

One could notice that a lot of anti-patterns in tests can be attributed to an anti-pattern in implementation: [the _god object_](https://en.wikipedia.org/wiki/God_object)_:_ having huge test files, having long tests, a test being hard to write, among others are warning signs. That said, my advice is:

- **Write software in a _top-down_ approach** , so that you can more easily understand the “components below” deriving from the “ones above” (and you only implement what’s needed);
- **Recognize when you have multiple reasons to change** a component so you can split it ([_single-responsibility principle_](https://en.wikipedia.org/wiki/Single-responsibility_principle)).

Bottom line: if it’s complicated, you’re doing it wrong. **The ideal solution is usually simple; you just need to be persevering and iterate until you achieve it.**

The presented list was not supposed to be exhaustive since there are good resources out there (e.g. [StackOverflow’s thread](https://stackoverflow.com/questions/333682/unit-testing-anti-patterns-catalogue), [Yegor Bugayenko’s list](https://www.yegor256.com/2018/12/11/unit-testing-anti-patterns.html)). [_Software Testing Anti-patterns_ by Kostis Kapelonis](http://blog.codepipes.com/testing/software-testing-antipatterns.html) is a must-read. You can find good patterns in Part III of [_Test-Driven Development by Example_ by Kent Beck](https://www.goodreads.com/book/show/387190.Test_Driven_Development). Finally, watch [TDD, Where Did It All Go Wrong by Ian Cooper](https://www.youtube.com/watch?v=EZ05e7EMOLM):

{% youtube EZ05e7EMOLM %}

* * *