Different kinds of testing
Tags: testing, typology, unittesting, integrationtesting
Published: 2020-08-02 17:46:23

The subject of testing is vast. It may seem simple from outside, but it's not. For example, one may define testing as checking that the software is fit for its purpose. But it encompasses a lot more: for example, mutation testing verifies that assertions do actually assert. In this post, I'd like to touch some testing flavors, what's their purpose and how they compare to each other.

## The need for testing

In an ideal world, we wouldn't need testing. We would just write bug-free performant code that would be the perfect direct implementation of the requirements.

Yet, the process of transforming requirements into code needs to be checked. Software development involves requirements gathering, writing documents, communicating with people, coping with organizational issues, etc. For the sake of simplification, in the rest of this post, we will consider that this process is perfect - even though it's not, and focus on code.

Let's mention the elephant in the room: some platforms _e.g._ [Coq](https://coq.inria.fr/) do allow **formal proof**. This means it can be mathematically proven that the code is bug-free. However, most of those platforms stay within the bounds of academia. As far as I know, there aren't that many attempts at using [formal verification](https://en.wikipedia.org/wiki/Formal_verification) in the industry.

By removing formal proof from the equation (pun intended), the only option left is testing. But **testing doesn't prove the absence of bugs**. In fact, testing is not only about bugs: it covers a lot of different areas. Let's look at some of them.

## Unit Testing

_Unit Testing_ is a fairly well-documented discipline: regardless of the language, a ton of books have been published on the subject. They generally repeat the same things.

>[...] Unit testing is a software testing method by which individual units of source code [...] are tested to determine whether they are fit for use.
>
>  [--Wikipedia](https://en.wikipedia.org/wiki/Unit_testing)

The only strongly-debated point is what constitutes a _unit_: in <abbr title="Object-Oriented Programming">OOP</abbr>, some argue it's the _class_; others argue that it's a _module_ _i.e._ a set of collaborating classes.

## Integration Testing

While _Unit Testing_ is fairly well defined and understood, _Integration Testing_ seems like virgin territory in comparison. I actually wrote the book [Integration Testing from the Trenches](https://blog.frankel.ch/books/integration-testing-trenches/) because I found no other books on the subject at that time.

> Integration testing [...] is the phase in software testing in which individual software modules are combined and tested as a group.
>
>  [-- Wikipedia](https://en.wikipedia.org/wiki/Integration_testing)

One of the core concepts behind _Integration Testing_ is the _System Under Test_. In _Unit Testing_, the _<abbr title="System Under Test">SUT</abbr>_ is the unit (_i.e._ the class or the module, as described above). In _Integration Testing_, the SUT needs to be defined per test: it can be as small as two units collaborating, and as big as the whole system.

There's a raging debate about _unit testing_ vs. _integration testing_. Some consider only the former, while other only the latter. If you want to know my personal stance on this, please read [this previous post](https://blog.frankel.ch/unit-test-vs-integration-test/).

## End-To-End Testing

At a time when most applications were Human-To-Machine, _End-To-End Testing_ meant testing the flow from the _User Interface_ to the database, and back again. Because web applications have become nearly ubiquitous, this involves the browser. For this reason, _<abbr title="End-to-End">E2E</abbr> Testing_ is very different from the previous approach. Automating user interactions with the browser is not trivial, even if the tools available improved with time.

The biggest issue in _E2E Testing_ comes from the brittleness of the <abbr title="User Interface">UI</abbr> layer. Modern architectures cleanly separate into a JavaScript-based front-end and a <abbr title="REpresentational State Transfer">REST</abbr> back-end. To cope with the mentioned brittleness, it helps to first test the REST layer via _Integration Testing_.

With _E2E Testing_, there's a whole continuum from _Unit Testing_ through _Integration Testing_. This is pretty well summarized in the famous _Testing Pyramid_.

<a href="https://martinfowler.com/articles/practical-test-pyramid.html"><img src="https://martinfowler.com/articles/practical-test-pyramid/testPyramid.png" alt="The Test Pyramid" /></a>

One can consider _E2E Testing_ as the most complete form of _Integration Testing_: in that case, the _SUT_ is the whole application.

## System Testing

With time passing, more and more applications are targeted at Machine-To-Machine interactions instead of <abbr title="Human-To-Machine">H2M</abbr> ones. _End-To-End Testing_ is to H2M what _System Testing_ is to <abbr title="Human-To-Machine">M2M</abbr>.

The benefit is that there's no browser involved. Hence, the same automation techniques from _Unit Testing_ and _Integration Testing_ can be reused.

## Performance Testing

Most of the previous approachs focus on testing _Functional Requirements_. It's easy to forget that the fitness of a software component encompasses both _Functional Requirements_ and _Non-Functional Requirements_. Performance counts as part of the <abbr title="Non-Functional Requirements">NFR</abbr>, along with reliability, etc.

Note that the _SUT_ in performance tests can be the whole system, or just a sub-component deemed critical. Yet, in that case, asserting the performance of each individual components might not be enough to guarantee the performance of the whole system.

> [...] performance testing is in general a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload.
>
>  [-- Wikipedia](https://en.wikipedia.org/wiki/Software_performance_testing)

From my experience, the term _Performance Testing_ is overused, and much too general. It has many facets: here are the most commons ones.

### Load Testing

_Load Testing_ is in general what is implicitly meant when people talk about _Performance Testing_. In this context, one sets different parameters for the tests. Those parameters model a representative load from the production environment. In reality, most organizations are unable to provide a quality sample of the production load, and the load is inferred.

For example, in an e-commerce context, testing parameters would include:

* The different types of requests: browsing the catalog, viewing distinct products, adding them to the cart, checking out, paying, etc.
* The repartition between them _e.g._ 70% browsing, 20% viewing, 6% cart manipulation, 3% checkout, and 1% payment
* The number of concurrent requests
* The number of concurrent user sessions, and how are they spread between requests
* etc.

### Endurance Testing

In regular _Load Testing_, tests run until they are finished executing. The goal is to check that the system behaves according to one's expectations under a specific load.

In _Endurance Testing_ also known as _Soak Testing_, the goal is to check how the system's behavior evolves along with time. In comparison, the duration of those tests are more in the order of days, or even weeks.

> Soak testing involves testing a system with a typical production load, over a continuous availability period, to validate system behavior under production use.
>
>  [-- Wikipedia](https://en.wikipedia.org/wiki/Soak_testing)

### Stress Testing

_Stress Testing_ is the testing of the system until it breaks. The load is increased gradually. The goal of _Stress Testing_ is to understand the performance limits of the system.

> Stress testing (sometimes called torture testing) is a form of deliberately intense or thorough testing used to determine the stability of a given system, critical infrastructure or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results.
>
>  [-- Wikipedia](https://en.wikipedia.org/wiki/Stress_testing)

Conclusions of such experiments are along the lines of: within a load of `x` requests/second, a `y` fraction of requests return a `5xx` error code.

## Chaos Testing

_Chaos Testing_, also known as Chaos Engineering, is the practice of randomly removing components of a system to understand how it breaks, and check its overall resiliency under duress.

> Chaos engineering is the discipline of experimenting on a software system in production in order to build confidence in the system's capability to withstand turbulent and unexpected conditions.
>
> [-- Wikipedia](https://en.wikipedia.org/wiki/Chaos_engineering)

The defining property of _Chaos Testing_ is that it's generally **performed in the production environment**. That doesn't mean that experiments made in other environments cannot bring actionable feedback, far from the contrary. However, everybody has experienced issues in production that never happened in previous environments. The root cause might be slightly different architectures, different data data, higher load, etc: in the end, the only environment that needs to be battle-tested and bulletproof is production.

While counter-intuitive at first, the idea is now firmly entrenched in mature organizations that are serious about the resilience of their systems.

## Security Testing

Just like _Performance Testing_ is too broad a term, _Security Testing_ covers many different areas. The most well-known form of _Security Testing_ is _Penetration Testing_.

> A penetration test [...] is an authorized simulated cyberattack on a computer system, performed to evaluate the security of the system.
> [...]
> The test is performed to identify both weaknesses [...], including the potential for unauthorized parties to gain access to the system's features and data, as well as strengths, enabling a full risk assessment to be completed.
> 
>  [-- Wikipedia](https://en.wikipedia.org/wiki/Penetration_test)

## Mutation Testing

Most forms of testing focus on the fitness of pieces of software, or the system as a whole. _Mutation Testing_ is unique in that its goal is to give a hindsight on unit tests, and more specifically on the misleading _Code Coverage_ metrics.

> Mutation testing [...] is used to design new software tests and evaluate the quality of existing software tests.
>
>  [-- Wikipedia](https://en.wikipedia.org/wiki/Mutation_testing)

I've already [written](https://blog.frankel.ch/introduction-to-mutation-testing/) on the subject before. Please read it if you want to know more.

## Exploratory Testing

A lot of testing approaches focus on _regression bugs_, bugs that appear in previously working software. For that, tests are automated: they are ran for every build. If a previously running test fails, the build fails as well.

The goal of _Exploratory Testing_ is to catch bugs that are out of the testing harness's reach. To achieve that, one interacts with the system in new and unexpected ways.

> Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution.
>
>  [-- Wikipedia](https://en.wikipedia.org/wiki/Exploratory_testing)

Because of its exploratory nature, this process is manual. My experience has shown me that _Exploratory Testing_ is undervalued. That's a pity, because I've witnessed testers who uncovered issues in the system that previous automated steps didn't find.

Despite its manual nature, _Exploratory Testing_ is not about clicking everywhere. It's an engineering practice, that require discipline and a rigorous approach to documenting the steps that led to the issue. The best testers are also able:

* To isolate the issue
* To reproduce the steps that led to it
* To create the issue in a way that will ease the job of developpers who will need to fix the bug

## User Acceptance Testing

Most forms of testing are performed by technical people: developers, automation testers, etc. On the opposite, _Acceptance Testing_ is performed by non-technical people, _e.g._ business analysts or end users. As the name implies, the goal is to assess whether they accept the software as fit for their usage.

> User acceptance testing consists of a process of verifying that a solution works for the user.
>
>  [-- Wikipedia](https://en.wikipedia.org/wiki/Acceptance_testing#User_acceptance_testing)

## Summary

| Kind        | Goal                                                      | Actor                                                                             | Automated |
|-------------|-----------------------------------------------------------|-----------------------------------------------------------------------------------|-----------|
| Unit        | Assess the fitness of a code unit                         | Developer                                                                         | ✅         |
| Integration | Check the collaboration of units                          | Developer                                                                         | ✅         |
| System      | Verify the behavior of the system as a whole              | Mix of different skillsets: backend developers, system administrators, DBAs, etc. | ❌         |
| End-to-End  | Validate the flow from the  GUI to the datastore and back | Front-end developer; Specialized automation tester                                | ✅         |
| Performance | Check different performance-related metrics               | System administrator with specialized testing skills                              | ✅ ❌    |
| Chaos       | Verify the resiliency of a system                         | Mix of different skillsets: sysadmins, network experts, etc.                      | ✅         |
| Penetration | Assess the vulnerability of a system                      | "Hacker"                                                                          | ❌         |
| Mutation    | Check the quality of tests                                | Developer                                                                         | ✅         |
| Exploratory | Find hidden bugs                                          | Manual tester                                                                     | ❌         |
| UAT         | Validate the system conforms to specifications            | Business people                                                                   | ❌         |

## Conclusion

The post is already too long, and I only brushed the surface of available testing kinds. In conclusion, one need to remember a couple of things:

1. It's easier to find a bug in the system than to point to its root cause. The bigger the SUT, the harder it is.
2. The right balance between manual testing and automated testing efforts needs to be found: this balance is context-dependent.
3. Testing is all about <abbr title="Return Over Investement">ROI</abbr>. This is plain risk management: the goal is to evaluate risks, their nature, their probability, their impact, the respective mitigation actions, etc. in regard to the associated costs.
4. Finally, testing is a strategic move: one should carefully plan one's testing efforts and the reasons behind them before writing a single one (and yes, it's against <abbr title="Test-Driven Development">TDD</abbr>).

Happy testing!

**To go further:**

* [Why are you testing your software?](https://blog.frankel.ch/why-are-you-testing-software/)
* [Starting with Cucumber for end-to-end testing](https://blog.frankel.ch/starting-with-cucumber-for-end-to-end-testing/)
* [Unit tests vs integration tests, why the opposition?](https://blog.frankel.ch/unit-test-vs-integration-test/)
* [Software performance testing](https://en.wikipedia.org/wiki/Software_performance_testing)
* [Unit testing](https://en.wikipedia.org/wiki/Unit_testing)
* [Integration testing](https://en.wikipedia.org/wiki/Integration_testing)
* [Software testing](https://en.wikipedia.org/wiki/Software_testing)
* [Soak testing](https://en.wikipedia.org/wiki/Soak_testing)
* [Stress testing](https://en.wikipedia.org/wiki/Stress_testing)
* [Chaos engineering](https://en.wikipedia.org/wiki/Chaos_engineering)
* [Security testing](https://en.wikipedia.org/wiki/Security_testing)
* [Penetration test](https://en.wikipedia.org/wiki/Penetration_test)
* [Mutation testing](https://en.wikipedia.org/wiki/Mutation_testing)
* [Exploratory testing](https://en.wikipedia.org/wiki/Exploratory_testing)
* [Acceptance testing](https://en.wikipedia.org/wiki/Acceptance_testing)
* [Smoke testing](https://en.wikipedia.org/wiki/Smoke_testing_(software))

_Originally published at [A Java Geek](https://blog.frankel.ch/different-kinds-testing/) on August 2<sup>nd</sup>, 2020_