# Test Frameworks

## Summary


Engineers write test cases to ensure that software works as expected. Often test cases require common services. These services are also agnostic of each test. The purpose of test frameworks is to provide a common platform for writing, organizing and executing test cases.

A test framework is a set of guidelines or rules that enable more efficient testing. To say it differently, a test framework provides a consistent interface between your code and your tests. Test frameworks basically provide the scaffolding. Test engineers can therefore focus on writing tests and testing the core functionality of their software.

The term **Test Automation Framework** is commonly used, though test frameworks can in rare cases be used to aid manual testing.

## Discussion

### What are some benefits of using a test framework?

Some benefits include improved test efficiency, lower maintenance costs, minimal manual intervention, maximum test coverage, code reuse, handling of recovery scenarios, and easy reporting. 

Obviously, test frameworks play an essential role in test automation, and test automation increases the depth and scope of testing. Lengthy or complex tests are hard to run manually. Developers can also focus on creating more tests than spend time executing tests. Essentially, better testing leads to higher quality software in a short time to market. 

It's recommended that you select a framework already available in the market rather than write your own custom framework. By doing this, you can focus on writing effective test cases and test suites.


### What's the typical execution process of a test framework?

Here we describe the typical process. Further details will vary depending on features that frameworks support.

The process starts by gathering all tests for execution. These tests are commonly grouped within **test suites** that are manually specified or could be discovered by convention. For example, all method/function names starting with a certain prefix are treated executable tests. 

Test executions start and end with **setup** and **teardown** code respectively. This setup/teardown code could be at test suite level, module level, test case level, or all of these. 

Within each test, one or more **assertions** are invoked. These compare current runtime status with expected status. If as expected, test execution continues. If not as expected, test fails. Stack trace may be recorded for later analysis. Extra logs may be taken. In all cases, final verdict from test execution is recorded. 

The final stage is **reporting**. Reports in preferred formats will contain summary, test verdicts, test-level execution metrics, and access to logs or traces.


### What features should I look for when choosing a test framework?

As in any tool, you should look at industry adoption, cost, performance, stability, scalability, (community) support, documentation, and features. Here are some features expected of a good framework:

    + **Agnostic**: Agnostic of System Under Test (SUT), automation libraries, platform or OS.
    + **Language**: What scripting languages are supported?
    + **Usage**: Easy to install, configure and use.
    + **Integration**: Integrates well with automation tools, Continuous Integration (CI) tools, issue tracking tools, etc.
    + **Reporting**: Generates detailed reports and analytics on test executions. Captures failure logs. Collects execution metrics.
    + **Customization**: Extend the framework. Group, tag, order or skip tests. Configure test environment.
    + **Modular**: Clear separation of framework layer and automation test suite. Automation infrastructure should follow coding conventions and design patterns. Promotes reuse of modules or libraries.
    + **Data**: Test scripts are separated from the data. Support for parameterized testing.
    + **Discovery**: Discover tests in project paths and automatically compose test suites for execution.
    + **Playback**: Record steps of manual tests and play them back for automated testing.

### What are some types of test frameworks?

Test frameworks differ in their approach to testing, which are briefly described as follows:  

    + **Linear**: Tests are independent of one another, often created by recording manual actions and replaying them.
    + **Module Based**: The application is tested in its smaller parts (functions, modules, sub-systems). Tests can be combined to enable system-level tests as well.
    + **Library Based**: Reusable libraries are identified and written. Good for test maintenance and scalability.
    + **Data Driven**: Focus is on executing a test with different input data. Data is read from databases or spreadsheets. There's a clear separation of test scripts and input data. Hard-coding of data within scripts is avoided.
    + **Keyword Driven**: An extension of data-driven approach. Test execution is basically a sequence of steps defined by keywords. Keywords are mapped to objects and actions in a *shared object repository*. Promotes reuse.
    + **Behaviour Driven**: Tests are written in a descriptive style so that even business stakeholders can understand them. Tests are in terms of behaviours and user-focused scenarios.
    + **Hybrid**: A mix of the above.

### Could you name some test frameworks out there?

A blog post on DZone (March 2018) identified ten open source test frameworks: Selenium, Carina, EarlGrey, Cucumber, Watir, Appium, RobotFramework, Apache JMeter, Gauge and Robotium. Selenium is mainly for web apps while Appium is for mobile apps. Carina can use the automation of Selenium or Appium. Watir is based on Ruby while RobotFramework is based on Python. Watir supports data-driven testing while RobotFramework uses keyword-driven approach for acceptance testing. 

Cucumber is a framework for Behaviour-Driven Development (BDD). Tests are written in **Gherkin** language and the focus is on acceptance testing. 

Another list on TestBeacon includes Serenity, Cypress, RobotFramework, RedwoodHQ, Sahi, Galen, Gauge, Citrus, and Karate-DSL. Serenity uses Selenium WebDriver and BDD tools such as Cucumber and JBehave. Cypress enabled Test-Driven Development (TDD). Galen is useful for testing layouts and user experience. Gauge might suit TDD and BDD practitioners. Citrus is suitable for testing network calls such as REST or SOAP. 

Specific to Android testing are Appium, Robotium, Selendroid, MonkeyTalk, Calabash, UI Automator, Magneto, Kobiton, Squish, SeeTest, KMAX, MonkeyRunner, Frank, KIF, and Testdroid. 

Wikipedia has a long list of test frameworks.


### How is a test automation tool different from a test framework?

There's a difference between a test framework and a test automation tool. An automation tool provides the implementation of testing actions. For example, Selenium WebDriver implements actions that mimic user actions on user interfaces. Thus, Selenium WebDriver is an automation tool.

However, Selenium WebDriver can be called from within any framework that supports it. For example, *pytest* is a test framework in Python, and since Selenium has Python support, we can call Selenium WebDriver APIs from Python scripts that are managed by *pytest*. Tools such as Selenium IDE is a simple framework that has the automation part built into it.


### How is a test management tool different from a test framework?

A test management tool is more about managing the testing process. This includes interfacing to various tools and processes that manage requirements, plan product releases, do reporting, track defects, execute tests, etc. There are plenty of such tools in the market. A few of them include qTest, PractiTest, Zephyr, Test Collab, and TestFLO for JIRA. 

A test framework is more about writing, managing and executing the tests. They are also about test automation. While frameworks may integrate with other tools and processes, their focus is more on testing rather than the bigger picture.

## Milestones

1989

At WyattSoftware, Gary Goldberg manages his test cases in spreadsheets. This is for a financial software written in Smalltalk. His colleague Ward Cunningham writes a framework to read the tests and execute them. This is probably one of the earliest test frameworks.

1994

Kent Beck writes **SUnit** for unit testing of Smalltalk code. It has only three classes (TestCase, TestSuite, TestResult) and twelve methods. 

1997

Inspired by SUnit, Kent Beck and Erich Gamma create **JUnit** for Java unit testing. 

2004

At ThoughtWorks, Jason Huggins creates what's now called **Selenium 1 Core** to automate web app testing. It uses scripts encoded in HTML tables. Selenium is today managed by the Software Freedom Conservancy. 

2012

At the Selenium Conference 2012, Dan Cuellar gives a lightning talk about automating iOS app testing using Selenium-style syntax written in C#. Later that year, he open sources the code and adds supports for Python. Jason Huggins adds Selenium WebDriver support. At the Mobile Testing Summit in November, it's launched as **Appium**. In 2016, Appium is handed over to JS Foundation.