# Software Performance Testing

## Summary


Performance testing helps identify bottlenecks in a system, establish a baseline for future testing, support a performance tuning effort, and determine compliance with performance goals and requirements.

Performance testing is an ongoing task throughout the life cycle of the project. It not only involves testers, but also developers and operations to run and maintain the application to its performance expectations. 

Performance, load, and stress tests are subcategories of performance testing, each intended for a different purpose.

## Discussion

### What is performance testing in software?

Performance testing is a type of non-functional testing. It's intended to determine the responsiveness, throughput, reliability, and/or scalability of a system under a given workload. 

Performance testing is testing that is performed to determine how fast some aspect of a system performs under a particular workload. It can serve different purposes, such as to demonstrate that the system meets performance criteria.

In the past, performance testing was a specialized activity managed by separate teams, not the developers. The process typically took weeks and even months. With the transition to Agile and DevOps, this has changed. Using open source and commercial tools, it's now become possible to do performance testing by developers in shorter time scales. 


### What should be the focus of performance testing?

Following are some focus areas for performance testing: 

    + **Speed**: Determines whether the software application responds quickly.
    + **Scalability**: Determines maximum user load the software application can handle.
    + **Stability**: Determines if the application is stable under varying loads.

### What are the common goals of performance testing?

Performance testing has the following goals:  

    + Assess production readiness
    + Evaluate against performance criteria
    + Compare performance characteristics of multiple systems or system configurations
    + Find the source of performance problems
    + Support system tuning
    + Find throughput levels

### What are the types of performance testing?

Each type of performance testing has a specific focus as noted below: 

    + **Load Testing**: Checks the application's ability to perform under anticipated user loads. The objective is to identify performance bottlenecks before the software application goes live.
    + **Stress Testing**: Also called *Breakpoint Testing*, this involves testing an application under extreme workloads to see how it handles high traffic or data processing. The objective is to identify breaking point of an application.
    + **Endurance Testing**: Also called *Soak Testing*, this is done to make sure the software can handle the expected load over a long period of time.
    + **Spike Testing**: Tests the software's reaction to sudden large spikes in the load generated by users.
    + **Volume Testing**: Large amount of data is populated in database and the overall software system's behavior is monitored. The objective is to check software application's performance under varying database volumes.
    + **Scalability Testing**: Determine the software application's effectiveness in "scaling up" to support an increase in user load. It helps plan capacity addition to your software system.

### What are the steps of performance testing?

We may list the following steps: 

    + **Identify the Test Environment**: Identify the physical test environment and the production environment, which includes the hardware, software, and network configurations.
    + **Identify Performance Acceptance Criteria**: Identify the response time, throughput, and resource utilization goals and constraints. In general, response time is a user concern, throughput is a business concern, and resource utilization is a system concern.
    + **Plan and Design Tests**: Identify key scenarios, determine variability among representative users such as unique login credentials and search terms.
    + **Configure the Test Environment**: Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for test.
    + **Implement the Test Design**: Develop the performance tests in accordance with the test design best practice.
    + **Execute the Test**: Run and monitor your tests. Validate the tests, test data, and results collection.
    + **Analyze Results, Report, and Retest**: Consolidate and share results data. Analyze the data both individually and as a cross-functional team. Re-prioritize the remaining tests and re-execute them as needed.

### What performance parameters are monitored during performance testing?

The following are commonly monitored: 

    + **Processor Usage**: Amount of time processor spends executing non-idle threads.
    + **Memory Use**: Amount of physical memory available to processes on a computer.
    + **Disk Time**: Amount of time disk is busy executing a read or write request.
    + **Bandwidth**: Shows the bits per second used by a network interface.
    + **Private Bytes**: Number of bytes a process has allocated that can't be shared amongst other processes.
    + **Memory Pages per Second**: Number of pages written to or read from the disk in order to resolve hard page faults.
    + **Page Faults per Second**: Overall rate in which fault pages are processed by the processor.
    + **Response Time**: Time from when a user enters a request until the first character of the response is received.
    + **Throughput**: Rate a computer or network receives requests per second.
    + **Maximum Active Sessions**: Maximum number of sessions that can be active at once.
    + **Hits per Second**: Number of hits on a web server during each second of a load test.
    + **Database Locks**: Locking of tables and databases.
    + **Thread Counts**: Number of threads that are running and currently active.

### Could you give pointers on performance testing in the cloud?

**Cloud-based performance testing** refers to the use of cloud technologies to test your application or service. Your application itself may not be on the cloud but the load generation, report collection and test monitoring is done in the cloud. The obvious benefit is lowering infrastructure cost, testing at scale, simulating load coming from different geographic regions, and more. 

This should not be confused with **cloud performance testing**, where the idea is to validate the performance of an application deployed in the cloud. The general approach to performance testing, types of tests and parameters monitored remain the same but with some aspects particular to cloud. For example, storage, processing, bandwidth and number of users become important. In particular, parameters related to management of virtual machines, containers, microservices, database connections, all need to be validated.

Tests should validate how well and fast can the cloud infrastructure adapt to varying load. Elasticity, scalability, availability, fault tolerance and reliability are all important measures. This includes the performance of logging, raising alarms, or sending notifications, across all the cloud services used by the application.


### Could you name a few performance testing tools?

Some open source tools include Apache JMeter, K6, LoadUI, and Grinder. Commercial tools requiring a license include HP LoadRunner, IBM Rational Performance Tester, and Microfocus SilkPerformer.

When selecting a particular tool, you must evaluate the supported features. One blogger has shared a feature comparison of some tools with respect to scripting, execution and reporting.

Wikipedia gives a useful list of load testing tools. 

## Milestones

1989

Mercury Interactive is founded in California. It releases **LoadRunner**, one of the first performance testing tools. 

Dec  
1998

**JMeter** is released as an open source functional plus performance testing tool. In November 2011, it becomes a top-level Apache project. 

2001

**OpenSTA** is released as an open source web load and performance testing tool. 

2004

Software Test & Performance magazine is launched by BZ Media. It's later renamed to Software Test & Quality Assurance magazine. A curated list of magazines and blogs for software testing is also available.

2011

Based on JMeter, commercial tool **BlazeMeter** is released. Fourth generation of BlazeMeter is released in 2018 and it enables Continuous Testing. BlazeMeter enhances JMeter with scalability, reporting, collaboration, and test management features.