Regression testing is an expensive maintenance process used to revalidate modified software. Regression test selection (RTS) techniques try to lower the cost of regression testing by selecting and running a subset of the existing test cases. Many such techniques have been proposed and initial studies show that they can produce savings. We believe, however, that issues such as the frequency with which testing is done have a strong effect on the behavior of these techniques. Therefore, we conducted an experiment to assess the effects of test application frequency on the costs and benefits of regression test selection techniques. Our results expose essential tradeoffs that should be considered when using these techniques over a series of software releases. Keywords regression testing, application frequency, empirical study After modifying software, developers typically want to know that unmodified code has not been adversely affected. When such unmodified code is adversely affected, we say that a regression error has occurred. Developers often do regression testing to search for such regression errors. The simplest regression testing strategy is to rerun all existing test cases. This method is simple to implement, but can be unnecessarily expensive, especially when changes affect only a small part of the system. Consequently, an alternative approach, regression test selection (RTS) technique, has been proposed [e.g., 1,2,5,6,10,14]. With this approach only a subset of test cases are selected and rerun. Since, in general, optimal test selection (i.e., selecting exactly the fault-revealing test cases) is impossible, the cost-benefit tradeoffs of RTS techniques is a central concern of regression testing research and practice. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are !lot made or distributed for profit or comiiiercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise. to republish, to post on seiiws or to rcdistribute to lists, requires prior specific pemdssion andior a fee. Copyright ACM 2000 1-581 13-206-9/00/6 ... $5.00 A common way to study this problem has been to find or create base and modified versions of a system and accompanying test suites. Next, a test selection algorithm is run and the size and effectiveness of the selected test suite is compared to the size and effectiveness of the original test suite [e.g., 4,11,16,17]. One limitation of this approach is that it models regression testing as a one-time activity rather than as the continuous process it is. For example, software releases often require many changes to a system with regression testing interspersed between some number of changes (rather than one regression test at the end of a release). In practice, many companies integrate changes and then regression test on a monthly, weekly, or even daily basis. Our hypothesis is that the amount of change made between regression testing sessions strongly affects the costs and benefits of different regression test selection techniques. That is, we believe that some test selection techniques will perform less cost-effectively as the amount of changes made between regression testing sessions grows. This is because they will select increasingly larger test suites and because these suites will become increasingly less cost-effective at finding faults. If this hypothesis is true, testing practitioners may be able to better manage and coordinate their integration and regression testing processes, thereby saving time and money. Therefore we have designed and implemented an experiment to examine this hypothesis. In the remainder of this paper we review the relevant literature, describe our research hypotheses, present the design and analysis of our experiment and discuss our conclusions and future research directions. BACKGROUND AND LITERATURE REVIEW Let P be a procedure or program, let P' be a modified version of P and let T be a test suite for P.
The effects of this factor on regression test selection costs have been studied in REF .
An empirical study of regression test application frequency
Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium
Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium
Computer Science