Mutation Testing has been shown to be a powerful technique in detecting software faults. Despite this advantage, in practice there is a need to deal with the equivalent mutants' problem. Automatically detecting equivalent mutants is an undecidable problem. Therefore, identifying equivalent mutants is cumbersome since it requires manual analysis, resulting in unbearable testing cost. To overcome this difficulty, researchers suggested the use of mutant classification, an approach that aims at isolating equivalent mutants automatically. From this perspective, the present paper establishes and empirically assesses possible mutant classification strategies. A conducted study reveals that mutant classification isolates equivalent mutants effectively when low quality test suites are used. However, it turns out that as the test suites evolve, the benefit of this practice is reduced. Thus, mutant classification is only fruitful in improving test suites of low quality and only up to a certain limit. To this end, empirical results show that the proposed strategies provide a cost-effective solution when they consider a small number of live mutants, i.e., 10-12. At this point they kill 92% of all the killable mutants. while running the same test case. Otherwise, it is called live. In terms of fault revealing power, the higher the number of killed mutants, the higher is the test case power. However, while some mutants are killable, some mutants cannot be killed. In that case, such mutants are qualified as equivalent ones. As a result, an equivalent mutant forms a program version functionally equivalent to the original program, i.e., there exist no test case able to distinguish this mutant from the original program. An equivalent mutant plays the role of a parasite in the testing process. Indeed, while it is expected to be killable, it remains always live. Even worse, a tedious effort could be uselessly dedicated to improving tests with no hope of killing it (in a way similar to covering infeasible statements or branches w.r.t. code coverage criteria [16] ). As a consequence, mutation testing requires the removal of these mutants. However, discarding equivalent mutants tackles an even harder problem, since judging programs' functional equivalence is known to be undecidable [4] . As a result tedious manual analysis is required. It has been empirically found that it takes approximately 15 minutes [22] to identify an equivalent mutant in a real world application. Since a vast number of equivalent mutants exist, the application cost of mutation is escalated. As a consequence, the criterion used to stop the testing process requires the identification of most if not all of the existing equivalent mutants. To decrease the undesirable effects of equivalent mutants, heuristic methods appear to be promising. Schuler and Zeller [21, 22] suggested a method to automatically classify mutants into the likely killable and the likely equivalent. The underlying idea of this approach is to measure the effects, called the impact, introduced by mutants on the runtime program execution. It has been found that mutants with an impact are more likely to be killable than those with no impact. Although mutant classification has been suggested [22], it is not evident how it could be applied in practice. Furthermore, it is not evident what the practical benefits of its utilization are. The present paper defines strategies that take advantage of the mutant classification. We call these strategies as mutant classification strategies. These turn mutants' impact as a guide towards improving a test suite. The primary aim of the present work is to define and evaluate the relative effectiveness and efficiency of mutant classification strategies. In other words, we address the question of how the mutation testing process should be performed in order to take advantage of mutant classification. To this end, we seek to identify the practical benefits of these strategies compared to the "traditional" mutation testing approach, i.e., the usual way of applying mutation testing (without mutant classification); see Section 3.3 for a definition of the "traditional" and the mutant classification approaches. The above intentions were investigated based on a set of industrial programs written in C and using the Proteum [6] mutation testing system. It has been found, that the examined strategies are cost-effective in improving the quality of a test suite. This improvement can lead to mutation scores of 92%-95% by examining up to twenty mutants per program. However, reaching higher mutation scores is not fruitful with the examined strategies. Therefore, in these cases the traditional approach should be undertaken. Despite this weakness, employing these strategies can lead to substantial benefits with respect to the equivalent mutants' identification in non-critical cases. Thus, mutant classification can provide a costeffective solution to the equivalent mutants' problem only up to a certain limit. To this end, empirical results show that when examining only a small number of mutants, i.e., (10-12) of mutants, the proposed strategies are beneficial and they kill more than 92% of all the killable mutants. Additionally, the obtained results show that the methods' effectiveness loss is a straight consequence of the accuracy of the proposed classification schemes; therefore, pointing out possible directions for future research. The present paper extends the previous work on mutant classification strategies [24] in several dimensions. First, by considering a wider range of programs. Second, by studying and analyzing (manually) the equivalent mutants for all C-language mutant operators. Third, by studying the behavior of the strategies when random test cases are used as initial ones. Finally, a cost-effective analysis of the proposed strategies and a comparison with the traditional mutation approach is also performed. The remainder of this paper is organized as follows: Sections 2 and 3 present the concepts underlying the present work and the mutant classification strategies. Section 4 presents and details the experimental setup. The empirical results are presented and analyzed in Section 5. Section 6 discusses the findings of the present study and considers some threats to the validity of the conducted study. Finally Sections 7 and 8 respectively present the related work and the conclusions of the paper. Using mutation as a testing criterion [17] requires a way of measuring the adequacy of testing. Generally, test adequacy can be measured based on the exact mutation score, which is defined as follows: However, calculating the exact mutation score is hard due to the existence of the so-called equivalent mutants [16] . These mutants cannot be killed and hence, they must be somehow identified in order to compute the mutation score. To this end, practitioners have two choices: a) to manually analyze the live mutants, which in practice is very hard, if not practically infeasible and b) to approximate the exact mutation score. The later approach is based on the actual mutation score, which is defined as follows: Using Actual MS has two main drawbacks. First, it requires from the tester to decide which value of the score is satisfactory for completing the process. Without a manual analysis, this choice is more or less arbitrary, resulting in a degraded confidence in the testing process. Second, it does not give any guidance to the tester as to which mutants should be targeted first, to increase the score. To address these difficulties, the present paper examines the use of mutants' impact [22], [21] as a possible way to automatically identifying the killable mutants. It also specifies the order of mutants that should be targeted. In other words, mutants' impact allows deciding whether live mutants can be killed or not. Therefore, the tester can concentrate only on these (likely killable) mutants according to the provided order. Additional details regarding the concept of mutants' impact are given in the following section.
REF used manual analysis to study mutant classification strategies and found that such techniques are helpful only to partially improve test suites (of low quality).
Mitigating the effects of equivalent mutants with mutant classification strategies
Sci. Comput. Program.
Sci. Comput. Program.
Computer Science