Defects4J is a large, peer-reviewed, structured dataset of real-world Java bugs. Each bug in Defects4J comes with a test suite and at least one failing test case that triggers the bug. In this paper, we report on an experiment to explore the effectiveness of automatic test-suite based repair on Defects4J. The result of our experiment shows that the considered state-of-the-art repair methods can generate patches for 47 out of 224 bugs. However, those patches are only test-suite adequate, which means that they pass the test suite and may potentially be incorrect beyond the test-suite satisfaction correctness criterion. We have manually analyzed 84 different patches to assess their real correctness. In total, 9 real Java bugs can be correctly repaired with test-suite based repair. This analysis shows that testsuite based repair suffers from under-specified bugs, for which trivial or incorrect patches still pass the test suite. With respect to practical applicability, it takes on average 14.8 minutes to find a patch. The experiment was done on a scientific grid, totaling 17.6 days of Communicated by: Sunghun Kim Empir Software Eng (2017Eng ( ) 22:1936Eng ( -1964 RQ3. Which bugs in Defects4j are under-specified? A bug is said to be under-specified if there exists a trivial patch that simply removes functionality. For those bugs, current repair approaches fail to synthesize a correct patch due to the lack of test cases. Those bugs are the most challenging bugs: to automatically repair them, one needs to reason on the expected functionality below what is encoded in the test suite, to take into account a source of information other than the test suite execution. One outcome of our experiment is to identify those challenging bugs. RQ4. How long is the execution of each repair system? The answer to this question also contributes to assess the practical applicability of automatic repair in the field. Our experiment considers 224 bugs that are spread over 231K lines of code and 12K test cases in total. We ran the experiment for over 17.6 days of computational time on Grid'5000 Bolze et al. (2006), a large-scale grid for scientific experiments. Our contributions are as follows: Answer to RQ1. The Defects4J dataset contains bugs that can be automatically repaired with at least one of the considered systems. jGenProg, jKali, and Nopol together synthesize test-suite adequate patches for 47 out of 224 bugs with 84 different patches. Some bugs are repaired by all three considered repair approaches (12/47). These results validate that automatic repair works on real bugs in large Java programs 1 . They can be viewed as a baseline for future usage of Defects4J in automatic repair research. Answer to RQ2. Our manual analysis of all 84 generated patches shows that 11 /84 are correct, 61 /84 are incorrect, and 12 /84 require a domain expertise, which we do not have. The incorrect patches tend to overfit the test cases. This is a novel piece of empirical evidence on real bugs that either the current test suites are too weak or the current automatic repair techniques are too dumb.These results strengthen the findings in Qi et al. (2015), on another benchmark, on another programming language. They show that the overfitting problem uncovered on the Genprog's benchmark by Qi et al. (2015) is not specific to the benchmark and its weakness, but is more fundamental. Answer to RQ3. Defects4J contains very weakly specified bugs. Correctly repairing those bugs can be considered as the next milestone for the field. This result calls for radically new automatic repair approach that reasons beyond the test suite execution, using other sources of information, and for test suite generation techniques tailored for repair. Answers to RQ4. The process of searching for a patch is a matter of minutes for a single bug. This is an encouraging piece of evidence that this research will have an impact for practitioners.
For Java, Martinez et al. REF reported on a remarkable large experiment, where three repair tools were executed on the bugs from Defects4J.
Automatic repair of real bugs in java: a large-scale experiment on the defects4j dataset
Empirical Software Engineering
Empirical Software Engineering
Computer Science