Statistics comes in two main flavors: frequentist and Bayesian. For historical and technical reasons, frequentist statistics have traditionally dominated empirical data analysis, and certainly remain prevalent in empirical software engineering. This situation is unfortunate because frequentist statistics suffer from a number of shortcomings-such as lack of flexibility and results that are unintuitive and hard to interpret-that curtail their effectiveness when dealing with the heterogeneous data that is increasingly available for empirical analysis of software engineering practice. In this paper, we pinpoint these shortcomings, and present Bayesian data analysis techniques that provide tangible benefits-as they can provide clearer results that are simultaneously robust and nuanced. After a short, high-level introduction to the basic tools of Bayesian statistics, we present the reanalysis of two empirical studies on the effectiveness of automatically generated tests and the performance of programming languages, respectively. By contrasting the original frequentist analyses with our new Bayesian analyses, we demonstrate the concrete advantages of the latter. To conclude we advocate a more prominent role for Bayesian statistical techniques in empirical software engineering research and practice. Availability. All machine-readable data and analysis scripts used in this paper's analyses are freely available online at https://bitbucket.org/caf/bda-in-ese Empirical research in software engineering. Statistical analysis of empirical data has become commonplace in software engineering research [84, 3, 36] , and it is even making its way into software development practices [45] . As we discuss below, the overwhelming majority of statistical techniques that are being used in software engineering empirical research are, however, of the frequentist kind, with Bayesian statistics hardly even mentioned. Of course, Bayesian statistics is a fundamental component of many machine learning techniques [7, 39] ; as such, it is used in software engineering research indirectly whenever machine learning is used. In this paper, however, we are 3 concerned with the direct usage of statistics to analyze empirical data from the scientific perspective-a pursuit that seems mainly confined to frequentist techniques in software engineering [36] . As we argue in the rest of the paper, this is a lost opportunity because Bayesian techniques do not suffer from several limitations of frequentist ones, and can support rich, robust analyses in several situations. To validate the impression that Bayesian statistics are not normally used in empirical software engineering, we carried out a small literature review of ICSE papers. 1 We selected all papers from the main research track of the latest six editions of the International Conference on Software Engineering (ICSE 2013 to ICSE 2018) that mention "empirical" in their title or in their section's name in the proceedings. This gave 25 papers, from which we discarded one [76] that turned out not to be an empirical study. The experimental data in the remaining 24 papers come from various sources: the output of analyzers and other tools [16, 17, 18, 55, 62] , the mining of repositories of software and other artifacts [13, 48, 50, 70, 87, 88] , the outcome of controlled experiments involving human subjects [61, 79, 85] , interviews and surveys [5, 8, 23, 44, 46, 49, 65, 72] , and a literature review [74] . As one would expect from a top-tier venue like ICSE, the 24 papers follow recommended practices in reporting and analyzing data, using significance testing (6 papers), effect sizes (5 papers), correlation coefficients (5 papers), frequentist regression (2 papers), and visualization in charts or tables (23 papers). None of the papers, however, uses Bayesian statistics. In fact, no paper but two [23, 87] even mentions the terms "Bayes" or "Bayesian". One of the exceptions [87] only cites Bayesian machine-learning techniques used in related work to which it compares. The other exception [23] includes a presentation of the two views of frequentist and Bayesian statistics-with a critique of pvalues similar to the one we make in Sect. 2.2-but does not show how the latter can be used in practice. The aim of [23] is investigating the relationship between empirical findings in software engineering and the actual beliefs of programmers about the same topics. To this end, it is based on a survey of programmers whose responses are analyzed using frequentist statistics; Bayesian statistics is mentioned to frame the discussion about the relationship between evidence and beliefs, but does not feature past the introductory second section. Our paper has a more direct aim: to concretely show how Bayesian analysis can be applied in practice in empirical software engineering research, as an alternative to frequentist statistics; thus, its scope is complementary to [23]'s. More generally, we are not aware of any direct application of Bayesian data analysis to empirical software engineering data with the exception of [28, 29] and [26] . The technical report [28] and its short summary [29] are our preliminary investigations along the lines of the present paper. Ernst [26] presents a conceptual replication of an existing study to argue the analytical effectiveness of multilevel Bayesian models.
Only recently, Furia et al. REF propose such guidelines after reanalysing two empirical studies with Bayesian techniques revealing its advantages in providing clearer results that are simultaneously robust and nuanced.
Bayesian Data Analysis in Empirical Software Engineering Research
ArXiv
ArXiv
Computer Science Mathematics