Abstract-Software engineering process information extracted from version control systems and bug tracking databases are widely used in empirical software engineering. In prior work, we showed that these data are plagued by quality deficiencies, which vary in its characteristics across projects. In addition, we showed that those deficiencies in the form of bias do impact the results of studies in empirical software engineering. While these findings affect software engineering researchers the impact on practitioners has not yet been substantiated. In this paper we, therefore, explore (i) if the process data quality and characteristics have an influence on the bug fixing process and (ii) if the process quality as measured by the process data has an influence on the product (i.e., software) quality. Specifically, we analyze six Open Source as well as two Closed Source projects and show that process data quality and characteristics have an impact on the bug fixing process: the high rate of empty commit messages in Eclipse, for example, correlates with the bug report quality. We also show that the product quality -measured by number of bugs reported -is affected by process data quality measures. These findings have the potential to prompt practitioners to increase the quality of their software process and its associated data quality. Keywords-case study; process quality; product quality; correlation; mining software repositories Empirical software engineering researchers make use of software process data such as bug reports and version control log files, e.g., to predict the number and locale of bugs in future software releases (e.g., Unfortunately, the process data characteristics and quality may have a major impact on the results of studies that rely on such data. In prior work we, therefore, introduced process data quality and characteristics measures, which make an evaluation of process data possible. Based on these measures we surveyed five Open Source Software (OSS) projects and one Closed Source Software (CSS) project and showed that such process data is plagued by quality issues across all surveyed projects [5] . In addition to the quality issues in these process datasets, we showed that there are vast differences in the data characteristics of these projects often used in empirical software engineering studies. These findings give rise to one main question: Why should one care about data quality and characteristics issues? In a first step, this lead to the following two research questions: RQ 1: Do quality issues in software engineering datasets have an impact on studies that rely on such datasets? We tried to answer both questions in prior work. In [6] and [7] we investigated historical data from several software projects, and found strong evidence of systematic bias. We then investigated potential effects of "unfair" or "imbalanced" datasets on the performance of prediction techniques. Our experiments suggested that bug-feature bias affects the performance of the award-winning BUGCACHE defect prediction algorithm and that this type of bias is a serious problem. We also discussed the impact of varying data characteristics across projects [5] and concluded that, e.g., the nature of used software engineering processes, the use of process support tools, and kind of testing (e.g., professionalized testing vs. beta user testing) result in differing data characteristics. Such differences may raise threats in generalizability of research results, algorithms, tools, and heuristics across projects' borders. Whereas RQ1 and RQ2 addressed the "why we should care about data quality and characteristics issues in empirical software engineering research", this paper would like to highlight why practitioners -software engineers and software project managers -should care about these issues. Therefore, we analyze the interplay of data quality and characteristics and its influence on the product (i.e., software) quality. Primarily, we answer the following two research questions: Basically, we analyze several research hypotheses based on our two new research questions (RQ3 and RQ4). To test our hypotheses, we selected six OSS projects, which are widely used in empirical software engineering studies. In addition, we used the data of two CSS projects. This allows a first comparison of results across different software engineering processes, project communication habits, and testing approaches. After the customary discussion of related work, this paper first explorers briefly the theoretical background of our study (Section III). We then present what software projects we selected, show what kind of data we used, and how we prepared them for this study (Section IV). In Section V, we
Bachmann and Bernstein REF explore among others if the process quality, as measured by the process data, has an influence on the product quality.
When process data quality affects the number of bugs: Correlations in software engineering datasets
2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)
2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)
Computer Science