A number of papers have investigated the relationships between design metrics and the detection of faults in objectoriented software. Several of these studies have shown that such models can be accurate in predicting faulty classes within one particular software product. In practice, however, prediction models are built on certain products to be used on subsequent software development projects. How accurate can these models be considering the inevitable differences that may exist across projects and systems? Organizations typically learn and change. From a more general standpoint, can we obtain any evidence that such models are economically viable tools to focus validation and verification effort? This paper attempts to answer these questions using fault and design data collected on two midsize Java systems developed in the same environment. Another contribution of the paper is the use of a novel exploratory analysis technique (MARS) to build such faultproneness models, whose functional form is a priori unknown. Results indicate that a model built on one system can be accurately used to rank classes within another system according to their fault-proneness. The downside, however, is that, because of system differences, the predicted fault probabilities are not representative of the system predicted. Furthermore, a cost-benefit model demonstrates that the MARS fault-proneness model is potentially viable, from an economical standpoint. The linear model is not, thus suggesting a more complex model is required. INTRODUCTION Measures of structural design properties such as coupling or complexity are widely considered to be indicators of external system quality attributes, such as reliability or maintainability. Many measures taking structural properties of object-oriented systems into account were proposed in the literature [1] [8] [6] . A number of case studies have provided empirical evidence that by using regression analysis techniques, highly accurate prediction models for class faultproneness can be built from existing OO design measures [4] [5] . The accuracy of the prediction models in these studies is usually quantified by some measure of goodness of fit of the model predictions to the actual fault data, or using within-system cross-validation techniques, such as V-crossvalidation [13] . The purpose of building such models is to apply them to other systems (e.g., different systems developed in the same development environment), in order to focus verification and validation efforts on fault-prone parts of those systems. But little is known about the accuracy of prediction models when they are actually applied under realistic conditions. In that sense, the existing studies can mostly be characterized as feasibility studies. In this paper, we build a fault-proneness prediction model based on a set of OO measures using data collected from a mid-sized Java system, and then apply the model to a different Java system developed by the same team. We then evaluate the accuracy the model's prediction in that system and the model's economic viability using a cost-benefit model. A second objective is to use a novel exploratory, regression-based technique called MARS (Multivariate Adaptive Regression Splines [11]). We investigate whether this technique can result in more accurate fault-proneness models than typical logistic regression models assuming a linear relationship between covariates and the logit term. Model functional forms are difficult to determine a priori as we do not have credible, precise theories to devise them. Results indicate that a model built on one system can be used to accurately rank classes within another system according to their fault-proneness. However, only the MARS model is accurate whereas the linear logistic regression model does not seem appropriate. A cost-benefit model demonstrates such a fault-proneness model is potentially viable as it can provide clear benefits. However, the probabilities of a model built on one system applied to another suggest that differences in system factors make these probabilities meaningless, though the class ranking is preserved. In other words, if this is confirmed, such models' probabilities would need to be calibrated for each system. The implications of such a limitations are discussed below. The paper is structured as follows. Section 2 describes the setting of the study. Section 3 briefly lays down our analysis methodology and introduces the modeling techniques, logistic regression, and MARS. Analysis results are detailed in Section 4, and we draw our conclusions in Section 5.
For example, Briand et al. REF built the fault-proneness model on one project and used this model to other projects.
Assessing the Applicability of Fault-Proneness Models Across Object-Oriented Software Projects
IEEE Trans. Software Eng.
Computer Science