Abstract-An effective data collection method for evaluating software Vious [15], [32]- [34] and companion [35] papers present development methodologies and for studying the software development data and evaluation results, obtained from two different softprocess is described. The method uses goal-directed data collection to evaluate methodologies with respect to the claims made for them. Such wrevdevelopmenti wenroeints (n theestuechnues claims are used as a basis for defining the goals of the data collection, previously mentioned were included in these studies.) The establishing a list of questions of interest to be answered by data analy-methodology described in this paper was developed as part of sis, defining a set of data categorization schemes, and designing a data studies conducted by the Naval Research Laboratory (NRL) collection form. and by NASA's Software Engineering Laboratory (SEL) [36] . The data to be collected are based on the changes made to the software The remainder of this section discusses motivation for data during development, and are obtained when the changes are made. To ensure accuracy of the data, validation is performed concurrently with cection and stheatbutes oaseful data collectione software development and data collection. Validation is based on inter-Section II is a step-by-step description of the data collection views with those people supplying the data. Results from using the meth-methodology. Section III describes the application of the odology show that data validation is a necessary part of change data methodology to the SEL environment. Section IV summarizes collection. Without it, as much as 50 percent of the data may be the lessons learned concerning data collection and its associerroneous. atdpolm,lmttosanaplcin. Feasibility of the data collection methodology was demonstrated-by ated problems, limitations, and applications. applying it to five different projects in two different environments. The application showed that the methodology was both feasible and Software Engineering Experimentation useful. The course of action in most sciences when faced with a Index Terms-Data collection, data collection methodology, error question of opinion is to obtain experimental verification. Softanalysis, error classification, software engineering experimentation. ware engineering disputes are infrequently settled that way. Data from experiments exist, but rarely apply to the question to be settled. There are a number of reasons for this state of I. INTRODUCTION affairs. Probably the two most important are the number of A CCORDING to the mythology of computer science, the potential confounding factors involved in software studies and first computer program ever written contained an error. the expense of attempting to do controlled studies in an indusError detection and error correction are now considered to be trial environment involving medium or large scale systems. the major cost factors in software development [1] -[3] Much Rather than attempting controlled studies, we have devised a current and recent research is devoted to finding ways of pre-method for conducting accurate causal analyses in production venting software errors. This research includes areas such as environments. Causal analyses are efforts to discover the causes requirements definition [4] , automatic and semiautomatic of errors and the reasons that changes are made to software. program generation [5], [6], functional specification [7], Such analyses are designed to provide some insight into the abstract specification [8] - [11], procedural specification [12], software development and maintenance processes, help confirm code specification [13] [27] , and language design [16] , [28] - [31] . errors. Relatively few examples of this kind of study exist in One result of this research is that techniques claimed to be the literature; some examples are [4] , [15], [32], [37] , [38] . effective for preventing errors are in abundance. Unfortunately, there have been few attempts at experimental verification of Attributes of Useful Data Collection such claims. The purpose of this paper is to show how to To provide useful data, a data collection methodology must obtain valid data that may be used both to learn more about displa certain attributes. Since much of the data of interest the software development process and to evaluate software are collected during the test phase, complete analysis of the development methodologies In production environments. Pre-data must await project completion. For accuracy reasons, it Manuscript received December 13, 1982; revised January 11, 1984.ismprathtdtacleioanvldtonrcedoThis work was supported in part by the National Aeronautics and Space currently with development. Administration under Grant NSF-S5123 to the University of Maryland. Developers can provide data as they make changes during V. R. Basili is with the Department of Computer Science, University development. In a reasonably well-controlled software devel- both data collection and data validation is quite important to the accuracy of the analysis. The projects studied vary widely with respect to factors such Careful validation means that the data to be collected must as application, size, development team, methodology, hardware, be carefully specified, so that those supplying data, those vali-and support software. Nonetheless, the same basic data collecdating data, and those performing the analyses will have a con-tion methodology was applicable everywhere. The schema used sistent view of the data collected. This is especially important has six basic steps, listed in the following, with considerable for the purposes of repetition of the studies in both the same feedback and iteration occurring at several different places. and different environments. 1) Establish the Goals of the Data Collection: We divide Careful specification of the data requires the data collectors goals into two categories: those that may be used to evaluate a to have a clear idea of the goals of the study. Specifying goals particular software development methodology relative to the is itself an important issue, since, without goals, one runs the claims made for it, and those that are common to all methodrisk of collecting unrelated, meaningless data. ologies to be studied. To obtain insight into the software development process, the As an example, a goal of a particular methodology, such as data collectors need to know the kinds of errors committed and information hiding [41] , might be to develop software that the kinds of changes made. To identify troublesome issues, the is easy to change. The corresponding data collection goal is to effort needed to make each change is necessary. For greatest evaluate the success of the developers in meeting this goal, i.e., usefulness, one would like to study projects from software evaluate the ease with which the software can be changed. production environments involving teams of programmers. Goals in this category may be of more interest to those who We may summarize the preceding as the following six are involved in developing or testing a particular methodology, criteria. and must be defined cooperatively with them. 1) The data must contain information permitting identifica-A goal that is of interest regardless of the methodology being tion of the types of errors and changes made. used is to help understand the environment and focus atten- 2) The data nmust include the cost of making changes. tion on techniques that are useful there. Another such goal is 3) Data to be collected must be defined as a result of clear to characterize changes in ways that permit comparisons across specification ofuthe goals ofsthe study. projects and environments. Such goals may interest soft-4) Data should include studies of projects from production ware engineers, programmers, managers, and others more than environments, involving teams of programmers. goals that are specific to the success or failure of a particular 5) Data analysis should be historical; data must be collected methodology. and validated concurrently with development. Consequences of Omitting Goals: Without goals, one is 6) Data classification schemes to be used must be carefully likely to obtain data in which either incomplete patterns or no specified for the sake of repeatability of the study in the same patterns are discernible. As an example, one goal of an early and different environments. study [15] was to characterize errors. During data analysis, it became desirable to discover the fraction of errors that were the result of changes made to the software for some reason other Our data collection methodology is goal oriented. It starts than to correct an error. Unfortunately, none of the goals of with a set of goals to be satisfied, uses these to generate a set the study was related to this type of change, and there were of questions to be answered, and then proceeds step-by-step no such data available. through the design and implementation of a data collection 2) Develop a List of Questions ofInterest: Once the goals of and validation mechanism. Analysis of the data yields answers the study have been established, they may be used to develop to the questions of interest, and may also yield a new set Of a list of questions to be answered by the study. Questions of questions. The procedure relies heavily on an interactive data interest define data parameters and categorizations that permit validation process; those supplying the data are interviewed for quantitative analysis of the data. In general, each goal will result validation purposes concurrently with the software develop-in the generation of several different questions of interest. As ment process. The methodology has been used in two different an example, if the goal is to characterize changes, some correenvironments to study five software projects developed by sponding questions of interest are: "What is the distribution of groups with different backgrounds, using very different soft-changes according to the reason for the change?", "What is the ware development methodologies. In both environments it distribution of changes across system components?", "What is yielded answers to most questions of interest and some insight the distribution of effort to design changes?" into the development methodologies used. Table I is a sumAs a second example, if the goal is to evaluate the ease with mary of characteristics of completed projects that have been which software can be changed, we may identify questions of studied. Definitions of the characteristics are the same as in interest such as: "Is it clear where a change has to be made in [40] . All examples used in this paper are taken from studies the software?", "Are changes confilned to single modules?", of the SEL environement. "What was the average effort involved in making a change?" 730 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. SE-10, NO. 6, NOVEMBER 1984 Questions of interest form a bridge between subjectively may result in data that cannot later be identified as fitting any determined goals of the study and the quantitative measures particular categorization. Each change then defines its own to be used in the study. They permit the investigators to deter-category, and the result is an overwhelming multiplicity of mine the quantities that need to be measured and the aspects data categories, with little data in each category. of the goals that can be measured. As an example, to discover 4) Design and Test Data Collection Form: To provide a how a design document is being used, one might collect data permanent copy of the data and to reinforce the programmers' that show how the document was being used when the need memories, a data collection form is used. Form design was for a change to it was discovered. This may be the only aspect one of the trickiest parts of the studies conducted, primarily of the document's use that is measurable. because forms represent a compromise among conflicting In addition to forcing sharper definition of goals, questions objectives. Typical conflicts are the desire to collect a comof interest have the desirable property of forcing the investiga-plete, detailed set of data that may be used to answer a wide tors to consider the data analyses to be performed before any range of questions of interest, and the need to minimize the time data are collected. and effort involved in supplying the data. Satisfying the former Goals for which questions of interest cannot be formulated leads to large, detailed forms that require much time to fill and goals that cannot be satisfied because adequate measures out. The latter requires a short, check-off-the-boxes type of cannot be defined may be discarded. Once (hereafter called modifications). Fig. 1 shows the last version of the form used for the SEL Each of these categories may be further subcategorized studies reported here. (An earlier version of the form was according to reason. As an example, modifications could be significantly modified as a result of experience gained in the subdivided into modifications resulting from requirements data collection and analysis processes.) The first sections of changes, modifications resulting from a change in the develop-the form request textual descriptions of the change and the ment support environment (e.g., compiler change), planned reason it was made. Following sections contain questions and enhancements, optimizations, and others. check-off tables that reflect various categorization schemes. Such a categorization permits characterization of the changes As an example, a categorization of time to design changes is with respect to the stability of the development environment, requested in the first question following the description of the with respect to different kinds of development activities, change. The completer of the form is given the choice of four etc. When matched with another categorization such as the categories (one hour or less, one hour to one day, one day to difficulty of making changes, this scheme also reveals which three days, and more than three days) that cover all possibilchanges are the most difficult to make. ities for design time. Each categorization scheme should be complete and consisConsequences of Not Using a Data Collection Form: Withtent, i.e., every change should fit exactly one of the subcate-out a data collection form, it is necessary to rely on the develgories of the scheme. To ensure completeness, we usually add oper's memories and on perusal of early versions of design the category "Other" as a subcategory. Where some changes are documentation and code to identify and categorize the changes not suited to the scheme, the subcategory "Not Applicable" made. This appro)ach leads to incomplete, inaccurate data. may be used. As collection and validation are concurrent with software devel-interest. As an example, to answer the question "What was opment; the shorter the lag between completing the form and the distribution of changes according to the reason for the conducting the interview, the more accurate the data. change?", a distribution such as that shown in Fig. 2 might be Perhaps the most significant problem during data collection computed from the data. and validation is ensuring that the data are complete, i.e., that every change has been described on a form. The better con-Application of the Schema trolled the development process, the easier this is to do. At Applying the schema requires iterating among the steps seveach stage of the process where configuration control is im-eral times. Defining the goals and establishing the questions of posed, change data may be collected. Where projects that we interest are tightly coupled, as are establishing the questions of have studied use formal configuration control, we have inte-interest designing and testing the form(s), and collecting and grated the configuration control procedures and the data col-validating the data. Many of the considerations involved in lection procedures, using the same forms for both, and taking implementing and integrating the steps of the schema have been advantage of configuration control procedures for validation omitted here so that the reader may have an overview of the purposes. Since all changes must be reviewed by a configura-process. The complete set of goals, questions of interest, and tion control board in such cases, we are guaranteed capture of data categorizations for the SEL projects are shown in [33] . all changes, i.e., that our data are complete. Furthermore, the data collection overhead is absorbed into the configuration SupportProceduresandFacilities control overhead, and is not visible as a separate source of irritaIn addition to the activities directly involved in the data tion to the developers. collection effort, there are a number of support activities and Consequences of Omitting Validation: One result of con-facilities required. Included as support activites are testing current development, data collection, and data validation is the forms, collection and validation procedures, training the that the accuracy of the collection process may be quantified. programmers, selecting a database system to permit easy analyAccuracy may be calculated by observing the number of mis-sis of the data, encoding and entering data into the database, takes made in completing data collection forms. One may then and developing analysis programs. compare, for any data category, prevalidation distributions with postvalidation distributions. We call such an analysis a III. DETAILS OF SEL DATA COLLECTION validation analysis. The validation analysis of the SEL data AND VALIDATION shows that it is possible for inaccuracies on the order of 50 perIn the SEL environment, program libraries were used to supcent to be introduced by omitting validation. To emphasize port and control software development. There was a full-time the consequences of omitting the validation procedures, we librarian assigned to support SEL projects. All project library present some of the results of the validation analysis ofthe SEL changes were routed through the librarian. In general, we data in Section III. define a change to be an alteration to baselined design, code, 6) Analyze Data: Data are analyzed by calculating the pa-or documentation. For SEL purposes, only changes to code, rameters and distributions needed to answer the questions of and documentation contained in the code, were studied. The (Fig. 1) We define an error to be a discrepancy between a specification more than 5 percent of the changes and errors are misclassified and its implementation. Although it was not always possible in any of the data collection categories. For the major cateto identify the exact location of an error, it was always possi-gories, such as whether a change is an error or modification, ble to identify exactly each error correction. As a result, we the type of change, and the type of error, the inaccuracy is generally use the term error to mean error correction, probably no more than 3 percent. For data validation purposes, the most important parts of For SEL3, we attempted to quantify the results of the valithe data collection procedure are the review by the change dation procedures more carefully. After validation, forms analyst, and the associated programmer interview to resolve were categorized according to our confidence in their accuracy. uncertainties about the data. We used four categories. tion). Fig. 4 shows the number of generated forms expressed as a percentage of total validated forms. Variation in Misclassification Fig. 3 shows that prevalidation SEL3 forms were significantly Data on misciassifilcations of change and error type subcatemore accurate than the prevalidation SELl or SEL2 forms. gories, such as shown in Table III , tend to vary considerably Data collection goals, questions of interest, and data collection forms may have to be revised accordingly. In addition to the procedural problems involved in designing and implementing a data collection study, we found several IV* RECOMMENDATIONS FOR DATA COLLECTORS other pitfalls that could have strongly affected our results and We believe we now have sufficient experience with change their interpretation. They are listed in the following. data collection to be able to apply it successfully in a wide vari-1) Perhaps the most significant of these pitfalls was the danety of environments. Although we have been able to make ger of interpreting the results without attempting to understand comparisons between the data collected in the two environ-factors in the environment that might affect the data. As an ments we have studied, we would like to make comparisons example, we found a surprisingly small percentage of interface with a wider variety of environments. Such comparisons will errors on all of the SEL projects. This was surprising since only be possible if more data become available. To encourage interfaces are an often-cited source of errors. There was also the establishment of more data collection projects, we feel it is other evidence in the data that the software was quite amenable important to describe a successful data collection methodol-to change. In trying to understand these results, we discussed ogy, as we have done in the preceding sections, to point out them with the principal designer of the SEL projects (all of the pitfalls involved, and to suggest ways of avoiding those which had the same application). It was clear from the discuspitfalls. sion that as a result of their experience with the application, 736 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. SE-10, NO. 6, NOVEMBER 1984 the designers had learned what changes to expect to their sys-being studied, so that the data may be used for comparison tems, organized the design so that the expected changes would purposes, and so that those filling out the forms understand be easy to make, and then reused the design from one project the terminology used. Conduct training sessions in filling out to the next. Rather than misinterpreting the data to mean that forms for newcomers. interfaces were not a significant software problem, we were 5) Integrate data collection and validation procedures into led to a better understanding of the environment we were the configuration control process. Data completeness and studying. accuracy are thereby improved, data collection is unobtrusive, 2) A second pitfall was underestimating the resources needed and collection and validation become a part of the normal to validate and analyze the data. Understanding the change development procedures. In cases where configuration control reports well enough to conduct meaningful, efficient program-is not used or is informal, allocate considerable time to promer interviews for validation purposes initially consumed con-grammer interviews, and, if possible, documentation search siderable amounts of the change analysts' time. Verifying that and code reading. the database was internally consistent, complete, and consistent 6) Automate as much of the data analysis process as possible. with the paper copies of reports was a continuing source of frustration and a sink for time and effort. Limitations 3) A third potential pitfall in data collection is the sensitiv-It has been previously noted that the main limitation of using ity of the data. Programmers and designers sometimes need to a goal-directed data collection approach in a production softbe convinced that error data will not be used against them. This w e i did not seem to be a significant problem on the projects studied fars Forvet results cannot be used to prove that a particular factor in the 2) Establish the goals of the data collection methodology and development process causes particular kinds of errors, but can define the questions of interest before attempting any data be used to suggest that certain approaches, when applied in the collection. Establishing goals and defining questions shoud be environment studied, will improve the development process. an iterative process performed in concert with the developers. The software developer may then be provided with a set of The developers' interests are then served as well as the data recommended approaches for improving the software developcollector's. ment process in his environment. 3) For initial data collection efforts, keep the set of data As an example, in the SEL environment neither external collection goals small. Both the volume of data and the time problems, such as requirements changes, nor global problems, consumed in gathering, validating, and analyzing it will be such as interface design and specification, were signifilcant. unexpectedly large. Furthermore, the development environment was quite stable. 4) Design the data collection form so that it may be used Most problems were associated with the individual programfor configuration control, so that it is tailored to the project(s) mer. The data show that in the SEL environment it would
Consequently, it is hard to answer concrete questions when the maintainability of a system is to be investigated, which also limits the applicability of structured measurement approaches such as GQM REF in this context.
A Methodology for Collecting Valid Software Engineering Data
IEEE Transactions on Software Engineering
IEEE Transactions on Software Engineering
Computer Science