The goal of science is conclusion stability, i.e. to discover some effect X that holds in multiple situations. Sadly, there are all too few examples of stable conclusions in software engineering (SE). In fact, the typical result is conclusion instability where what is true for project one, does not hold for project two. We can find numerous studies of the following form: there is as much evidence for as against the argument that some aspect X adds value to a software project. Below are four examples of this type of problem which we believe to be endemic within SE. ported from other organizations were as useful as local data (for the purposes of building effort models). From a total of seven studies, three found that models from other organizations were not significantly worse than those based on local data, while four found that they were significantly different (and worse).
Although research in the field of software engineering often shows conclusion instability (where what is true for project one, does not hold for project two) REF , and expert judgement is common practice, studies do emphasize pitfalls.
Special issue on repeatable results in software engineering prediction
Empirical Software Engineering
Empirical Software Engineering
Computer Science