  Machine learning (ML) has been widely used in the literature to automate
software engineering tasks. However, ML outcomes may be sensitive to
randomization in data sampling mechanisms and learning procedures. To
understand whether and how researchers in SE address these threats, we surveyed
45 recent papers related to three predictive tasks: defect prediction (DP),
predictive mutation testing (PMT), and code smell detection (CSD). We found
that less than 50% of the surveyed papers address the threats related to
randomized data sampling (via multiple repetitions); only 8% of the papers
address the random nature of ML; and parameter values are rarely reported (only
18% of the papers). To assess the severity of these threats, we conducted an
empirical study using 26 real-world datasets commonly considered for the three
predictive tasks of interest, considering eight common supervised ML
classifiers. We show that different data resamplings for 10-fold
cross-validation lead to extreme variability in observed performance results.
Furthermore, randomized ML methods also show non-negligible variability for
different choices of random seeds. More worryingly, performance and variability
are inconsistent for different implementations of the conceptually same ML
method in different libraries, as also shown through multi-dataset pairwise
comparison. To cope with these critical threats, we provide practical
guidelines on how to validate, assess, and report the results of predictive
methods.
