Machine Learning algorithms have provided important core functionality to support solutions in many scientific computing applications -such as computational biology, computational linguistics, and others. However, it is difficult to test such applications because often there is no "test oracle" to indicate what the correct output should be for arbitrary input. To help address the quality of scientific computing software, in this paper we present a technique for testing the implementations of machine learning classification algorithms on which such scientific computing software depends. Our technique is based on an approach called "metamorphic testing", which has been shown to be effective in such cases. Also presented is a case study on a real-world machine learning application framework, and a discussion of how programmers implementing machine learning algorithms can avoid the common pitfalls discovered in our study. We also conduct mutation analysis and cross-validation, which reveal that our method has very high $ A preliminary version of this paper was presented at the 9th International Conference on Quality Software (QSIC 2009) effectiveness in killing mutants, and that observing expected cross-validation result alone is not sufficient to test for the correctness of a supervised classification program. Metamorphic testing is strongly recommended as a complementary approach. Finally we discuss how our findings can be used in other areas of computational science and engineering.
A case study done on real world machine learning application framework shows that MRs can effectively detect faults REF .
Testing and Validating Machine Learning Classifiers by Metamorphic Testing ✩
The Journal of systems and software
The Journal of systems and software
Computer Science Medicine