DeepMutation: A Mutation Testing Framework for Deep Learning Systems Deep neural networks (DNNs) are increasingly expanding their realworld applications across domains, e.g., image processing, speech recognition and natural language processing. However, there is still limited tool support for DNN testing in terms of test data quality and model robustness. In this paper, we introduce a mutation testingbased tool for DNNs, DeepMutation, which facilitates the DNN quality evaluation, supporting both feedforward neural networks (FNNs) and stateful recurrent neural networks (RNNs). It not only enables to statically analyze the robustness of a DNN model against the input as a whole, but also allows to identify the vulnerable segments of a sequential input (e.g. audio input) by runtime analysis. It is worth noting that DeepMutation specially features the support of RNNs mutation testing. The tool demo video can be found on the project website https:sites.google.comviewdeepmutationpp.

A Quantitative Analysis Framework for Recurrent Neural Network Recurrent neural network (RNN) has achieved great success in processing sequential inputs for applications such as automatic speech recognition, natural language processing and machine translation. However, quality and reliability issues of RNNs make them vulnerable to adversarial attacks and hinder their deployment in realworld applications. In this paper, we propose a quantitative analysis framework  DeepStellar  to pave the way for effective quality and security analysis of software systems powered by RNNs. DeepStellar is generic to handle various RNN architectures, including LSTM and GRU, scalable to work on industrialgrade RNN models, and extensible to develop customized analyzers and tools. We demonstrated that, with DeepStellar, users are able to design efficient test generation tools, and develop effective adversarial sample detectors. We tested the developed applications on three real RNN models, including speech recognition and image classification. DeepStellar outperforms existing approaches three hundred times in generating defecttriggering tests and achieves 97% accuracy in detecting adversarial attacks. A video demonstration which shows the main features of DeepStellar is available at: https:sites.google.comviewdeepstellartooldemo.