  Deep neural networks (DNNs) have achieved remarkable performance across a
wide area of applications. However, they are vulnerable to adversarial
examples, which motivates the adversarial defense. By adopting simple
evaluation metrics, most of the current defenses only conduct incomplete
evaluations, which are far from providing comprehensive understandings of the
limitations of these defenses. Thus, most proposed defenses are quickly shown
to be attacked successfully, which result in the "arm race" phenomenon between
attack and defense. To mitigate this problem, we establish a model robustness
evaluation framework containing a comprehensive, rigorous, and coherent set of
evaluation metrics, which could fully evaluate model robustness and provide
deep insights into building robust models. With 23 evaluation metrics in total,
our framework primarily focuses on the two key factors of adversarial learning
(\ie, data and model). Through neuron coverage and data imperceptibility, we
use data-oriented metrics to measure the integrity of test examples; by delving
into model structure and behavior, we exploit model-oriented metrics to further
evaluate robustness in the adversarial setting. To fully demonstrate the
effectiveness of our framework, we conduct large-scale experiments on multiple
datasets including CIFAR-10 and SVHN using different models and defenses with
our open-source platform AISafety. Overall, our paper aims to provide a
comprehensive evaluation framework which could demonstrate detailed inspections
of the model robustness, and we hope that our paper can inspire further
improvement to the model robustness.
