  Deep neural networks are easily misled by adversarial examples. Although lots
of defense methods are proposed, many of them are demonstrated to lose
effectiveness when against properly performed adaptive attacks. How to evaluate
the adversarial robustness effectively is important for the realistic
deployment of deep models, but yet still unclear. To provide a reasonable
solution, one of the primary things is to understand the error (or gap) between
the true adversarial robustness and the evaluated one, what is it and why it
exists. Several works are done in this paper to make it clear. Firstly, we
introduce an interesting phenomenon named gradient traps, which lead to
incompetent adversaries and are demonstrated to be a manifestation of
evaluation error. Then, we analyze the error and identify that there are three
components. Each of them is caused by a specific compromise. Moreover, based on
the above analysis, we present our evaluation suggestions. Experiments on
adversarial training and its variations indicate that: (1) the error does exist
empirically, and (2) these defenses are still vulnerable. We hope these
analyses and results will help the community to develop more powerful defenses.
