Can We Really Trust Explanations? Evaluating the Stability of Feature Attribution Explanation Methods via Adversarial Attack (2022)

Abstract: “Explanations can increase the transparency of neural networks and make them more trustworthy. However, can we really trust explanations generated by the existing explanation methods? If the explanation methods are not stable enough, the credibility of the explanation will be greatly reduced. Previous studies seldom considered such an important issue. To this end, this paper proposes a new evaluation frame to evaluate the stability of current typical feature attribution explanation methods via textual adversarial attack. Our frame could generate adversarial examples with similar textual semantics. Such adversarial examples will make the original models have the same outputs, but make most current explanation methods deduce completely different explanations. Under this frame, we test five classical explanation methods and show their performance on several stability-related metrics. Experimental results show our evaluation is effective and could reveal the stability performance of existing explanation methods.”

## introduction fueled by recent rapid development in deep learning, nlp systems have obtained promising results in several fields, such as medical, law and commerce @xcite . however, besides the predicted results, users concern more on how these results are generated @xcite . to this end, lots of emphases have been set upon the explanation methods for neural networks @xcite @xcite . although the current explanation methods have increased the transparency of the neural networks and provided explanations as supports for predicted results, most of them ignored important questions: are these methods reliable and the generated explanations really trustful? besides the widely used focused properties of explanation methods, such as faithfulness, plausibility @xcite @xcite , readableness @xcite and compactness @xcite , we believe stability is an important but often overlooked property @xcite . when we put a small perturbation on the input, which would not change the input semantic and the output of the original model, we believe that the explanation method is not stable enough when we obtain the same outputs with quite different explanations. for example, figure 1 shows all results of major explanation methods would change when we just replace fine by refined, including lime @xcite , leave-one-out @xcite , vanilla gradient @xcite , smooth gradient @xcite , integrated gradient @xcite . to fulfill the stability testing, we intuitively consider existing word-substitution based textual adversarial attack methods 0 @xcite , since it is under the black-box 1 settings and figure 1 : an example of the result of our adversarial attack. we select a sentence from sst-2 and show the adversarial examples for explanation method vanilla gradient @xcite . ori and adv stand for original sentence and corresponding adversarial example respectively. we show the three most important tokens and sign them in different colors. finally, we put the attack on five typical feature attribution explanation methods. experimental results show their performance on stability. we find perturbation-based explanation methods perform better on stability than gradient-based methods. all of the source code and data will be available soon. ## formulation in this section, we first introduce the basic information of the common textual adversarial attack in section 3.1. then we introduce how to formulate explanation adversarial attack in section 3.2. ## attack method according to section 3.2, we need to measure the explanation difference. therefore, we propose two metrics in section 4.1. then we present our detailed attack strategies to attack existing explanation methods in section 4.2. ## discussion beyond the above experiments, our discussions would address the following research questions: ## conclusion this paper proposes a new evaluation frame to evaluate the stability of typical feature attribution explanation methods via adversarial attack. various experimental results on different experimental settings reveal their performance on stability, which also show the effectiveness of our proposed evaluation frame. we also conduct experiments to show the proposed frame is dependent of specific step. therefore, we hope the proposed evaluation frame could be applied to evaluating the stability of feature attribution explanation methods in the future and attract more research on this important but often overlooked property.