Identifying the Source of Vulnerability in Explanation Discrepancy: A Case Study in Neural Text Classification (2022)

Abstract: Some recent works observed the instability of post-hoc explanations when input side perturbations are applied to the model. This raises the interest and concern in the stability of post-hoc explanations. However, the remaining question is: is the instability caused by the neural network model or the post-hoc explanation method? This work explores the potential source that leads to unstable post-hoc explanations. To separate the influence from the model, we propose a simple output probability perturbation method. Compared to prior input side perturbation methods, the output probability perturbation method can circumvent the neural model’s potential effect on the explanations and allow the analysis on the explanation method. We evaluate the proposed method with three widely-used post-hoc explanation methods (LIME (Ribeiro et al., 2016), Kernel Shapley (Lundberg and Lee, 2017a), and Sample Shapley (Strumbelj and Kononenko, 2010)). The results demonstrate that the post-hoc methods are stable, barely producing discrepant explanations under output probability perturbations. The observation suggests that neural network models may be the primary source of fragile explanations.

## introduction despite the remarkable performance of neural network models in natural language processing (nlp), the lack of interpretability has raised much concern in terms of their reliability and trustworthiness @xcite @xcite . a common way to improve a model's interpretability is to generate explanations for its predictions from the posthoc manner. we call these explanations post-hoc explanations @xcite . post-hoc explanations demonstrate the relationship between the input text and the model prediction by identifying feature importance scores @xcite . in general, a feature with a higher importance score is more important in contributing to the prediction result. based on feature importance scores, we can select top important features as the model explanation. however, some recent works @xcite @xcite @xcite have observed explanation discrepancy when input-side perturbation is applied to the model. one question to this observation is what makes the explanation discrepant? explanations generated by a post-hoc method @xcite @xcite depend on a model's prediction probabilities. if perturbations at the input side cause model prediction probabilities to change, post-hoc explanations may change accordingly. in figure 1 (a), we demonstrate a simple example of the process that generates explanations using a post-hoc method. the explanation is generated depending on the probability p . in figure 1 (b), we demonstrate an example of the same process with perturbation at the input side. the explanation is generated depending on the probability p . the output probabilities in the two examples are not the same, i.e. p ̸ = p . in figure 1 (a ) and (b), it is noticeable that the feature importance score of the same feature has changed. for instance, the feature "love" has different importance scores in the two examples. since feature importance scores are inconsistent, the explanations in the two examples are different. we call this explanation discrepancy, which will be introduced more in subsection 2.2. however, the prediction label in figure 1 (a), ŷ, and the prediction label in figure 1 (b), ȳ, are equal, which is ŷ = ȳ = positive. this indicates that input side perturbations may not flip the model prediction label, while can make output probabilities change, hence further leading to explanation discrepancy. we argue that, under input side perturbations, it is difficult to identify the source causing the explanation discrepancy. one intuitive justification is that the perturbation at the input side has to pass through both the model and the post-hoc explanation method. both the model and the post-hoc explanation method are possible factors that result in unstable explanations. for example, the model's prediction behavior may change under input side perturbations, that is focusing on different features to make predictions, hence resulting in the explanation discrepancy @xcite . or the explanation method itself may be vulnerable to input perturbations, producing discrepant explanations. the instability may not be told from the prediction results, but reflected in the explanations, i.e., explanation discrepancy in this paper, we propose a simple strategy to demonstrate the potential source that causes explanation discrepancy. to circumvent the potential influence of the model on the explanations, we design an output probability perturbation method by slightly modifying the prediction probabilities, as shown in figure 1 (c ). in this work, we focus on the model-agnostic post-hoc methods, lime @xcite , kernel shapley (lundberg and lee, 2017a), and sample shapley @xcite , that explain the black-box models. if a similar explanation discrepancy can be observed when only output probability perturbation is applied, it would suggest that post-hoc explanation methods may be unstable because the potential influence from the black-box model has been blocked. otherwise, we should not blame post-hoc explanation methods as the source of vul-nerability in fragile explanations @xcite . ## previous works post-hoc explanation methods most works focus on explaining neural network models in a posthoc manner, especially generating a local explanation for each model prediction. the white-box explanation methods, such as gradient-based explanations @xcite , and attention-based explanations @xcite , either require additional information (e.g. gradients) from the model or incur much debates regarding their faithfulness to model predictions @xcite . another line of work focuses on explaining black-box models in the model-agnostic way. @xcite proposed a perturbation-based explanation method, leave-one-out, that attributes feature importance to model predictions by erasing input features one by one. @xcite proposed to estimate feature contributions locally via linear approximation based on pseudo examples. some other works proposed the variants of the shapley value @xcite to measure feature importance, such as the sample shapley method @xcite , the kernel shapley method (lundberg and lee, 2017a), and the l/c-shapley method @xcite . model robustness recent works have shown the vulnerability of model due to adversarial attacks @xcite @xcite . some adversarial examples are similar to original examples but can quickly flip model predictions, which causes concern on model robustness @xcite . in the text domain, a common way to generate adversarial examples is by heuristically manipulating the input text, such as replacing words with their synonyms @xcite @xcite , misspelling words @xcite , inserting/removing words @xcite , or concatenating triggers @xcite . explanation robustness previous work explored explanation robustness by either perturbing the inputs @xcite @xcite or manipulating the model @xcite @xcite . for example, slack's group fooled post-hoc explanation methods by hiding the bias for black-box models based on the proposed novel scaffolding technique @xcite . however, all of these works cannot disentangle the sources that cause fragile explanation. differently, the proposed method mitigates the influence of model to the explanation by perturbing model outputs. ## conclusion in this work, our main contribution is to identify the primary source of fragile explanation, where we propose an output probability perturbation method. with the help of this proposed method, observation results can illustrate a conclusion that the primary potential source that caused fragile explanations in the previous studies is the black-box model itself, which also illustrate that some limitations of prior methods. furthermore, in subsection 3.4, we analyze the kernel calculation inside the lime algorithm @xcite . we apply the condition number of the matrix and simulation experiments to demonstrate that the kernel calculation matrix inside lime has a low condition number. this result further suggests the stability of the lime algorithm.