Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods (2023)

Abstract: A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. A common practice for evaluating whether an interpretability method is faithful has been to use evaluation-by-agreement – if multiple methods agree on an explanation, its credibility increases. However, recent work has found that saliency methods exhibit weak rank correlations even when applied to the same model instance and advocated for alternative diagnostic methods. In our work, we demonstrate that rank correlation is not a good fit for evaluating agreement and argue that Pearson-r is a better-suited alternative. We further show that regularization techniques that increase faithfulness of attention explanations also increase agreement between saliency methods. By connecting our findings to instance categories based on training dynamics, we show that the agreement of saliency method explanations is very low for easy-to-learn instances. Finally, we connect the improvement in agreement across instance categories to local representation space statistics of instances, paving the way for work on analyzing which intrinsic model properties improve their predisposition to interpretability methods.

## introduction following the meteoric rise of the popularity of neural nlp models during the neural revolution, they have found practical usage across a plethora of domains and tasks. however, in a number of high-stakes domains such as law @xcite , finance @xcite , and medicine @xcite , the opacity of deep learning ♣ equal contribution ♥ work done while author was at takelab 1 our code is available at https://github.com/ mttk/unity-in-xai-court methods needs to be addressed. in the area of explainable artificial intelligence (xai), one of the major recent efforts is to unveil the neural black box and produce explanations for the end-user. there are various approaches to rationalizing model predictions, such as using the attention mechanism @xcite , saliency methods @xcite @xcite @xcite , rationale generation bydesign @xcite @xcite , or self-rationalizing models @xcite . these methods have to simultaneously satisfy numerous desiderata to have practical application in high-stakes scenarios: they have to be faithful -an accurate representation of the inner reasoning process of the model, and plausibleconvincing to human stakeholders. when evaluating faithfulness in using attention as explanations, @xcite have shown that attention importance scores do not correlate well with gradient-based measures of feature importance. the authors state that although gradient-based measures of feature importance should not be taken as ground truth, one would still expect importance measures to be highly agreeable, bringing forth the agrement-as-evaluation paradigm @xcite . while the imperfect agreement is something one could expect as interpretability methods differ in their formulation, and it is reasonable to observe differences in importance scores, subsequent work has shown that saliency methods exhibit low agreement scores even when applied to the same model instance @xcite . since a single trained model instance can only have a single feature importance ranking for its decision, disagreement of saliency methods implies that at least one, if not all methods, do not produce faithful explanations -placing doubt on their practical relevance. it has been hypothesized that unfaithfulness of attention is caused by input entanglement in the hidden space @xcite . this claim has later been experimentally verified through results showing that regularization techniques targeted to reduce entanglement significantly improve the faithfulness of attention-based explanations @xcite . while entanglement in the hidden space is clearly a problem in the case of attention explanations, where attention weights directly pertain to hidden states, we also hypothesize that representation entanglement could cause similar issues for gradient-and propagationbased explainability methods -which might not be able to adequately disentangle importance when propagating toward the inputs. in our work, we first take a closer look at whether the rank correlation is an appropriate method for evaluating agreement and confirm that, as hypothesized in previous work, small differences in values of saliency scores significantly affect agreement scores. we argue that a linear correlation method such as pearson-r is less sensitive to perturbations since the exact ranking order of features is not as crucial for agreement as the relative importance values, which pearson-r adequately captures. we hypothesize that the cause of saliency method disagreements is rooted in representation entanglement and experimentally show that agreement can be significantly improved by regularization techniques such as tying @xcite and conicity @xcite . the fact that regularization methods, which were originally aimed at improving faithfulness of attention, also improve agreement between saliency methods suggests that the two problems have the same underlying cause. taking the analysis deeper, we apply techniques from dataset cartography @xcite and show that, surprisingly, the explanations of easy-to-learn instances exhibit a lower agreement than of ambiguous instances. we further analyze how local curvature of the representation space morphs when regularization techniques are applied, paving the way for further analysis of (dis)agreements between interpretability methods. ## background and related work explainability methods come in different flavors determined by the method of computing feature importance scores. saliency methods perform posthoc analysis of the trained black-box model by either leveraging gradient information @xcite , modifying the backpropagation rules @xcite , or training a shallow interpretable model to locally approximate behavior of the blackbox model @xcite , all with the goal of assigning scalar saliency scores to input features. alternatively, if the analyzed model is capable of generating text, one can resort to selfrationalization by prompting the trained model to generate an explanation for its decision @xcite . in contrast to post-hoc explanations, inherently interpretable models produce explanations as part of their decision process, either by masking a proportion of input tokens and then performing prediction based on the remaining rationale @xcite @xcite , or jointly performing prediction and rationale generation in cases where datasets with annotated rationales are available @xcite . for some time, the attention mechanism @xcite has also been considered inherently interpretable. however, the jury is still out on whether such explanations can be considered faithful @xcite @xcite . faithfulness is one of the most important desiderata of explanation methods (jacovi and goldberg, 2020) -faithful explanations are those that are true to the inner decision-making process of the model. approaches to evaluating faithfulness rely on measuring how the confidence of the model changes when inputs are perturbed @xcite or completely dropped from the model @xcite . however, perturbations to input often result in corrupted instances that fall off the data manifold and appear nonsensical to humans @xcite or fail to identify all salient tokens properly (roar; @xcite -raising questions about the validity of perturbation-based evaluation. recursive roar @xcite alleviates the issues of its predecessor at the cost of requiring many prohibitively expensive retraining steps, further motivating us to seek efficient solutions which do not require retraining the model multiple times. another option is to leverage the evaluation-by-agreement (jain and wallace, 2019) paradigm, which states that an interpretability method should be highly agreeable with other methods to be considered faithful. however, since empirical evidence has shown that saliency methods exhibit poor agreement between their explanations @xcite , @xcite recommend practitioners consider alternative methods for evaluating the quality of interpretability methods, such as diagnostic tests. finally, methods such as data staining @xcite and lexical shortcuts @xcite artificially introduce tokens that act as triggers for certain classes -creating a ground truth for faithfulness which can be used as a comparison. nevertheless, such methods have a certain drawback in that they only offer the ground truth importance of a few artificially inserted tokens, but offer no insight regarding the relative importance of the remainder of the input. each of the aforementioned methods for estimating faithfulness of interpretability methods has its drawbacks (jacovi and goldberg, 2020), and we argue each should be taken in conjunction with others to increase the credibility of their collective verdict. ## preliminaries in this section, we delineate our experimental setup, detailing the considered datasets, models, their training procedure, the saliency methods which we use to interpret the decisions of the models, and the regularization techniques we use to improve agreement between saliency methods. ## improving agreement in this section, we present two modifications of the existing evaluation-by-agreement procedure: ## conclusion we analyzed two prototypical models from different families in jwa and dbert with the goal of finding out the cause of low agreement between saliency method interpretations. we first take a closer look at kendall-τ , the previously used rankorder correlation metric, and demonstrate that it can be prone to exhibiting high differences in agreement for small perturbations in importance scores. to account for this, when analyzing agreement be-