On the Sensitivity and Stability of Model Interpretations in NLP (2022)

Abstract: Recent years have witnessed the emergence of a variety of post-hoc interpretations that aim to uncover how natural language processing (NLP) models make predictions. Despite the surge of new interpretation methods, it remains an open problem how to define and quantitatively measure the faithfulness of interpretations, i.e., to what extent interpretations reflect the reasoning process by a model. We propose two new criteria, sensitivity and stability, that provide complementary notions of faithfulness to the existed removal-based criteria. Our results show that the conclusion for how faithful interpretations are could vary substantially based on different notions. Motivated by the desiderata of sensitivity and stability, we introduce a new class of interpretation methods that adopt techniques from adversarial robustness. Empirical results show that our proposed methods are effective under the new criteria and overcome limitations of gradient-based methods on removal-based criteria. Besides text classification, we also apply interpretation methods and metrics to dependency parsing. Our results shed light on understanding the diverse set of interpretations.

## introduction as complex nlp models are widely deployed in real-world applications, there is an increasing interest in understanding how these models come to certain decisions. as a result, the line of research on interpretation techniques grows rapidly, facilitating a broad range of model analysis, from building user trust on models @xcite to exposing subtle biases @xcite . in this paper, we focus on post-hoc interpretations in nlp. given a trained model and a specific input text, post-hoc interpretations assign an importance score to each token in the input which indicates its contribution to the model output. cur-rent methods in this direction can be roughly divided into three categories: gradient-based methods @xcite ; referencebased methods @xcite ; and perturbation-based methods @xcite . despite the emergence of new techniques, one critical issue is that there is little consensus on how to define and evaluate the faithfulness of these techniques, i.e., whether they reflect the true reasoning process by a model. a widely employed criterion, especially in nlp, is the removal-based criterion @xcite , which removes or only preserves a set of tokens given by interpretations and measures how much the model prediction would change. however, as pointed out in prior work @xcite , the corrupted version of an input produced during evaluations falls out of the distribution that models are trained on, and thus results in an inaccurate measurement of faithfulness. this limitation prevents removal-based metrics from being used as the golden standard for evaluating interpretations. to remedy this, we complement the removal-based criterion with two other criteria, sensitivity and stability, which are overlooked in prior works. sensitivity is based on the notion that models should be more sensitive to perturbations on tokens identified by a faithful explanation. in contrast to the removal-based criterion, which completely removes important tokens, the sensitivity criterion adds small but adversarial perturbations in a local region of the token embedding, and thus preserves the structure of input sentences as well as interactions between context words. this criterion is recently discussed in @xcite in computer vision, while we provide comprehensive analyses on various nlp models and tasks. note that while the removal-based criterion asks the question: if some important tokens did not 'exist', what would happen, the sensitivity criterion asks: if some im-portant tokens were 'changed' adversarially, what would happen. stability assumes that a faithful interpretation should not produce substantially different explanations for two inputs that the model finds similar. there are several attempts to generate such a pair of inputs. the most relevant one is @xcite . however, their method is only applicable to differentiable interpretations. our work proposes a new paradigm based on adversarial word substitution that employs a black-box algorithm to generate a semantically related neighbor of the original input, which is specially designed for nlp and applicable to all interpretations techniques. the above two metrics highlight the connection between interpretability and robustness. experiments show that interpretations which perform well on the removal-based criterion might not do well on the new criteria. motivated by the limitations of existing interpretations and the desiderata of sensitivity, we propose robustness-based methods, based on projected gradient descent (pgd) attacks @xcite and certifying robustness @xcite @xcite . we demonstrate that the new methods achieve top performance under sensitivity and stability. moreover, as a simple improvement to gradient-based methods, our methods avoid the gradient saturation issues of gradient-based methods under the removal-based criterion. another limitation of removal-based metrics emerges when interpreting dependency parsingwhen input tokens are removed, the tree structure is drastically changed and a model might not be able to produce a meaningful parse tree. thus, there are little discussion for dependency parsing interpretations. in this paper, we propose a new paradigm to interpret dependency parsers leveraging prepositional phrase (pp) attachment ambiguity examples. to our best knowledge, this is the first work to study interpretations on dependency parsing. we demonstrate that sensitivity does not change the output tree structure as much as removal-based ones do, and provide analyses for interpretation methods with our paradigm and metrics. our contributions can be summarized as follows. 1. we discuss two overlooked notions of faithfulness in nlp interpretations. our notions emphasize the connection between interpretability and robustness. we systematically evaluate interpretations under we first review the well-established removalbased criterion and emphasize its relation to the two criteria defined in this paper 1) sensitivity, and 2) stability, for which we propose novel paradigms to adapt them to various nlp tasks. removal-based criterion a well-established notion of interpretation faithfulness is that the presence of important tokens should have more meaningful influence on the model's decision than random tokens, quantified by the removal-based criterion. we adopt the comprehensiveness and the sufficiency score in @xcite . the comprehensiveness score measures how much the model performance would drop after the set of "relevant" tokens identified by an interpretation is removed. a higher comprehensiveness score suggests the tokens are more influential to the model output, and thus a more faithful explanation. the sufficiency score measures to what extent the original model performance is maintained when we solely preserve relevant tokens. a lower sufficiency score means less change in the model prediction, and thus a more faithful explanation. see deyoung et al. (2020) for detailed definitions. note that completely removing input tokens produces incomplete texts. large perturbation of this kind lead to several issues as pointed out by prior studies @xcite . ours: sensitivity instead of removing important tokens, the sensitivity criterion adds local but adversarial noise to embedding vectors of the important tokens and measures the magnitude of the noise needed to change the model prediction. this is inspired by the notion that models should be more sensitive to perturbations being added to relevant tokens compared to random or irrelevant tokens. from the adversarial robustness perspective @xcite , this notion implies that by perturbing the most relevant tokens, we can reach the local decision boundary of a model with the minimum perturbation magnitude. given the sequence of relevant tokens r k , sensitivity adds perturbation to its embedding e (r k ) but keeps the remaining token embeddings unchanged. then, it measures the minimal perturbation norm, denoted as ϵ r k , that changes the model prediction for this instance: ours: stability another desired property of faithfulness is that a faithful interpretation should not give substantially different importance orders for two input points that the model finds similar. to construct a pair of similar inputs, we propose to generate contrast examples to the original one by synonym substitutions. a contrast example of x, x, satisfies (1) has at most k different but synonymous tokens with x; (2) the prediction score at x changes less than τ compared to the score at x. the goal of these two conditions is to generate (almost) natural examples where the changes of model outputs are smaller than a threshold τ . given all contrast examples, we search for the one that leads to the largest rank difference d between the importance order for x, m (x) and the alternated order m (x): specifically, we first extract synonyms for each token x i following @xcite . then, in the decreasing order of m (x), we greedily search for a substitution of each token that induces the largest change in m (x) and repeat this process until the model output score changes by more than τ or the pre-defined constraint k is reached. finally, we measure the difference d between two importance ranks using spearman's rank order correlation @xcite . we call this criterion stability. a higher score indicates that the ranks between this input pair are more similar, and thus a more faithful interpretation. note that instead of using the gradient information of interpretation methods to perturb importance ranks like @xcite , our algorithm treats interpretations as black-boxes, which makes it applicable to non-differentiable ones. also, compared to @xcite , who manually construct similar input pairs, our method is a fully automatic one as suggested by their paper. ## interpretations via adversarial robustness techniques experiments indicate that existing methods do not work well with the sensitivity and stability metrics (sec. 4.2). in this section, we define a new class of interpretation methods by adopting techniques in adversarial robustness to remedy this. we first give a brief review of existing interpretation approaches and then introduce our new methods. ## experiments on text classification in this section, we present the results on text classification tasks under the three criteria. we find that the correlation between interpretation faithfulness based on different criteria are relatively low in some cases. results verify the effectiveness of our new methods. ## experiments on structured prediction structured prediction tasks are in the center of nlp applications. however, applying interpretation methods and criteria to these tasks are difficult because 1) the required output is a structure instead of a single score. it is hard to define the contribution of each token to a structured output, and 2) compared to text classification tasks, removing parts of the input like what removal-based criteria do, would cause more drastic changes to model predictions as well as the groundtruth. therefore, existing works often conduct experiments only on binary or multi-class text classification tasks. to remedy these issues, we investigate interpretations for dependency parsing, with an special focus on analyzing how models resolve the pp attachment ambiguity, which avoids interpreting the structured output as a whole. we show that our sensitivity metric is a better metric for dependency parsing as it causes negligible changes to model outputs compared to removal-based metrics. ## related work interpretation methods various post-hoc interpretation methods are proposed to explain the behaviors of black-box models. these methods can be roughly categorized into three classes: gradientbased methods @xcite , which leverage local gradient information; reference-based methods @xcite , which consider the model output difference between the original point and a reference point; and perturbation-based methods @xcite @xcite , which query model outputs on perturbed data. in our work, we propose 0 20 gradient-based robustness-based w/ input gradinp gradient-based robustness-based w/o input vagrad perturbation-based occlusion 0.5 1.0 0 20 pgdinp 0.5 1.0 vapgd 0.5 1.0 lime figure 4: where do interpretations place the disambiguating nouns. the results demonstrate obvious patterns in different categories. the x-axis is the top-k interval. scales in {10%, 20%, . . ., 100%}. the y-axis is the number of examples that an interpretation ranks the disambiguating noun within each top-k interval. new interpretation methods called robustness-based methods, which adopt techniques in the adversarial robustness domain and bridge the gap between the gradient-based and the reference-based methods. evaluating interpretation methods one line of studies explores approaches to evaluate interpretations. several studies propose measurements for faithfulness. a large proportion of them occlude tokens identified as important by interpretations and measure the confidence change of models (deyoung et al., 2020; jain and wallace, 2019; zaidan @xcite . some other works propose to evaluate the faithfulness by checking to what extent they satisfy some desired axioms @xcite @xcite . besides, alvarez-melis and jaakkola (2018); @xcite ; @xcite ; @xcite reveal limitations in interpretation faithfulness through testing the robustness of interpretations. another group of studies measure the plausibility of interpretations, i.e., whether the explanations conform with human judgments @xcite , or assist humans or student models to predict model behaviors on new data @xcite . note that although there exist many hybrid works that evaluate both the faithfulness and the plausibility of interpretations by combining a suite of diagnostic tests @xcite @xcite , @xcite advocate to explicitly distinguish between the two measurements. in our work, we focus on interpretation faithfulness but consider two new metrics. we apply them to the dependency parsing task. also, notice that the stability could be regarded as an automatic input consistency tests suggested by @xcite . ## conclusion in our work, we enhanced the existed definition of interpretation faithfulness by two other notions and proposed corresponding quantitative metrics: sensitivity and stability, for each of them. we studied interpretations under the two notions along with the existed one. we found that interpretations have inconsistent performance regarding different criteria. we proposed a new class of interpretations, motivated by the adversarial robustness techniques, which achieves the best performance under the sensitivity and the stability criteria. we further proposed a novel paradigm to evaluate interpretations on the dependency parsing task, which moves beyond text classification in the literature. our study shed light on understanding the behavior of model interpretations and suggested the community to put more efforts on defining an appropriate evaluation pipeline for interpretation faithfulness.