Evaluating Saliency Methods for Neural Language Models (2021)

Abstract: Saliency methods are widely used to interpret neural network predictions, but different variants of saliency methods often disagree even on the interpretations of the same prediction made by the same model. In these cases, how do we identify when are these interpretations trustworthy enough to be used in analyses? To address this question, we conduct a comprehensive and quantitative evaluation of saliency methods on a fundamental category of NLP models: neural language models. We evaluate the quality of prediction interpretations from two perspectives that each represents a desirable property of these interpretations: plausibility and faithfulness. Our evaluation is conducted on four different datasets constructed from the existing human annotation of syntactic and semantic agreements, on both sentence-level and document-level. Through our evaluation, we identified various ways saliency methods could yield interpretations of low quality. We recommend that future work deploying such methods to neural language models should carefully validate their interpretations before drawing insights.

## introduction while neural network models for natural language processing (nlp) have recently become popular, a general complaint is that their internal decision mechanisms are hard to understand. to alleviate this problem, recent work has deployed interpretation methods on top of the neural network models. among them, there is a category of interpretation methods called saliency method that is especially widely adopted @xcite @xcite @xcite . at a very high level, these methods assign an importance score to each feature in the input feature set f , regarding a specific prediction y made by a neural network model m . such feature importance scores can hopefully shed light on the neural network models' internal decision mechanism. v u.s. companies wanting to expand in europe sg u.s. companies wanting to expand in europe ig u.s. companies wanting to expand in europe while analyzing saliency interpretations uncovers useful insights for their respective task of interest, different saliency methods often give different interpretations even when the internal decision mechanism remains the same (with f , y and m held constant), as exemplified in in this paper, we address this problem by building a comprehensive quantitative benchmark to evaluate saliency methods. our benchmark focuses on a fundamental category of nlp models: neural language models. following the concepts proposed by @xcite , our benchmark evaluates the credibility of saliency interpretations from two aspects: plausibility and faithfulness. in short, plausibility measures how much these interpretations align with basic human intuitions about the model decision mechanism, while faithfulness measures how consistent the interpretations are regarding perturbations that are supposed to preserve the same model decision mechanism on either the input feature f or the model m . with these concepts in mind, our main contribution is materializing these tests' procedure in the context of neural language modeling and building four test sets from existing linguistic annotations to conduct these tests. our study covering sotalevel models on three different network architectures reveals that saliency methods' applicability depends heavily on specific choices of saliency methods, model architectures, and model configurations. we suggest that future work deploying these methods to nlp models should carefully validate their interpretations before drawing conclusions. this paper is organized as follows: section 2 briefly introduces saliency methods; section 3 describes the plausibility and faithfulness tests in our evaluation; section 4 presents the datasets we built for the evaluation; section 5 presents our experiment setup and results; section 6 discusses some limitations and implications of the evaluation; section 7 concludes the paper. ## saliency the notion of saliency discussed in this paper is a category of neural network interpretation methods that interpret a specific prediction y made by a neural network model m , by assigning a distribution of importance ψ(f ) over the input feature set f of the original neural network model. the most basic and widely used method is to assign importance by the gradient @xcite , which we refer to as vanilla gradient method (v). for each @xmath0, ψ(x) = ∂py ∂x , while p y is the score of prediction y generated by m . we also examine two improved version of gradient-based saliency: smoothgrad (sg) @xcite and integrated gradients (ig) @xcite . smoothgrad reduces the noise in vanilla gradient-based scores by constructing several corrupted instances of the original input by adding gaussian noise, followed by averaging the scores. integrated gradients computes feature importance by computing a line integral of the vanilla saliency from a "baseline" point f 0 to the input f in the feature space. we refer the readers to the cited papers for details of these saliency methods. there is a slight complication in the meaning of f when applying these methods in the context of nlp: all the methods above will generate one importance score for each dimension of the word embedding, but most applications of saliency to nlp want a word-level importance score. hence, we need composition schemes to combine scores over word embedding dimensions into a single score for each word. in the rest of this paper, we assume the "features" in the feature set f are input words to the language model, and word-level importance scores are composed using the gradient • input scheme @xcite ). foot_0 ## evaluation paradigm in this section, we first introduce the notion of plausibility and faithfulness in the context of neural network interpretations (following jacovi and goldberg (2020)), and then, respectively, introduce the test we adopt to evaluate them. ## data 3 following the formulation in section 3, we constructed four novel datasets for our benchmark, as exemplified in for number agreement, our synthetic dataset is constructed from selected sections of syneval, a targeted language model evaluation dataset from @xcite , where the verbs and the subjects could be easily induced with heuristics. we only use the most challenging sections where strongly interceding attractors are involved. our natural dataset for this task is filtered from penn treebank @xcite , including training, development, and test. we choose ptb because it offers not only human-annotated postags necessary for benchmark construction but also dependent subjects of verbs for further analysis. for gender agreement, our synthetic dataset comes from the unambiguous winobias coreference resolution dataset used in @xcite , and we only use the 1000-example subset where there is respectively one male and one female antecedent. because this dataset is intentionally designed such that most humans will find pronouns of either gender equally likely to follow the prefix, no such pronoun gender is considered to be "correct". hence, without loss of generality, we assign the female pronoun to be the expected case. 4 our natural dataset for this task is filtered from conll-2012 shared task dataset for coreference resolution @xcite including training, develop-ment, and test). the prefix of each test example covers a document-level context, which usually spans several hundred words. plausibility test for number agreement, the cue set c is the set of all nouns that have the same morphological number as the verb. in contrast, the attractor set a is the set of all nouns with a different morphological number. for gender agreement, the cue set c is the set of all nouns with the same gender as the pronoun, while the attractor set a is the set of all nouns with a different gender. ## discussion most existing work on evaluating saliency methods focuses only on computer vision models @xcite @xcite @xcite . in the context of nlp, @xcite is the first work to conduct such evaluations for nlp and the only prior work that conducts such evaluations for neural language models but has several limitations as we have already pointed out in section 3. arras et al. (2019); atanasova et al. (2020); hao (2020) conducted similar evaluations based on specifically designed diagnostic toy tasks and/or text classification, while bastings and filippova (2020) casted doubt on whether these conclusions could be generalized to sequence generation tasks. li et al. (2020) evaluated various interpretation methods for neural machine translation models by building proxy models on only the top-k important input words as determined by the interpretation methods, but such evaluation requires generating interpretations for a large training set and hence is intractable for even mildly computationallyexpensive methods such as smoothgrad and integrated gradients. on a slightly different line, deyoung et al. ( 2020 ) built a benchmark to evaluate a specific category of nlp models that generate rationales during predictions, which is a different path towards building explainable nlp models. our evaluation is not without its limitations. the first limitation, inherited from earlier work by @xcite , is that our plausibility test only concerns the words in cue/attractor sets rather than other words in the input prefix. such limitation is inevitable because the annotations from which we build our ground-truth interpretations are only concerned with a specific lexical agreement. this limitation can be mitigated by combining plausibility tests with faithfulness tests, which concern all the input prefix words. the second limitation is that the test sets used in these benchmarks need to be constructed in a caseto-case manner, according to the chosen lexical agreements and the input perturbations. while it is hard to create plausibility test sets without human interference, future work could explore automatic input consistency tests by utilizing adversarial input generation techniques in nlp @xcite @xcite . it should also be noted that while our work focuses on evaluating a specific category of interpretation methods for neural language models, our evaluation paradigm can be easily extended to evaluating other interpretation methods such as attention mechanism, and with other sequence models such as masked language models (e.g., bert). we would also like to extend these evaluations beyond english datasets, especially to languages with richer morphological inflections. ## conclusion we conduct a quantitative evaluation of saliency methods on neural language models based on the perspective of plausibility and faithfulness. our evaluation shows that a model interpretation can either fail due to a lack of plausibility or faithfulness, and the interpretations are trustworthy only when they do well with both tests. we also noticed that the performance of saliency interpretations are generally sensitive to even minor model configuration changes. hence, trends of interpretation quality on a specific model configuration should not be over-generalized to other configurations.