Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions (2020)

Abstract: Modern deep learning models for NLP are notoriously opaque. This has motivated the development of methods for interpreting such models, e.g., via gradient-based saliency maps or the visualization of attention weights. Such approaches aim to provide explanations for a particular model prediction by highlighting important words in the corresponding input text. While this might be useful for tasks where decisions are explicitly influenced by individual tokens in the input, we suspect that such highlighting is not suitable for tasks where model decisions should be driven by more complex reasoning. In this work, we investigate the use of influence functions for NLP, providing an alternative approach to interpreting neural text classifiers. Influence functions explain the decisions of a model by identifying influential training examples. Despite the promise of this approach, influence functions have not yet been extensively evaluated in the context of NLP, a gap addressed by this work. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. As suspected, we find that influence functions are particularly useful for natural language inference, a task in which ‘saliency maps’ may not have clear interpretation. Furthermore, we develop a new quantitative measure based on influence functions that can reveal artifacts in training data.

## introduction deep learning models have become increasingly complex, and unfortunately their inscrutability has grown in tandem with their predictive power @xcite . this has motivated efforts to design example-specific approaches to interpreting black box nlp model predictions, i.e., 1 code is available at https://github.com/ xhan77/influence-function-analysis . while widely used and potentially useful for some lexicon-driven tasks (e.g., sentiment analysis), we argue that by virtue of being constrained to highlighting individual input tokens, saliency maps will necessarily fail to explain predictions in more complex semantic tasks involving reasoning, such as natural language inference (nli), where fine-grained interactions between multiple words or spans are key @xcite . moreover, saliency maps are inherently limited as a model debugging tool; they may tell us which inputs the model found to be important, but not why. to address these shortcomings, we investigate the use of what lipton (2018) referred to as explanation by example. instead of constructing importance scores over the input texts on which the model makes predictions, such methods rank training examples by their influence on the model's prediction for the test input @xcite @xcite . specifically, we are interested in the use of influence functions @xcite , which are in a sense inherently 'faithful' in that they reveal the training examples most responsible for particular predictions. these do not require any modifications to the model structure. this paper presents a series of experiments intended to evaluate the potential utility of influence functions for better understanding modern neural nlp models. in this context, our contributions in-… figure 1: a sentiment analysis example interpreted by gradient-based saliency maps (left) and influence functions (right). note that this example is classified incorrectly by the model. positive saliency tokens and highly influential examples may suggest why the model makes the wrong decision; tokens and examples with negative saliency or influence scores may decrease the model's confidence in making that decision. ## explaining black-box model predictions machine learning models in nlp depend on two factors when making predictions: the input text and the model parameters. prior attempts to interpret opaque nlp models have typically focused on the input text. our work investigates the complementary approach of interpreting predictions by analyzing the influence of examples in training data. saliency maps aim to provide interpretability by highlighting parts of the input text, whereas influence functions seek clues in the model parameters, eventually locating interpretations within the training examples that influenced these estimates. in this section we explain the two interpretation methods in detail. foot_0 ## experimental setup we are interested in analyzing and comparing the two interpretation approaches (gradient-based attributions and influence functions) on relatively shallow, lexicon-driven tasks and on more complex, reasoning-driven tasks. we focus on sentiment analysis and natural language inference (nli) as illustrative examples of these properties, respectively. both models are implemented on top of bert encoders @xcite . in particular we use bert-base, with the first 8 of the 12 layers frozen, only fine-tuning the last 4 transformer layers and the final projection layer. 3 it is worth noting that influence functions are guaranteed to be accurate only when the model is strictly convex (i.e., its hessian is positive definite and thus invertible) and is trained to convergence. however, deep neural models like bert are not convex, and one often performs early stopping during training. we refer to @xcite for details on how influence functions can nonetheless provide good approximations. to summarize briefly: for the non-convexity issue, we add an appropriate 'damping' term to the model's hessian so that it is positive definite and invertible. concerning non-convergence: the approximated influence may still be interpretable as the true influence of each training example plus a constant offset that does not depend on the individual examples. aside from this theory, we also perform a sanity check in §4 to show that influence functions can be applied to bert in practice on the two tasks that we consider. ## interpreting nli predictions with influence functions are saliency-based explanations useful for nli? gradient-based saliency maps are faithful by construction, but this does not mean that they will highlight input tokens that humans find plausible or useful. we hypothesize that highlighting individual input tokens as important is likely most useful for 'shallow' classification tasks like sentiment analysis, and less so for more complex reasoning tasks such as nli. to contrast the types of explanations these methods offer in this context, we show explanations for a prediction made for a typical example in hans in the form of a saliency map and influential examples in test input p: the :::::: manager was ::::::::: encouraged by the secretary. h: the secretary encouraged the manager. [entail] p: i don't know if i was in heaven or hell, said lillian carter, the president's mother, after a visit. h: the president's mother visited. most ::::::: opposing training examples p: h'm, yes, that might be, said john. h: yes, that might be the case, said john. the relationship classification decision in nli is often made through an interaction between multiple words or spans. therefore, an importance measure on each individual token might not give us much useful insight into model prediction. though influence functions also do not explicitly tell us which latent interactions between words or spans informed the model prediction, we can test whether the model is relying on some hypothesized artifacts in a post-hoc way by looking at patterns in the influential training examples. in ## related work interpreting nlp model predictions by constructing importance scores over the input tokens is a widely adopted approach @xcite . since the appearance and rise of attentionbased models, many work naturally inspect attention scores and interpret with them. however, we are aware of the recent discussion over whether attention is a kind of faithful explanation @xcite . using vanilla attention as interpretation could be more problematic in now ubiquitous deep transformerbased models, such as we use here. gradient-based saliency maps are locally 'faithful' by construction. other than the vanilla gradients @xcite and the "gradient × input" method @xcite we use in this work, there are some variants that aim to make gradient-based attributions robust to potential noise in the input @xcite . we also note that feng et al. (2018) find that gradient-based methods sometimes yield counter-intuitive results when iterative input reductions are performed. other token-level interpretations include input perturbation @xcite which measure a token's importance by the effect of removing it, and lime @xcite which can explain any model's decision by fitting a sparse linear model to the local region of the input example. the main focus of this work is the applicability of influence functions @xcite as an interpretation method in nlp tasks, and to highlight the possibility of using this to surface annotation artifacts. other methods that can trace the model's decision back into the training examples include deep weighted averaging classifiers @xcite , which make decisions based on the labels of training examples that are most similar to the test input by some distance metrics. @xcite use kernel-based deep architectures that project test inputs to a space determined by a group of sampled training examples and make explanations through the most activated training instances. while these methods can similarly identify the 'influential' training examples, they require special designs or modifications to the model and could sacrifice the model's performance and generalizability. other general methods for model interpretability include adversarial-attack approaches that identify that part of input texts can lead to drastically different model decisions when minimally edited @xcite , probing approaches that test internal representations of models for certain tasks and properties @xcite , and generative approaches that make the model jointly extract or generate natural language explanations to support predictions @xcite @xcite . specific to the nli task, gururangan et al. (2018) recognize and define some possible artifacts within nli annotations. @xcite create a diagnostic dataset that we use in this work and suggest that the model could be exploiting some artifacts in training data based on its poor performance on the diagnostic set. beyond nli, the negative influence of artifacts in data was explored in other text classification tasks @xcite @xcite , focusing on approaches to adversarial learning to demote the artifacts. ## conclusion we compared two complementary interpretation methods-gradient-based saliency maps and influence functions-in two text classification tasks: sentiment analysis and nli. we first validated the reliability of influence functions when used with deep transformer-based models. we found that in a lexicon-driven sentiment analysis task, saliency maps and influence functions are largely consistent with each other. they are not consistent, however, on the task of nli. we posit that influence functions may be a more suitable approach to interpreting models for such relatively complex natural language 'understanding' tasks (while simpler attribution methods like gradients may be sufficient for tasks like sentiment analysis). we introduced a new potential use of influence functions: revealing and quantifying the effect of data artifacts on model predictions, which have been shown to be very common in nli. future work might explore how rankings induced over training instances by influence functions can be systematically analyzed in a stand-alone manner (rather than in comparison with interpretations from other methods), and how these might be used to improve model performance. finally, we are interested in exploring how these types of explanations are actually interpreted by users, and whether providing them actually establishes trust in predictive systems.