Interpreting Language Models with Contrastive Explanations (2022)

Abstract: Model interpretability methods are often used to explain NLP model decisions on tasks such as text classification, where the output space is relatively small. However, when applied to language generation, where the output space often consists of tens of thousands of tokens, these methods are unable to provide informative explanations. Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics.Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding.To disentangle the different decisions in language modeling, we focus on explaining language models contrastively: we look for salient input tokens that explain why the model predicted one token instead of another. We demonstrate that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena, and that they significantly improve contrastive model simulatability for human observers. We also identify groups of contrastive decisions where the model uses similar evidence, and we are able to characterize what input tokens models use during various language generation decisions.

## introduction despite their success across a wide swath of natural language processing (nlp) tasks, neural language models (lms) are often used as black boxes: how they make certain predictions remains obscure @xcite . this is in part due to the high complexity of the lm task itself, as well as that of the model architectures used to solve it. we argue that this is also due to the fact that interpretability methods commonly used in nlp, such input: can you stop the dog from output: barking 1. why did the model predict "barking"? can you stop the dog from 2. why did the model predict "barking" instead of "crying"? can you stop the dog from 3. why did the model predict "barking" instead of "walking"? can you stop the dog from as gradient-based saliency maps @xcite , are not as informative for lm predictions compared to other tasks like text classification. for example, to explain why an lm predicts "barking" given "can you stop the dog from ____", we demonstrate in experiments that the input token preceding the prediction is often marked as the most influential token to the prediction ( to better explain lm decisions, we propose interpreting lms with contrastive explanations @xcite . contrastive explanations aim to identify causal factors that lead the model to produce one output instead of another output. we believe that contrastive explanations are especially useful to handle the complexity and the large output space of language modeling. in in this paper, we first extend three interpretability methods to compute contrastive explanations ( §3). we then perform a battery of experiments aimed at examining to what extent these contrastive explanations are superior to their non-contrastive counterparts from various perspectives: ## contrastive explanations for language models in this section, we describe how we extend three existing input saliency methods to the contrastive setting. these methods can also be easily adapted to tasks beyond language modeling, such as machine translation (appendix a). ## do contrastive explanations identify linguistically appropriate evidence? first, we ask whether contrastive explanations are quantifiably better than non-contrastive explanations in identifying evidence that we believe a priori should be important to the lm decision. in order to do so, we develop a methodology in which we specify certain types of evidence that indicate how to make particular types of linguistic distinctions, and measure how well each variety of explanation method uncovers this specified evidence. ## do contrastive explanations help users predict lm behavior? to further evaluate the quality of different explanation methods, we next describe methodology and experiments to measure to what extent explanations can improve the ability of users to predict the output of the model, namely model simulatability @xcite . ## conclusion and future work in this work, we interpreted language model decisions using contrastive explanations by extending three existing input saliency methods to the contrastive setting. we also proposed three new methods to evaluate and explore the quality of contrastive explanations: an alignment evaluation to verify whether explanations capture linguistically appropriate evidence, a user evaluation to measure model simulatability of explanations, and a clustering-based aggregate analysis to investigate model properties using contrastive explanations. we find that contrastive explanations are better aligned to known evidence related to major grammatical phenomena than their non-contrastive counterparts. moreover, contrastive explanations allow better contrastive simulatability of models for users. from there, we studied what kinds of decisions require similar evidence and we used contrastive explanations to characterize how models make certain linguistic distinctions. overall, contrastive explanations give a more intuitive and fine-grained interpretation of language models. future work could explore the application of these contrastive explanations to other machine learning models and tasks, extending other interpretability methods to the contrastive setting, as well as using what we learn about models through contrastive explanations to improve them.