Interpreting Predictions of NLP Models (2020)

Abstract: Although neural NLP models are highly expressive and empirically successful, they also systematically fail in counterintuitive ways and are opaque in their decision-making process. This tutorial will provide a background on interpretation techniques, i.e., methods for explaining the predictions of NLP models. We will first situate example-specific interpretations in the context of other ways to understand models (e.g., probing, dataset analyses). Next, we will present a thorough study of example-specific interpretations, including saliency maps, input perturbations (e.g., LIME, input reduction), adversarial attacks, and influence functions. Alongside these descriptions, we will walk through source code that creates and visualizes interpretations for a diverse set of NLP tasks. Finally, we will discuss open problems in the field, e.g., evaluating, extending, and improving interpretation methods.

## tutorial description neural models have become the de-facto standard tool for nlp tasks. these models are becoming increasingly powerful-recent work shows that large neural models substantially improve accuracy on a wide range of downstream tasks @xcite . however, today's models still make egregious errors: they reinforce racial biases @xcite , fail in counterintuitive ways @xcite , and often solve tasks using simple surface-level patterns @xcite . these model insufficiencies are exacerbated by the inability to understand why models made the predictions they do. interpretation methods seek to fill this void. in particular, example-specific interpretations provide post-hoc explanations for indi-vidual model predictions. these explanations come in various forms, e.g., attributing the importance of the input features through saliency maps @xcite , perturbing the inputs and observing the model's response @xcite , or locating a model's local decision boundary @xcite . this tutorial will provide an introduction to the various types of example-specific interpretations. we will present the technical details of existing methods, including saliency maps, adversarial attacks, input perturbations, influence functions, and other methods. we will cover how these interpretations are applied to various tasks and input-output formats, e.g., text classification using lstms, masked language modeling using bert @xcite , and text generation using gpt-2 @xcite . for each task, we will walk through example use cases of interpretations: highlighting model weaknesses @xcite , increasing/decreasing user trust @xcite , and understanding hard-to-formalize criteria such as bias, safety, and fairness @xcite . alongside the tutorial, we will present source code implementations of various interpretation methods using allennlp interpret @xcite . ## details and prerequisites the tutorial will be of the cutting-edge type. the tutorial slides and the accompanying code is available online at https://www.ericswallace. com/interpretability . prerequisites attendees should have a basic understanding of different tasks in nlp such as text classification, sequence tagging, and reading comprehension (predicting spans in a passage). attendees should also have a basic understanding of neural network methods for nlp, including: reading list doshi-velez and kim (2017) provide a great overview and motivation for interpretability research. lipton (2018) and @xcite discuss some of the challenges of defining and evaluating interpretability. @xcite help demonstrate the fragility of nlp models. lime @xcite and saliency maps @xcite are now standard interpretations. @xcite provides example nlp interpretations (interested readers can inspect their code). ## tutorial outline the tutorial will present three hours of content with a thirty-minute break. motivation this section will discuss why we care about interpretability. it will paint a landscape of today's neural models, describe how models are brittle and behave counterintuitively, and explain how interpretations can open the "black box" of machine learning. introduction to interpretations this section will situate example-specific interpretations in the context of other methods. we will discuss: example-specific interpretations this section will introduce example-specific interpretations in more detail. we will discuss the challenges and approaches to evaluating such interpretations. we will also cover the critiques and shortcomings of using attention as explanations @xcite . we will then explain why we focus on gradient-based methods: they are model-agnostic, easy to compute, and (largely) faithful to a model's behavior. understanding what parts of an input led to a prediction this section will discuss: