The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020)

Abstract: We present the Language Interpretability Tool (LIT), an open-source platform for visualization and understanding of NLP models. We focus on core questions about model behavior: Why did my model make this prediction? When does it perform poorly? What happens under a controlled change in the input? LIT integrates local explanations, aggregate analysis, and counterfactual generation into a streamlined, browser-based interface to enable rapid exploration and error analysis. We include case studies for a diverse set of workflows, including exploring counterfactuals for sentiment analysis, measuring gender bias in coreference systems, and exploring local behavior in text generation. LIT supports a wide range of models—including classification, seq2seq, and structured prediction—and is highly extensible through a declarative, framework-agnostic API. LIT is under active development, with code and full documentation available at https://github.com/pair-code/lit .

## introduction advances in modeling have brought unprecedented performance on many nlp tasks (e.g. @xcite , but many questions remain about the behavior of these models under domain shift @xcite and adversarial settings @xcite , and for their tendencies to behave according to social biases @xcite or shallow heuristics (e.g. @xcite . for any new model, one might want to know: what kind of examples does my model perform poorly on? why did my model make this prediction? and critically, does my model behave consistently if i change things like textual style, verb tense, or pronoun gender? despite the recent explosion of work on model understanding and evaluation (e.g. @xcite @xcite , there is no "silver bullet" for analysis. practitioners must often experiment with many techniques, looking at local explanations, aggregate metrics, and counterfactual variations of the input to build a full understanding of model behavior. existing tools can assist with this process, but many come with limitations: offline tools such as tfma @xcite can provide only aggregate metrics, interactive frontends (e.g. @xcite may focus on single-datapoint explanation, and more integrated tools (e.g. @xcite @xcite often work with only a narrow range of models. switching between tools or adapting a new method from research code can take days of work, distracting from the real task of error analysis. an ideal workflow would be seamless and interactive: users should see the data, what the model does with it, and why, so they can quickly test hypotheses and build understanding. with this in mind, we introduce the language interpretability tool (lit), a toolkit and browserbased user interface (ui) for nlp model understanding. lit supports local explanationsincluding salience maps, attention, and rich visualizations of model predictions-as well as aggregate analysis-including metrics, embedding spaces, and flexible slicing-and allows users to seamlessly hop between them to test local hypotheses and validate them over a dataset. lit provides first-class support for counterfactual generation: new datapoints can be added on the fly, and their effect on the model visualized immediately. sideby-side comparison allows for two models, or two datapoints, to be visualized simultaneously. we recognize that research workflows are constantly evolving, and designed lit along the following principles: ## user interface and functionality lit has a browser-based ui comprised of modules (figure 1 ) which contain controls and visualizations for specific tasks ( at the most basic level, lit works as a simple demo server: one can enter text, press a button, and see the model's predictions. but by loading an evaluation set, allowing dynamic datapoint generation, and an array of interactive visualizations, metrics, and modules that respond to user input, lit supports a much richer set of user journeys: j1 -explore the dataset. users can interactively explore datasets using different criteria across modules like the data table and the embeddings module (similar to smilkov et al. (2016)), in which a pca or umap (mcinnes et al., 2018) projection can be rotated, zoomed, and panned to explore clusters and global structures (figure 1 -top left). ## case studies sentiment analysis. how well does a sentiment classifier handle negation? we load the development set of the stanford sentiment treebank (sst; @xcite , and use the search function in lit's data table (j1, j2) to find the 56 datapoints containing the word "not". looking at the metrics table (j6), we find that surprisingly, our bert model @xcite gets 100% of these correct! but we might want to know if this is truly robust. with lit, we can select individual datapoints and look for explanations (j3). for example, take the negative review, "it's not the ultimate depression-era gangster movie.". as shown in figure 2 , salience maps suggest that "not" and "ultimate" are important to the prediction. we can verify this by creating modified inputs, using lit's datapoint editor (j4). removing "not" gets a strongly positive prediction from "it's the ultimate depression-era gangster movie.", while replacing "ultimate" to get "it's not the worst depression-era gangster movie." elicits a mildly positive score from our model. gender bias in coreference. does a system encode gendered associations, which might lead to incorrect predictions? we load a coreference model trained on ontonotes @xcite , and load the winogender @xcite dataset into lit for evaluation. each winogender example has a pronoun and two candidate referents, one a occupation term like ("technician") and one an "other participant" (like "customer"). our model predicts coreference probabilities for each candidate. we can explore the model's sensitivity to pronouns by comparing two examples side-by-side (see figure 3 (a).) we can see how commonly the model makes similar errors by paging through the dataset (j1), or by selecting specific slices of interest. for example, we can use the scalar plot module (j2) (figure 3 (b)) to select datapoints where the occupation term is associated with a high proportion of male or female workers, according to the u.s. bureau of labor statistics (bls; @xcite . in the metrics table (j6), we can slice this selection by pronoun type and by the true referent. on the set of male-dominated occupations (< 25% female by bls), we see the model performs well when the ground-truth agrees with the stereotypee.g. when the answer is the occupation term, male pronouns are correctly resolved 83% of the time, compared to female pronouns only 37.5% of the time (figure 3 (c)) . to find out how t5 arrived at this prediction, we utilize the "similarity searcher" component through the counterfactual generator tab (figure 4 (d) ). this performs a fast approximate nearest-neighbor lookup @xcite ) from a pre-built index over the training corpus, using embeddings from the t5 decoder. with one click, we can retrieve 25 nearest neighbors and add them to the lit ui for inspection (as in figure a .1). we see that the words "captain" and "former" appear 34 and 16 times in these examples-along with 3 occurrences of "replaced by" (figure 4 (e))-suggesting a strong prior toward our erroneous phrase. ## system design and components the lit ui is written in typescript, and communicates with a python backend that hosts models, datasets, counterfactual generators, and other interpretation components. lit is agnostic to modeling frameworks; data is exchanged using numpy arrays and json, and components are integrated through a declarative "spec" system (section 4.4) that minimizes cross-dependencies and encourages modularity. a more detailed design schematic is given in the appendix, figure a.2. ## related work a number of tools exist for interactive analysis of trained ml models. many are general-purpose, such as the what-if tool @xcite , captum @xcite , manifold @xcite , or interpretml @xcite , while others focus on specific applications like fairness, including fairvis @xcite and fairsight @xcite . and some provide rich support for counterfactual analysis, either within-dataset (what-if tool) or dynamically generated as in @xcite . for nlp, a number of tools exist for specific model classes, such as rnns @xcite , transformers @xcite , or text generation @xcite . more generally, allennlp interpret (wallace et al., 2019) introduces a modular framework for interpretability components, focused on singledatapoint explanations and integrated tightly with the allennlp @xcite framework. while many components exist in other tools, lit aims to integrate local explanations, aggregate analysis, and counterfactual generation into a single tool. in this, it is most similar to errudite @xcite , which provides an integrated ui for nlp error analysis, including a custom dsl for text transformations and the ability to evaluate over a corpus. however, lit is explicitly designed for flexibility: we support a broad range of workflows and provide a modular design for extension with new tasks, visualizations, and generation techniques. limitations lit is an evaluation tool, and as such is not directly useful for training-time monitoring. as lit is built to be interactive, it does not scale to large datasets as well as offline tools such as tfma @xcite . (currently, the lit ui can handle about 10,000 examples at once.) because lit is framework-agnostic, it does not have the deep model integration of tools such as allennlp interpret @xcite or captum @xcite . this makes many things simpler and more portable, but also requires more code for techniques like integrated gradients @xcite that need to directly manipulate parts of the model. ## conclusion and roadmap lit provides an integrated ui and a suite of components for visualizing and exploring the behavior of nlp models. it enables interactive analysis both at the single-datapoint level and over a whole dataset, with first-class support for counterfactual generation and evaluation. lit supports a diverse range of workflows, from explaining individual predictions to disaggregated analysis to probing for bias through counterfactuals. lit also supports a range of model types and techniques out of the box, and is designed for extensibility through simple, framework-agnostic apis. lit is under active development by a small team. planned upcoming additions include new counterfactual generation plug-ins, additional metrics and visualizations for sequence and structured output types, and a greater ability to customize the ui for different applications. lit is open-source under an apache 2.0 license, and we welcome contributions from the community at https://github.com/pair-code/lit . (a) (b) figure a.5: confusion matrix (a) and side-by-side comparison of predictions and salience maps (b) on two sentiment classifiers. in model comparison mode, the confusion matrix can compare two models, and clicking an off-diagonal cell with select examples where the two models make different predictions. in (b) we see one such example, where the model in the second row ("sst 1") predicts incorrectly, even though gradient-based salience show both models focusing on the same tokens.