I nterpre T : An Interactive Visualization Tool for Interpreting Transformers (2021)

Abstract: With the increasingly widespread use of Transformer-based models for NLU/NLP tasks, there is growing interest in understanding the inner workings of these models, why they are so effective at a wide range of tasks, and how they can be further tuned and improved. To contribute towards this goal of enhanced explainability and comprehension, we present InterpreT, an interactive visualization tool for interpreting Transformer-based models. In addition to providing various mechanisms for investigating general model behaviours, novel contributions made in InterpreT include the ability to track and visualize token embeddings through each layer of a Transformer, highlight distances between certain token embeddings through illustrative plots, and identify task-related functions of attention heads by using new metrics. InterpreT is a task agnostic tool, and its functionalities are demonstrated through the analysis of model behaviours for two disparate tasks: Aspect Based Sentiment Analysis (ABSA) and the Winograd Schema Challenge (WSC).

## introduction in recent years, transformer-based models @xcite such as bert @xcite , gpt-2 @xcite , xlnet @xcite and roberta @xcite have demonstrated state-of-the-art performance in many nlp tasks and have become the gold standard. however, there are many open questions regarding the behavior of these models. phenomena such as why transformers perform well on specific examples but not others, as well as how their internal mechanisms facilitate their ability to generalize to new tasks and settings (or lack therof) are not yet fully understood. observations and insights which help answer these questions will be pivotal in driving the construction of more powerful and robust models. the pursuit of such answers have spurred the development of a wide variety of analytical studies and tools to enable the visualization of information encapsulated in transformer-based models. @xcite , studied the attention mechanisms of a pre-trained bert model to find that certain heads correspond to specific linguistic patterns. @xcite investigated the distribution of phrase-level information throughout the layers of bert using t-sne (van der @xcite . the visualization tools of @xcite and @xcite perform a layerwise analysis of bert's hidden states to understand the internal workings of transformer-based models that are fine-tuned for question-answering tasks. other tools, such as @xcite , focus on visualizations of the attention matrices of pretrained transformer models. in the work of @xcite , the authors introduce an interactive platform for the visualization and interpretation of nlp models. the tool includes, among other capabilities, attention visualizations, embedding space visualizations, and aggregate analysis. other related tools include those by @xcite and @xcite . the increasingly large body of work on the interpretability and evaluation of transformer-based models reveals the growing need for the development of tools and systems to aid in the fine-grained analysis and understanding of these models and their performance on complex language understanding tasks. with this goal in mind, we present interpret 1 , a tool for interpreting transformers. a key contribution of interpret is that it is a single system that enables users to track hidden representations of tokens throughout each layer of a transformer model, as well as visualize and analyze attention head behaviors. similarly to @xcite , interpret enables dynamic point selection, aggregation of attention head statistics, visualization of attention head matrices, and the ability to compare models. novel contributions made in interpret include the ability to track and visualize token embeddings through each layer of a transformer (section 3.2), highlight distances between certain token embeddings through illustrative plots (section 3.6), and identify task-related functions of attention heads by using new metrics (section 3.3). section 4 demonstrates how the new features introduced in interpret can be used to obtain novel insights into the underlying mechanisms used by transformers to tackle diverse tasks such as aspect-based sentiment analysis (absa) and the winograd schema challenge (wsc). more generally, these demonstrations illustrate how such features enable rich, granular analysis of transformer models. ## system design and workflow the system flow consists of two main stages: offline extraction of model specific and task specific information such as targets, predictions, relevant hidden states, and attention matrices (henceforth referred to as "collateral") and running the application itself. during the offline stage, the extracted hidden states are processed using t-sne before being saved to a file. the collateral generated for a specific model and task is independent of collateral from other models and tasks, which enables the user to either run the app to examine a single model or to compare two different models that were evaluated on the same task and data. in this latter case, the collateral files for the two models are linked at runtime. a detailed specification for the collateral, along with the source code used to run interpret can be found in our github. ## use cases the examples presented in this section focus on the analysis of bidirectional encoders using inter-pret, however the system can be applied to generative models or encoder-decoder architectures as well, so long as the appropriate collateral can be generated. further examples of use cases along with instructions on how to use interpret for custom applications is detailed in our github. ## conclusion and future work interpret is a generic system for interpreting transformers, as evident through its suite of tools for understanding general model behaviors and for enabling granular analysis of attention patterns and hidden states for individual examples. the capabilities provided by interpret empower users with new insights into what their models are learning, as illustrated in the visualization of the mit-igation of the "domain gap" for absa and in the novel discovery of the spatial convergence of coreferent terms in wsc. these examples showcase how the fine-grained analysis enabled by in-terpret affords a higher level of insight that is indispensable for interpreting model behavior for complex language understanding tasks. interpret is an ongoing development effort. future work will include support for additional use cases as well as additional analysis and interactivity features, such as the ability to dynamically add and modify examples while the app is running.