ex BERT : A V isual A nalysis T ool to E xplore L earned R epresentations in T ransformer M odels (2020)

Abstract: Large Transformer-based language models can route and reshape complex information via their multi-headed attention mechanism. Although the attention never receives explicit supervision, it can exhibit recognizable patterns following linguistic or positional information. Analyzing the learned representations and attentions is paramount to furthering our understanding of the inner workings of these models. However, analyses have to catch up with the rapid release of new models and the growing diversity of investigation techniques. To support analysis for a wide variety of models, we introduce exBERT, a tool to help humans conduct flexible, interactive investigations and formulate hypotheses for the model-internal reasoning process. exBERT provides insights into the meaning of the contextual representations and attention by matching a human-specified input to similar contexts in large annotated datasets. By aggregating the annotations of the matched contexts, exBERT can quickly replicate findings from literature and extend them to previously not analyzed models.

## introduction learned contextualized representations of a neural network can contain meaningful information. uncovering this information plays a vital role in understanding and interpreting the learned structure of neural networks @xcite . one way to identify information is to probe the representations by using them as features in classifiers for linguistic tasks, or by identifying contexts that lead to similar patterns @xcite @xcite . with transformers @xcite overtaking recurrent models as the primary architectures for many nlp tasks, analyzing attention has become another common strategy for interpretabil-ity @xcite . these efforts focus on selecting a model, such as bert @xcite , and exploring the transformer's contextual embeddings and attentions across layers to determine whether and where it learns to represent linguistic features. previous studies have uncovered specific attention heads that learn particular dependencies @xcite . however, once the standard linguistic probing tasks are exhausted, it is challenging to develop new hypotheses to test. toward that end, interactive visualizations provide a successful strategy to develop new insights and strategies. visualization tools can offer concise summaries of useful information and allow interaction with large models. attention visualizations have thus taken significant steps toward these goals of making explorations fast and interactive for the user @xcite . however, interpreting attention patterns without understanding the attended-to embeddings, or relying on attention alone as a faithful explanation, can lead to faulty interpretations @xcite @xcite . to address this challenge, we developed exbert, a tool that combines the advantages of static analyses with a dynamic and intuitive view into both the attentions and internal representations of the underlying model. exbert provides these insights for any user-specified model and corpus by probing whether the representations capture meaningful information. we demonstrate that exbert can replicate insights from the analysis by @xcite and easily extend it to other models. it is open-source, extensible, and compatible with many current transformer architectures, both autoregressive and masked language models. exbert is available at exbert.net. 2 background ## overview exbert focuses on displaying a succinct view of both the attention and the internal representations of each token. figure 1 shows an overview of the tool's two main components. the attention view provides an interactive view of the self-attention of the model, where users can change layers, select heads, and view the aggregated attention. the corpus view presents a user with aggregate statistics that aim to describe and summarize the hidden representations of a currently selected token or set of attention heads. for simplicity, the tool defaults to focus on single-sentence examples. ## discussion in this paper, we introduced an interactive visualization tool, exbert, that can reveal an intelligible structure in the learned representations of transformer models. we demonstrated, through an attention visualization and nearest neighbor searching techniques, that exbert can replicate research that explores what attentions and representations learn and detect biases in text inputs. we acknowledge that exbert is limited compared to more global analyses since it only presents statistics across a small number of neighbors for a single token at a time. these neighbors do not necessarily reveal a head's or an embedding's global behavior. however, exbert can effectively narrow the scope and refine hypotheses through quick testing iterations. these hypotheses about the model behavior can, in a later step, be evaluated by robust statistical tests on a global level. to assist researchers with their model investigations, we host a demo of the tool with multiple models at exbert.net.