Ecco: An Open Source Library for the Explainability of Transformer Language Models (2021)

Abstract: Our understanding of why Transformer-based NLP models have been achieving their recent success lags behind our ability to continue scaling these models. To increase the transparency of Transformer-based language models, we present Ecco – an open-source library for the explainability of Transformer-based NLP models. Ecco provides a set of tools to capture, analyze, visualize, and interactively explore the inner mechanics of these models. This includes (1) gradient-based feature attribution for natural language generation (2) hidden states and their evolution between model layers (3) convenient access and examination tools for neuron activations in the under-explored Feed-Forward Neural Network sublayer of Transformer layers. (4) convenient examination of activation vectors via canonical correlation analysis (CCA), non-negative matrix factorization (NMF), and probing classifiers. We find that syntactic information can be retrieved from BERT’s FFNN representations in levels comparable to those in hidden state representations. More curiously, we find that the model builds up syntactic information in its hidden states even when intermediate FFNNs indicate diminished levels of syntactic information. Ecco is available at https://www.eccox.io/

## introduction the transformer architecture @xcite has been powering many recent advances in nlp. a breakdown of this architecture is provided by alammar (2018) and will help understand this paper's details. pre-trained language models based on the architecture @xcite @xcite @xcite ; brown figure 1 : a set of tools to make the inner workings of transformer language models more transparent. by introducing tools that analyze and visualize input saliency (for natural language generation), hidden states, and neuron activations, we aim to enable researchers to build more intuition about transformer language models. et al., 2020) continue to push the envelope in various tasks in nlp and, more recently, in computer vision @xcite . our understanding of why these models work so well, however, still lags behind these developments. ecco provides tools and interactive explorable explanations foot_2 aiding the examination and intuition of: ecco creates rich, interactive interfaces directly inside jupyter notebooks (ragan-kelley et al., 2014) running on pre-trained models from the hugging face transformers library @xcite . currently it supports gpt2 @xcite , bert @xcite , and roberta @xcite . support for more models and explainability methods is under development and open for community contribution. ## input saliency when a computer vision model classifies a picture as containing a husky, an input saliency map (figure 2 ) can tell us whether the classification was made due to the visual properties of the animal itself or because of the snow in the background @xcite . this is a method of attribution explaining the relationship between a model's output and inputs -helping us detect errors and biases to better understand the system's behavior. multiple methods exist for assigning feature importance scores to the inputs of an nlp model @xcite . instead of assigning scores to pixels, in the nlp domain these methods assign scores to input tokens. the literature is most often concerned with this application for classification tasks rather than natural language generation. ecco enables generating output tokens and then interactively exploring the saliency values for each output token. ## hidden states examination another method to glean information about the inner workings of a language is by examining the hidden states produced by every transformer block. ecco provides multiple methods to examine the hidden states and to visualize how they evolve across the layers of the model. ## neuron activations the feed-forward neural network (ffnn) sublayer is one of the two major components inside a transformer block (in addition to self-attention). it often makes up two-thirds of a transformer block's parameters, thus providing a significant portion of the model's representational capacity. previous work @xcite @xcite @xcite @xcite @xcite has examined neuron firings inside deep neural networks in both the nlp and computer vision domains. ecco makes it easier to examine neuron activations by collecting them and providing tools to analyze them and reduce their dimensionality to extract underlying patterns. ## case study: probing ffnn neuron activations for pos information in this section, we use ecco to examine the representations of bert's feed-forward neural network using probing classifiers. our work is most similar to @xcite . there has been plenty of work on probing bert focused on the hidden states, but none to our knowledge that trained probes to extract token information from the ffnn representation. ## system design ecco is implemented as a python library that provides a wrapper around a pre-trained language model. the wrapper collects the required data from the language model (e.g., neuron activations, hidden states) and makes the needed calculations (e.g., input saliency, nmf dimensionality reduction). the interactive visualizations are built using web technologies manipulated through d3.js @xcite . ecco is built on top of open source libraries including scikit-learn @xcite , matplotlib @xcite , numpy @xcite , pytorch @xcite and transformers @xcite . canonical correlation analysis is calculated using the code opensourced 4 by the authors @xcite @xcite . ## limitations ecco's input saliency feature is currently only supported for gpt2-based models, while neuron ac-4 https://github.com/google/svcca tivation collection and dimensionality reduction are supported for gpt2 in addition to bert and roberta. we echo the sentiment of leavitt and morcos (2020) that visualization has a role in building intuitions, but that researchers are encouraged to use that as a starting point towards building testable and falsifiable hypotheses of model interpretability. ## conclusion as language models proliferate, more tools are needed to aid debugging models, explain their behavior, and build intuitions about their innermechanics. ecco is one such tool combining ease of use, visual interactive explorables, and multiple model explainability methods. ecco is open-source software foot_3 and contributions are welcome.