LM -Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models (2022)

Abstract: The opaque nature and unexplained behavior of transformer-based language models (LMs) have spurred a wide interest in interpreting their predictions. However, current interpretation methods mostly focus on probing models from outside, executing behavioral tests, and analyzing salience input features, while the internal prediction construction process is largely not understood. In this work, we introduce LM-Debugger, an interactive debugger tool for transformer-based LMs, which provides a fine-grained interpretation of the model’s internal prediction process, as well as a powerful framework for intervening in LM behavior. For its backbone, LM-Debugger relies on a recent method that interprets the inner token representations and their updates by the feed-forward layers in the vocabulary space. We demonstrate the utility of LM-Debugger for single-prediction debugging, by inspecting the internal disambiguation process done by GPT2. Moreover, we show how easily LM-Debugger allows to shift model behavior in a direction of the user’s choice, by identifying a few vectors in the network and inducing effective interventions to the prediction process. We release LM-Debugger as an open-source tool and a demo over GPT2 models.

## introduction transformer-based language models (lms) are the backbone of modern nlp models @xcite , but their internal prediction construction process is opaque. this is problematic to endusers that do not understand why the model makes specific predictions, as well as for developers who wish to debug or fix model behaviour. recent work @xcite suggested that the construction process of lm predictions can be viewed as a sequence of updates to the token representation. specifically, figure 1 : illustration of the main capabilities of lm-debugger. our tool interprets dominant changes in the output distribution induced by the feed-forward layers across the network (self-attention layers are not shown), and enables configuring interventions for shifting the prediction in directions of the user's choice. @xcite showed that updates by the feedforward network (ffn) layers, one of the building blocks of transformers @xcite , can be decomposed into weighted collections of subupdates, each induced by a ffn parameter vector, that can be interpreted in the vocabulary space. in this work, we make a step towards lm transparency by employing this interpretation approach to create lm-debugger, a powerful tool for inspection and intervention in transformer lm predictions. lm-debugger provides three main capabilities for single-prediction debugging and model analysis (illustrated in figure 1 ). first, for a given input (e.g. "my wife is working as a"), it interprets the model's prediction at each layer in the network, and the major changes applied to it by ffn layers. this is done by projecting the token representa-tion before and after the ffn update as well as the major ffn sub-updates at any layer to the output vocabulary. second, it allows intervening in the prediction by changing the weights of specific subupdates, e.g. increasing (decreasing) a sub-update that promotes music-related (teaching-related) concepts, which results in a modified output. last, for a given lm, lm-debugger interprets all the ffn parameter vectors across the network and creates a search index over the tokens they promote. this allows an input-independent analysis of the concepts encoded by the model's ffn layers, and enables configuring general and effective interventions. we demonstrate the utility of lm-debugger for two general use-cases. in the context of prediction debugging, we use the fine-grained tracing of lm-debugger to inspect the internal disambiguation process performed by the model. furthermore, we demonstrate how our tool can be used to configure a few powerful interventions that effectively control different aspects in text generation. we release lm-debugger as an open-source tool at https://github.com/mega002/lm-debugger and host a demo of gpt2 @xcite at https://lm-debugger.apps.allenai.org . 1 this to increase the transparency of transformer lms and facilitate research in analyzing and controlling nlp models. ## underlying interpretation method lm-debugger establishes a framework for interpreting a token's representation and updates applied to it at each layer in the network. this framework builds upon recent findings by @xcite , who viewed the token representation as a changing distribution over the output vocabulary, and the output from each ffn layer as a collection of weighted sub-updates to that distribution, which are often interpretable to humans. we next elaborate on the findings we rely on at this work. consider a transformer lm with l layers and an embedding matrix @xmath0, over a vocabulary v. let @xmath1, ..., w t s.t. ∀@xmath2, ..., t : w @xmath3, then at each layer ℓ = 1, ..., l, the hidden representation x ℓ i of the i-th token is being processed and updated by a ffn layer through a residual connection @xcite token representation as a distribution over the output vocabulary. the token representation before (x ℓ i ) and after (x ℓ i ) the ffn update at any layer ℓ is interpreted by projecting it to the vocabulary space and converting it to a distribution: the final model output is defined by @xmath4. the ffn output as a weighted collection of sub-updates. each ffn layer is defined with two parameter matrices k ℓ , v ℓ ∈ r dm×d , where d m is the intermediate hidden dimension, and a non-linearity function f (bias terms are omitted): geva et al. (2022) interpreted the ffn output by (a) decomposing it into sub-updates, each induced by a single ffn parameter vector, and (b) projecting each sub-update to the vocabulary space. formally, eq. 1 can be decomposed as: ## lm-debugger lm-debugger leverages both static and dynamic analysis of transformer ffn layers and the updates they induce to the output distribution for debugging and intervention in lm predictions. these capabilities are provided in two main views, which we describe next. ## debugging lm predictions by tracing ffn updates in this section, we demonstrate the utility of lm-debugger for interpreting model behaviour upon a given example. as an instructive example, we will consider the case of sense disambiguation. when generating text, lms often need to perform sense disambiguation and decide on one plausible continuation. for example, the word "for" in the input "the book is for" has two plausible senses of purpose (e.g. "reading") and person (e.g. "him") @xcite . we will now inspect the prediction by gpt2 @xcite and track the internal sense disambiguation process for this example. to this end, we enter the input in the prediction view and click trace, which provides a full trace of the prediction across layers. to person sense. until layer 11 (out of 24), the toptokens in the output distribution are mostly related to sale/example purposes. starting from layer 12, the prediction slowly shifts to revolve about the audience of the book, e.g. anyone and ages, until layer 18 where sale is eliminated from the top position. in the last layers, tokens become more specific, e.g. beginners and adults. to examine the major updates through which the prediction has formed, we can click on specific sub-updates in the trace to inspect the topscoring tokens in their projections. we observe that in early layers, tokens are often related to purpose sense (e.g. instance in l2d1855 and buyers in l12d659), in intermediate layers tokens are a mix of both senses (readers in l16d3026 and preschool in l17d2454, and sale/free in l16d1662), and mostly person sense in the last layers (users in l18d685, people in l20d3643, and those in l21d2007). ## configuring effective interventions for controlled text generation beyond interpretability, lm-debugger enables to intervene in lm predictions. we show this by finding value vectors that promote specific concepts and applying simple and effective interventions. ## implementation details the prediction view is implemented as a react web application with a backend flask server that runs an api for executing models from the transformers library by huggingface @xcite . the exploration view is a streamlit web application, which (a) sends user search queries to an elasticsearch index with the top tokens of all vector value projections, and (b) visualize clusters of value vectors created with the scikit-learn package @xcite . our current implementation supports any gpt2 model from huggingface, and other auto-regressive models can be plugged-in with only a few local modifications (e.g. translating the relevant layer names). more details and instructions for how to deploy and run lm-debugger are provided at https://github.com/mega002/ lm-debugger . ## related work interpreting single-predictions and the general behavior of lms is a growing research area that attracted immense attention in recent years @xcite . lm-debugger is a the first tool to interpret and intervene in the prediction construction process of transformer-based lms based on ffn updates. existing interpretation and analysis frameworks mostly rely on methods for behavioral analysis @xcite ) by probing models with adversarial @xcite or counterfactual examples @xcite , input saliency methods that assign importance scores to input features @xcite , and analysis of the attention layers @xcite . more related to lm-debugger, other tools analyze patterns in neuron activations @xcite @xcite . unlike these methods, we focus on interpreting the model parameters and on intervening in their contribution to the model's prediction. the functionality of lm-debugger is mostly related to tools that trace hidden representations across layers. similarly to lm-debugger, alammar (2021); nostalgebraist (2020) interpret the token representation in terms of the output vocabulary. we take this one step further and interpret the ffn updates to the representation, allowing to observe not only the evolution of the representation but also the factors that induce changes in it. our intervention in ffn sub-updates relates to recent methods for locating and editing knowledge in the ffn layers of lms @xcite . different from these methods, lm-debugger aims to provide a comprehensive and fine-grained interpretation of the prediction construction process across the layers. ## conclusion we introduce lm-debugger, a debugger tool for transformer-based lms, and the first tool to analyze the ffn updates to the token representations across layers. lm-debugger provides a fine-grained interpretation of single-predictions, as well as a powerful framework for intervention in lm predictions.