LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models (2024)

Abstract: We present the LM Transparency Tool (LM-TT), an open-source interactive toolkit for analyzing the internal workings of Transformer-based language models. Differently from previously existing tools that focus on isolated parts of the decision-making process, our framework is designed to make the entire prediction process transparent, and allows tracing back model behavior from the top-layer representation to very fine-grained parts of the model. Specifically, it (i) shows the important part of the whole input-to-output information flow, (ii) allows attributing any changes done by a model block to individual attention heads and feed-forward neurons, (iii) allows interpreting the functions of those heads or neurons. A crucial part of this pipeline is showing the importance of specific model components at each step. As a result, we are able to look at the roles of model components only in cases where they are important for a prediction. Since knowing which components should be inspected is key for analyzing large models where the number of these components is extremely high, we believe our tool will greatly support the interpretability community both in research settings and in practical applications.

## introduction recent advances in natural language processing led to remarkable capabilities of the transformer language models, especially with scale @xcite @xcite @xcite @xcite . this, along with the wide adoption of such models in high-stakes settings, makes understanding the internal workings of these models vital from the safety, reliability and trustworthiness perspectives. existing tools for analyzing sequence models' predictions enable users to compute input tokens attribution scores, read token promotions performed by different model components, or analyze textual patterns responsible for the activation of model's neurons @xcite @xcite @xcite @xcite . however, these focus only on specific parts of the decision-making process and none of them is designed to make the entire prediction process transparent. in contrast, we introduce lm transparency tool, a framework that allows tracing back model behavior to very fine-grained model parts. one of the key advantages of our pipeline is the ability to look only at those model components that were relevant for a selected prediction. indeed, e.g. syntactic attention heads @xcite , induction heads @xcite , knowledge neurons @xcite , etc. perform their function only in specific cases and are "dormant" otherwise -therefore, looking at them makes sense only for those certain examples. to make this possible, our tool first shows the information flow routes introduced by @xcite : this is a subset of intermediate token representations and model components that together form the most important part of the entire inputto-output processing. then, the tool further allows (i) attributing any changes done by those important model blocks to individual attention heads and feed-forward neurons, as well as (ii) interpreting the functions of those heads and neurons. importantly, lm-tt is highly efficient: due to relying on @xcite , it is 100 times faster than typical patching-based alternatives @xcite . overall, the lm transparency tool: ## user interface and functionality inside transformer language models, each representation evolves from the current input token embedding foot_1 to the final representation used to predict the next token. this evolution happens through additive updates coming from attention and feedforward blocks. the resulting stack of same-token representations is usually referred to as "residual stream" @xcite , and the overall computation inside the model can be viewed as a se-quence of residual streams connected through layer blocks. formally, we can see it as a graph where nodes correspond to token representations and edges correspond to operations inside the model (attention heads, feed-forward layers, etc.). our tool visualizes the "important" part of this graph, importances of model components at varying levels of granularity (individual heads and neurons), as well as an interpretation of representations and updates coming from model components. ## system design and components our application is a web-based toolkit, offering easy and interactive access that is cross-platform compatible. this approach allows users to utilize and share the tool remotely, emphasizing convenience and flexibility. ## related work existing tools for analyzing sequence models' predictions include lm-debugger @xcite , visit @xcite , ecco (alammar, 2021), lit @xcite , inseq @xcite , and captum @xcite . these tools enable users to compute input tokens attribution scores, read token promotions performed by different model components via logit lens, or analyze textual patterns responsible for the activation of the model's neurons. however, these are not able to extract a relevant part of model computations and indicate component importances. to identify parts of the model relevant for some task, a recent trend in mechanistic interpretability is to rely on causal interventions on the computational graph of the model, aka "activation patching" @xcite @xcite @xcite @xcite . usually, this process involves the following steps: 1) selecting a dataset and metric, 2) manually creating contrastive examples, 3) searching for important edges in the graph via activation patching. the latter requires running a forward pass per each patched element and uses many patches to explain a single prediction. although recent approaches aim to automate some parts of this workflow @xcite , the entire process requires a large human effort and involves significant computational costs: this imposes constraints on the tool development and limits its applicability. differently, lm-tt relies on a recent method by @xcite which refuses from the patching constraints by relying on attribution to define the importances. furthermore, lm-tt incorporates additional functionalities such as showing fine-grained component importances, logit lens analysis at different levels of granularity, and attention visualization not only via attention weights but also via contributions. this enables users to gain a more comprehensive understanding of the functions executed by each component. ## conclusions we release the lm transparency tool, an opensource toolkit for analyzing transformer-based language models that allows tracing back model behavior to specific parts of the model. specifically, it (i) shows the important part of the whole input-tooutput information flow, (ii) allows attributing any changes done by a model block to individual attention heads and feed-forward neurons, (iii) allows interpreting the functions of those heads or neurons. notably, due to the nature of the underlying method, our tool reduces the number of components to be analyzed by highlighting model components that were relevant to the prediction. this greatly simplifies the study of large language models, with potentially thousands of attention heads and hundreds of thousands of neurons to look at. moreover, the ui accelerates the inspection process, unlike other frameworks that lack this feature. this assists researchers and practitioners in efficiently generating hypotheses regarding the behavior of the model.