Transformer-specific Interpretability (2024)

Abstract: Transformers have emerged as dominant play- ers in various scientific fields, especially NLP. However, their inner workings, like many other neural networks, remain opaque. In spite of the widespread use of model-agnostic interpretability techniques, including gradient-based and occlusion-based, their shortcomings are becoming increasingly apparent for Transformer interpretation, making the field of interpretability more demanding today. In this tutorial, we will present Transformer-specific interpretability methods, a new trending approach, that make use of specific features of the Transformer architecture and are deemed more promising for understanding Transformer-based models. We start by discussing the potential pitfalls and misleading results model-agnostic approaches may produce when interpreting Transformers. Next, we discuss Transformer-specific methods, including those designed to quantify context- mixing interactions among all input pairs (as the fundamental property of the Transformer architecture) and those that combine causal methods with low-level Transformer analysis to identify particular subnetworks within a model that are responsible for specific tasks. By the end of the tutorial, we hope participants will understand the advantages (as well as current limitations) of Transformer-specific interpretability methods, along with how these can be applied to their own research.

## tutorial description with transformers @xcite demonstrating exceptional performance across every domain they venture into such as language, speech, vision, and music, the necessity to understand their underlying mechanisms has become more crucial than ever before. many model-agnostic interpretability techniques that were commonly used for earlier generations of deep learning architectures, such as probing, occlusion-based, and feature attribution methods, were swiftly adapted for use with the transformer architecture. however, these approaches demonstrate notable disagreement with each other and a lack of stability when moving from one domain to another @xcite @xcite . their effectiveness in drawing reliable conclusions has therefore been an ongoing matter of debate @xcite . recently, a game-changing trend has emerged: the development of analysis methods that are precisely tailored to the model architecture of transformers, built upon their underlying mathematical foundations. these methods make use of specific features of transformers, including their layered structure (layers, heads, tokens), the division of labor between the attention mechanism, feed-forward layers, and residual streams. these techniques span from those aimed at measuring token-to-token interactions (known as context mixing, @xcite @xcite @xcite , to others striving to reverse engineer the model decision and decompose it into understandable pieces (known as mechanistic interpretability, @xcite . this tutorial focuses on transformer-specific interpretability methods. we will first briefly review the internal structure of the transformer architecture to establish our notations. next, we will explain why it is necessary to design methods tailored to the model architecture, exposing the limitations of model-agnostic approaches when applied to transformer analysis using practical examples. subsequently, we will introduce transformerspecific techniques, delving into their mathematics, and categorizing them according to their purposes, using experimental results across a number of domains, such as text, speech, and music, as well as across several languages. our tutorial will conclude with a discussion on current limitations in interpretability and promising future directions. ## tutorial type the tutorial will be cutting-edge, covering the latest research advancements in the interpretability of transformers, which serve as the backbone architecture of modern nlp systems. the only acl tutorials similar to ours are "interpretability and analysis in neural nlp" @xcite and "fine-grained interpretation and causation analysis in deep nlp models" @xcite , held at acl 2020 and naacl 2021, respectively. both focused on general modelagnostic interpretability techniques. our tutorial, however, will question the effectiveness of those general-purpose analysis methods and mark the next chapter: a transition from model-agnostic approaches to transformer-specific methods. ## target audience given the widespread use of transformers across various applications in both text and speech, we expect that our audience will be not only folks engaged in interpretability but also those from various tracks within the computational linguistics community who have not kept up with the recent advancements within interpretability research. in fact, we have been frequently asked at *acl conferences and our industry meetings, particularly by individuals outside of the interpretability track, seeking guidance on the most effective interpretability techniques to employ in their projects for noninterpretability purposes, such as training monitoring, model compression, or model tuning. in terms of expected prerequisite background, we expect audience members to be familiar with the basic concepts of transformer models. for the jupyter notebooks that will be covered, we expect experience with pytorch and the transformers library. ## outline of tutorial structure the tutorial will consist of 30 minute slots of lectures and interactive seminars for which we will provide jupyter notebooks. a small part of the tutorial will be focused on interpretability techniques from the organisers (e.g. @xcite @xcite , but the majority of the work discussed will be work from other labs to provide an honest and broad overview of the current state of interpretability research in nlp. in addition to the key papers mentioned in section 4, we would recommend attendees that are interested in gaining a broader understanding of general interpretability techniques to explore the following survey papers: @xcite @xcite 6 special requirements there are no special technical requirements, other than standard conference equipment (computer, screen, and projector). if participants wish to participate in the interactive parts, they should bring their laptops. ## diversity our tutorial focuses on transformer-specific interpretability across several domains, including text, speech, music, (and vision, to some extent). as transformers have gained widespread adoption within the cl community, we anticipate engaging a diverse and extensive audience. to ensure diversity, we have both professors and phd students on our instructor team. hosein mohebbi is a phd candidate at tilburg university. he is part of the indeep consortium project, doing research on the interpretability of deep neural models for both text and speech. during his master's, his research revolved around the interpretation of pre-trained language models and the utilization of interpretability techniques to accelerate their inference time. his research has been published in leading nlp venues such as acl, eacl, emnlp, and blackboxnlp, where he also regularly serves as a reviewer. he is also one of the organizers of blackboxnlp 2023-2024, a workshop focusing on analyzing and interpreting neural networks for nlp. jaap jumelet is a phd candidate at the institute for logic, language and computation at the university of amsterdam. his research focuses on gaining an understanding of how neural models are able to build up hierarchical representations of their input, by leveraging hypotheses from (psycho-)linguistics. his research has been published at leading nlp venues, including tacl, acl, and conll. he is a co-organiser for black-boxnlp in 2023-2024. he has been involved in numerous courses in the ai master of the university of amsterdam, all with a focus on nlp and interpretability. michael hanna is a phd candidate at the university of amsterdam, as part of the institute for logic, language and computation. his research focuses on understanding the abilities of pre-trained language models, and linking these behaviors to lowlevel mechanisms using causal methods. his work has been published in leading interpretability and nlp venues such as neurips, emnlp, and eacl. he previously designed and led a workshop on mechanistic interpretability as part of the university of amsterdam's artificial intelligence masters program. afra alishahi is an associate professor at the department of cognitive science and artificial intelligence at tilburg university, netherlands. her main research interests are developing computational models of human language, studying the emergence of linguistic structure in grounded models of language learning, and developing tools and techniques for analyzing linguistic representations in neural models of language. she has served as program chair for conll and as ac and sac for many recent cl conferences, and is one of the founders of the blackboxnlp workshops. she has acted as acl tutorial co-chair and taught tutorials at acl and esslii; most recently she offered a tutorial on interpretability of linguistic knowledge in neural language models as part of lectures on computational linguistics in pisa, italy. willem zuidema is associate professor of nlp, explainable ai and cognitive modelling at the university of amsterdam. he has published widely in nlp, ai and cognitive science venues, including tacl, jair, acl, emnlp and neurips. since 2016, many of his publications have focused on interpretability in ai. he has taught many undergraduate and graduate courses (including interpretability and explainability in ai in amsterdams's msc ai, 2022, 2023), and two courses at graduate summerschools ( @xcite . he leads a project on interpretability that involves 5 universities @xcite . he has served on many program committees, including acl, naacl, emnlp, blackboxnlp, and helped organize workshops and conferences; in 2016, he was tutorial co-chair for acl.