Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Models (2024)

Abstract: While transformer models exhibit strong capabilities on linguistic tasks, their complex architectures make them difficult to interpret. Recent work has aimed to reverse engineer transformer models into human-readable representations called circuits that implement algorithmic functions. We extend this research by analyzing and comparing circuits for similar sequence continuation tasks, which include increasing sequences of Arabic numerals, number words, and months. By applying circuit interpretability analysis, we identify a key sub-circuit in both GPT-2 Small and Llama-2-7B responsible for detecting sequence members and for predicting the next member in a sequence. Our analysis reveals that semantically related sequences rely on shared circuit subgraphs with analogous roles. Additionally, we show that this sub-circuit has effects on various math-related prompts, such as on intervaled circuits, Spanish number word and months continuation, and natural language word problems. Overall, documenting shared computational structures enables better model behavior predictions, identification of errors, and safer editing procedures. This mechanistic understanding of transformers is a critical step towards building more robust, aligned, and interpretable language models.

## introduction transformer-based large language models (llms) like gpt-4 have demonstrated impressive natural language capabilities across a variety of tasks @xcite . however, these models largely remain black boxes due to their complex, densely connected architectures. understanding how these models work is important for ensuring safe and aligned deployment, especially as they are already being used in high-impact real-world settings @xcite @xcite . several researchers argue that the ability to interpret ai decisions is essential for the safe implemen-* equal contribution tation of sophisticated machine learning technologies @xcite . previous studies show that ai interpretability is vital for ai safety, for catching deception, and for addressing misalignment @xcite . mechanistic interpretability, a sub-field of interpretability, aims to reverse engineer models into understandable components (such as neurons or attention heads) @xcite . by uncovering underlying mechanisms, researchers can better predict model behaviors @xcite and understand emergent phenomena @xcite @xcite . to analyze computations within models, a recent approach has been to find circuits, which are subgraphs of neural networks that represent algorithmic tasks @xcite . recent work in interpretability has uncovered transformer circuits that implement simple linguistic tasks, such as identifying indirect objects in sentences @xcite . however, only a few studies have focused on the existence of shared circuits @xcite , in which circuits utilize the same subcircuits for similar tasks. identifying shared circuits assists in aligning ai via methods such as model editing @xcite , which precisely targets problematic areas for more efficient re-alignment without erroneously altering healthy components. documenting the existence of shared circuits enables safer, more predictable model editing with fewer risks, as editing a circuit may affect another if they share sub-circuits @xcite . therefore, interpretability enables safer alignment by understanding adverse effect prevention. while models use the same components for different tasks, such as when there are far more tasks/features than neurons @xcite , our focus is on locating components which are shared due to similar, re-usable functionality, and not for vastly different functionalities. our work tackles the hypothesis that llms may re-use circuits across analogous tasks that share common abstractions. for instance, similar sequence continuation tasks, such as number words ("one two three") and months ("jan feb mar"), can be analogously mapped to one another via the natural number abstraction (for example, one and jan are mapped to 1). as these tasks share a common abstraction, llms may have learned to efficiently re-use components that utilize shared patterns. understanding how llms re-use components based on commonalities can shed light on how they represent and associate semantic concepts with one another @xcite . not only would this enhance understanding of how llms actually perceive information, but it may have potential applications in transfer learning @xcite . thus, in this paper, we demonstrate the existence of shared circuits for similar sequence continuation tasks, as the similarity across these tasks is clear, allowing us to cleanly pinpoint functionality. our key finding is that there exist shared sub-circuits between similar tasks in gpt-2 small @xcite and llama-2-7b @xcite , where the shared components have the same functionality across tasks. the main contributions of this work are foot_0 : multi-layer perceptron. each attention layer output is passed to a multi-layer perceptron (mlp). the mlps in transformers are generally made of two linear layers, parameterized by weight matrices w 1 and w 2 , with a relu activation function in between. residual stream. attention head and mlp outputs are added to the residual stream, from which components read from and write to. components in non-adjacent layers are able to interact via indirect effects from the additivity of the residual stream @xcite . transformer circuit discovery. in transformer circuits, evidence has shown that in general, mlps associate input information with features @xcite , while attention heads move information @xcite . prior work has employed causal interventions to locate circuits for specific tasks @xcite , such as for the indirect object identification (ioi) task, in which the goal is to complete sentences with the correct subject @xcite . one type of causal intervention is called knockout, which, after a model has processed a dataset, replaces (or ablates) the activations of certain components with other values, such as activations sampled from another distribution. the sampled activations may come from a corrupted dataset, which outputs the wrong answer, but resembles the same dataset without the information of interest (for example "1 2 3" becomes "8 1 4" to preserve information about numbers, while removing sequence information). after running again, if the ablated nodes do not change model performance much, they are deemed as not part of a circuit of interest. another type of causal intervention is activation patching, which takes the corrupted dataset as input, and then restores the activations at a certain component with the original activations to observe how much that restored component recovers the original performance. the automatic circuit dis-covery (acdc) technique employs iterative patching automatically find circuit graphs for given tasks @xcite ; however, this technique only seeks to automate finding the connectivity of circuit graphs, and not their functionality interpretation. interpretability of sequential tasks. in concurrent work, successor heads were found by @xcite to increment tokens with a natural ordering. however, @xcite only analyze the output of the successor head, rather than looking at the output of a model, and does not look for circuits for sequences with more than one member. as shown in our paper, a successor head is not enough to evaluate a sequence. additionally, we study shared interactions between components. thus, we build on @xcite by showing how successor heads interact with the model as a whole. @xcite found circuits for "greater-than" sequence tasks (for example completing the sentence, "the war lasted from the year 1732 to the year 17", with any valid two-digit end @xmath0. greater-than tasks allow any year greater than a value to be valid, which differs from our sequence completion tasks that only have one valid answer. the authors noted that "similar tasks had similar... circuits", but compared across number tasks, and not across non-number tasks such as months. in our work, we study tasks that are more dissimilar. related to sequence continuation, @xcite and ( quirke and barez, 2023 ) discover circuits for arithmetic computation. shared circuits for similar tasks. locating shared circuits is a relatively new research topic. previous studies have noted that circuits for the induction task @xcite are found in circuits for the ioi task. recently, @xcite discovered shared circuits for the ioi task and colored objects task (where the aim is to identify the correct color for an object given a set of colored objects). the authors utilized an interven-tion experiment to improve the colored objects circuit by modifying subject inhibition heads of the ioi circuits to inhibit the wrong color answers. in our paper, we focus on tasks which are much more similar and map to a common abstraction. while the ioi task and colored objects task both share similar sub-tasks such as "inhibiting tokens", the focus of our paper is on enhancing our understanding of how llms represent analogous concepts by discovering sub-circuits which represent common abstractions, instead of just shared sub-tasks. ## methodology circuit discovery process. our approach begins by applying iterative pruning to obtain connectivities for circuits of similar tasks. then, we employ methods to deduce component functionalities shared by similar tasks. we approach circuit discovery in two types of stages: foot_1 1. connectivity discovery consists of applying causal mediation analysis techniques for identifying important connections for varying component granualarity levels (eg. residual stream, attention head, mlp, neuron). ## explaining shared component functionalities in gpt-2 small in this section we analyze main results, and include extended results in appendix §g. ## interpreting sequence continutation in llama-2-7b we apply iterative pruning using datasets of sequences (without non-sequence words in between) to llama-2-7b, which has 1024 attention heads and 32 mlps, to find important attention heads 5.25, 16.0, and 20.17 7 . specifically, attention head 5.25 is found to be a sequence detection head like gpt-2 small's attention head 4.4, and attention head 20.17 is found to be a successor head like gpt-2 small's attention head 9.1. given that we can find analogues of the sub-circuit's components described in §5.1 from one model to another, we show that this sub-circuit type appears in more than one model 8 . appendix c details the setup of these experiments. important components. as the full circuits' component sets are between 80 to 100 nodes and are too large to include in this paper, we only discuss important components. unlike in gpt-2 small, for llama-2-7b we find most of the mlps to be necessary to be able to perform sequence continuation. attention patterns. as shown in figure 5 , 6 successor heads were found concurrently via independent discovery by @xcite . 7 for llama-2-7b, we define a component as important is their ablation from a task's circuit causes an average performance drop of at least 12% for all tasks 8 future studies can examine if this appears in even more models. llama-2-7b's attention head 5.25 appears to be a sequence detection head similar to gpt-2 small's attention head 4.4. moreover, it is not just a similar member detection head like gpt-2 small's attention head 1.5; when sequence members are out-oforder, the component does not detect these members. but when the members of any of the 3 tasks are in-order, it is able to detect them. likewise, in figure 6 , llama-2-7b's attention head 20.17 appears to be a successor head similar to gpt-2 small's attention head 9.1. we discuss attention head 16.0 in appendix 6. ov scores. attention head 20.17 has a higher than average successor head score of 33.3%, compared to the average score of 0.46%. it is one of only twelve attention heads (out of 1024) which detects the "next" token of a sequence. ## ablating sequence continuation components on math-related prompts we examine how ablating sequence continuation components affects the generated output for llama-2-7b on various math-related prompts. this demonstrates how these circuits are not just specific to a narrow range of sequence continuation tasks, but have effects on a wide range of mathematical reasoning tasks. we observe the ablation effects on the following prompt types: (1) additive sequences for @xmath1, and (2) arithmetic. in on intervaled sequences. as shown in on arithmetic. ablating sequence continuation head sets destroy the model's ability to perform addition and subtraction. this suggests that sequence continuation, addition, and subtraction tasks share circuits. as shown in ## conclusion understanding the inner workings of neural networks is essential for fostering alignment and safety. in this paper, we identify that across similar sequence continuation tasks, there exist shared subcircuits that exhibit similar functionality. specifically, we find sequence token detection heads and components associated with next sequence outputs. the aim of this work is to advance our understanding of how transformers leverage shared computational structures across similar tasks. by locating and comparing these circuits across models, we hope to gain insight into semantic representations of abstract concepts, which may provide evidence for universally converging hierarchical associations across models @xcite @xcite .