Copy Suppression: Comprehensively Understanding a Motif in Language Model Attention Heads (2024)

Abstract: We present the copy suppression motif: an algorithm implemented by attention heads in large language models that reduces loss.If i) language model components in earlier layers predict a certain token, ii) this token appears earlier in the context and iii) later attention heads in the model suppress prediction of the token, then this is copy suppression. To show the importance of copy suppression, we focus on reverse-engineering attention head 10.7 (L10H7) in GPT-2 Small. This head suppresses naive copying behavior which improves overall model calibration, which explains why multiple prior works studying certain narrow tasks found negative heads that systematically favored the wrong answer. We uncover the mechanism that the negative heads use for copy suppression with weights-based evidence and are able to explain 76.9% of the impact of L10H7 in GPT-2 Small, by this motif alone.To the best of our knowledge, this is the most comprehensive description of the complete role of a component in a language model to date. One major effect of copy suppression is its role in self-repair. Self-repair refers to how ablating crucial model components results in downstream neural network parts compensating for this ablation. Copy suppression leads to self-repair: if an initial overconfident copier is ablated, then there is nothing to suppress. We show that self-repair is implemented by several mechanisms, one of which is copy suppression, which explains 39% of the behavior in a narrow task. Interactive visualizations of the copy suppression phenomena may be seen at our web app https://copy-suppression.streamlit.app/.

## introduction mechanistic interpretability research aims to reverse engineer neural networks into the algorithms that network components implement @xcite . a central focus of this research effort is the search for explanations for the behavior of model components, such as circuits @xcite , neurons @xcite @xcite and attention heads @xcite . however, difficulties in understanding machine learning models has often limited the breadth of these explanations or the complexity of the components involved @xcite . in this work we explain how "negative heads" (which include 'negative name mover heads' from @xcite and 'anti-induction heads' from @xcite ) function on the natural language training distribution in gpt-2 small. previous work found that negative heads systematically write against the correct completion on narrow datasets, and we explain these observations as instances of copy suppression. copy suppression accounts for a majority of the head's behavior and reduces the model's overall loss. to the best of our knowledge, our explanation is the most comprehensive account of the function of a component in a large language model (section 5 reviews related literature). we define negative heads as attention heads which primarily reduce the model's confidence in particular token completions. we show that the main role of negative heads in gpt-2 small is copy suppression (figure 1 ), which is defined by three steps: early naive prediction of ' love'. ' love' ? the main role of head l10h7 in gpt-2 small is copy suppression. ## how negative heads copy suppress in this section, we show that copy suppression explains 76.9% of l10h7's behavior on openweb-text. to reach this conclusion, we perform the following set of experiments: ## applications of copy suppression in this section, we explore some different applications of copy suppression. first, we connect it to the previously observed phenomena of antiinduction, while also providing evidence that it occurs in several different sizes and classes of models. second, we discuss the phenomena of self-repair, which refers to how neural network components can sometimes compensate for perturbations made to earlier components. the final attention heads involved in this circuit are called name mover heads; they copy the io to the model's output. iii) we can measure the extent to which ioi occurs by measuring the logit difference metric, which is equal to the difference between the ' john' and ' mary' logits in the above example. copy suppression heads like l10h7 usually come after the name mover heads. they detect the io prediction, attend back to the first instance of the io, and suppress it (but not enough to change the model's prediction). this is a relatively clean domain in which to study copy suppression. ## conclusion in summary, in this work we firstly introduced copy suppression, a description of the main role of an attention head across gpt-2 small's training distribution. secondly, we applied weights-based arguments using qk and ov circuits to mechanistically verify our hypotheses about copy suppression. finally, we showed how our comprehensive analysis has applications to open problems in ablation-based interpretability (section 4). two limitations of our work include our understanding of the query inputs to self-repair heads, and the transferability of our results to different models. in both section 3.2 and 4 we found that copy suppression and self-repair rely on more than simply unembedding directions, and we hope that future work can fully explain this observation. further, while we show that some of our insights generalize to large models (section 4.1 and a), we don't have a mechanistic understanding of copy suppression in these cases. despite this, our work shows that it is possible to explain llm components across broad distributions with a high level of detail. for this reason, we think that our insights will be useful for future interpretability research.