Attribution Patching Outperforms Automated Circuit Discovery (2024)

Abstract: Automated interpretability research has recently attracted attention as a potential research direction that could scale explanations of neural network behavior to large models. Existing automated circuit discovery work applies activation patching to identify subnetworks responsible for solving specific tasks (circuits). In this work, we show that a simple method based on attribution patching outperforms all existing methods while requiring just two forward passes and a backward pass. We apply a linear approximation to activation patching to estimate the importance of each edge in the computational subgraph. Using this approximation, we prune the least important edges of the network. We survey the performance and limitations of this method, finding that averaged over all tasks our method has greater AUC from circuit recovery than other methods.

## introduction mechanistic interpretability is a subfield of ai interpretability that focuses on attributing model behaviors to its components, thus reverse engineering the network @xcite . this field aims to identify subnetworks (circuits) within the model which are responsible for solving specific tasks @xcite . prior attempts at finding circuits in language models have led to finding networks of attention heads and multi-layer perceptrons (mlps) that partially or fully explain model behaviors at tasks such as indirect object identification, modular arithmetic, completion of docstrings, and predicting successive dates @xcite @xcite . however, almost all previous work has been limited to relatively small models since manually applying mechanistic interpretability methods has not currently scaled to end-to-end circuits in larger models @xcite . ## related work automated circuit discovery refers to finding the important subgraph of models' computational graphs for performance on particular tasks @xcite . existing algorithms include efficient heuristics @xcite and gradientdescent based methods @xcite . acdc is related to pruning @xcite and other compression techniques @xcite , but differs in how the compressed networks are reflective of the circuits that the model uses to compute outputs to certain tasks and the goal of acdc is not to speed up forward passes (all techniques studied in this work use slow forward passes). concurrent work has further established attribution-based circuit discovery @xcite @xcite . activation patching is a technique for analyzing the role of individual components in a model. it involves targeted manipulations of activations during a forward pass (further explained in section 3.1). previous works applied this technique under various names, such as interchange interventions @xcite , causal mediation analysis @xcite and causal tracing @xcite . we adapt the terminology used by @xcite . transformer circuits. our work builds upon the framework for understanding transformers for interpretability as introduced by @xcite . individual attention heads and mlps (collectively called nodes) read and write information to a central communication channel, also called the residual stream. in these terms we can examine dependencies of nodes with the output of earlier nodes, i.e we can measure the effect of attention heads in layer 0 on the attention heads in layer 2. in the following, we view these dependencies as edges between nodes, building on existing work using this perspective @xcite @xcite . ## edge attribution patching we present edge attribution patching (eap) as a technique to identify relevant model components for solving a specific task. in the following, we view language models as directed, acyclic graphs. in these terms, we aim to find small subgraphs that retain good performance on narrow tasks. we determine the importance of a specific edge through targeted manipulation of activations during a forward pass. we compare two approaches, attribution patching and activation patching, in order to motivate eap. ## limitations we introduced edge attribution patching as an approximation to activation patching. however, we found that edge activation patching outperformed acdc, a technique based on activation patching (section 4). in this section, we investigate whether attribution patching's success is due to extremely accurate approximations (in section 5.1 we find that the answer is no), and whether there is any further use for acdc (in section 5.2 we find that the answer is yes). we use the docstring task as a case study due to the small model size used. ## conclusion we provide evidence that edge attribution patching (eap) outperforms acdc in identifying circuits while being substantially faster to compute. this result is surprising, as eap is an approximation for activation patching, the method applied by acdc. however, running acdc on the prepruned subnetwork found by eap can improve the identification of relevant edges. therefore, we suggest future circuit discovery experiments to run eap first and then apply acdc.