RAVEL : Evaluating Interpretability Methods on Disentangling Language Model Representations (2024)

Abstract: Individual neurons participate in the representation of multiple high-level concepts. To what extent can different interpretability methods successfully disentangle these roles? To help address this question, we introduce RAVEL (Resolving Attribute-Value Entanglements in Language Models), a dataset that enables tightly controlled, quantitative comparisons between a variety of existing interpretability methods. We use the resulting conceptual framework to define the new method of Multi-task Distributed Alignment Search (MDAS), which allows us to find distributed representations satisfying multiple causal criteria. With Llama2-7B as the target language model, MDAS achieves state-of-the-art results on RAVEL, demonstrating the importance of going beyond neuron-level analyses to identify features distributed across activations. We release our benchmark at https://github.com/explanare/ravel.

## introduction a central goal of interpretability is to localize an abstract concept to a component of a deep learning model that is used during inference. however, this is not as simple as identifying a neuron for each concept, because neurons are polysemantic -they represent multiple high-level concepts @xcite @xcite @xcite @xcite . several recent interpretability works @xcite @xcite tackle this problem using a featurizer that disentangles the activations of polysemantic neurons by mapping to a space of monosemantic features that each represent a distinct concept. intuitively, these methods should have a significant advantage over approaches that identify concepts with sets of neurons. however, these methods have not been benchmarked. to facilitate these method comparisons, we introduce a diagnostic benchmark, ravel (resolving attribute-value entanglements in language models). ravel evaluates interpretability methods on their ability to localize and disentangle the attributes of different types of entities encoded as text inputs to language models (lms). for example, the entity type "city" has instances such as "paris" or "tokyo", which each have attributes for "continent", namely "europe" and "asia". an interpretability method must localize this attribute to a group of neurons n, learn a featurizer f (e.g., a rotation matrix or sparse autoencoder), and identify a feature f (e.g., a linear subspace of the residual stream in a transformer) for the attribute. ravel contains five types of entities (cities, people names, verbs, physical objects, and occupations), each with at least 500 instances, at least 4 attributes, and at least 50 prompt templates per entity type. the metric we use to assess interpretability methods is based on interchange interventions (also known as activation patching). this operation has emerged as a workhorse in interpretability, with a wide swath of research applying the technique to test if a high-level concept is stored in a model representation and used during inference @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . specifically, we use the lm to process a prompt like "paris is in the continent of" and then intervene on the neurons n to fix the feature f to be the value it would have if the lm were given a prompt like "tokyo is a large city." if this leads the lm to output "asia" instead of "europe", then we have evidence that the feature f encodes the attribute "continent". then, we perform the same intervention when the lm processes a prompt like "people in paris speak". if the lm outputs "french" rather than "japanese', then we have evidence that the feature f has disentangled the attributes "continent" and "language". a variety of existing interpretability methods are easily cast in the terms needed for ravel evaluations, including supervised probes @xcite @xcite , principal component analysis @xcite , differential binary masking (dbm: cao et al. 2020; @xcite @xcite , sparse autoencoders @xcite , and distributed alignment search (das @xcite . our apples-to-apples comparisons reveal conceptual similarities between the methods. in addition, we propose multi-task training objectives for dbm and das. these objectives allow us to find representations satisfying multiple causal criteria, and we show that multi-task das is the most effective of all the methods we evaluate at identifying disentangled features. this contributes to the growing body of evidence that interpretability methods need to identify features that are distributed across neurons. ## the ravel dataset the design of ravel is motivated by four highlevel desiderata for interpretability methods: the attribute disentanglement task we begin with a set of entities @xmath0, . . . , e n }, each with attributes @xmath1, . . . , a k }, where the correct value of a for e is given by a e . our interpretability task asks whether we can find a feature f that encodes the attribute a separately from the other attributes a \ {a}. for transformer-based models @xcite , a feature might be a dimension in a hidden representation of an mlp or a linear subspace of the residual stream. we do not know a priori the degree to which it is 2 possible to disentangle a model's representations. however, our benchmark evaluates interpretability methods according to the desiderata given above and so methods will need to be faithful to the model's underlying structure to succeed. in other words, assuming methods are faithful, we can favor methods that achieve more disentanglement. ## interpretability methods we use ravel to evaluate a variety of interpretability methods on their ability to disentangle attributes while generalizing to novel templates and entities. each method uses data from the training split to find a set of neurons n, learn a featurizer f, and find a feature f a that captures an attribute @xmath2. in section 4, we describe the baseline procedure we use for considering different sets of neurons. in this section, we define methods for learning a featurizer and identifying a feature given a set of neurons. for each method, the core intervention for a is given by ii(m, f a , x, x ′ ) where f a is defined by the method. in this section, we use getvals(m(x), n) to mean the activations of neurons n when m processes input x. ## experiments we evaluate the methods described in section 3 on ravel with llama2-7b @xcite , a 32-layer decoder-only transformer model, as the target lm. implementation details of each method are provided in appendix b. ## related work intervention-based interpretability methods intervention-based techniques, branching off from interchange intervention @xcite or activation patching @xcite , have shown promising results in uncovering causal mechanisms of lms. they play important roles in recent interpretability research of lms such as causal abstraction @xcite , causal tracing to locate factual knowledge @xcite , path patching or causal scrubbing to find causal circuits @xcite @xcite , and distributed alignment search @xcite . previous works suggest that activation interventions that result in systematic counterfactual behaviors provide clear causal insights into model components. isolating individual concepts lms learn highly distributed representations that encode multiple concepts in a overlapping sets of neurons @xcite @xcite . various methods have been proposed to find components that capture a concept, such as finding a linear subspace that modifies a concept @xcite @xcite @xcite and generating a sparse feature space where each direction captures a word sense or is more interpretable @xcite @xcite . however, these methods have not been evaluated against each other on their ability to isolate concepts. isolating an individual concept is also related to the goal of "disentanglement" in representation learning @xcite , where each direction captures a single generative factor. in this work, we focus on isolating the causal effect of a representation. knowledge representation in lms understanding knowledge representation in lms starts with probing structured linguistic knowledge @xcite @xcite . recent work expands to factual knowledge stored in transformer mlp layers @xcite @xcite , associations represented in linear structures @xcite @xcite , and deeper study of the semantic enrichment of subject representation @xcite . these findings suggest lms store knowledge modularly, motivating the disentanglement objective in our work. ## conclusion we present ravel a benchmark for evaluating the ability of interpretability methods to localize and disentangle entity attributes in lms in a causal, 8 generalizable manner. we show how ravel can be used to evaluate five different families of interpretability methods that are commonly used in the community. we benchmark several strong interpretability methods on ravel with llama2-7b model as baselines, and we introduce a multi-task objective that improves the performance of differential binary masking (dbm) and distributed alignment search (das). multi-task das achieves the best results in our experiments. results on our attribute disentanglement task also offer insights into the different levels of entanglement between attributes and the emergence of disentangled representations across layers in the llama2-7b model. the community has seen an outpouring of innovative new interpretability methods. however, these methods have not been systematically evaluated for whether they are faithful, generalizable, causally effective, and able to isolate individual concepts. we release ravel foot_2 to the community and hope it will help drive the assessment and development of interpretability methods that satisfy these criteria.