Rigorously Assessing Natural Language Explanations of Neurons (2023)

Abstract: Natural language is an appealing medium for explaining how large language models process and store information, but evaluating the faithfulness of such explanations is challenging. To help address this, we develop two modes of evaluation for natural language explanations that claim individual neurons represent a concept in a text input. In the *observational mode*, we evaluate claims that a neuron a activates on all and only input strings that refer to a concept picked out by the proposed explanation E . In the *intervention mode*, we construe E as a claim that neuron a is a causal mediator of the concept denoted by E . We apply our framework to the GPT-4-generated explanations of GPT-2 XL neurons of Bills et al. (2023) and show that even the most confident explanations have high error rates and little to no causal efficacy. We close the paper by critically assessing whether natural language is a good choice for explanations and whether neurons are the best level of analysis.

## introduction the ability to generate natural language explanations of large language models (llms) would be an enormous step forward for explainability research. such explanations could form the basis for safety assessments, bias detection, and model editing, in addition to yielding fundamental insights into how llms represent concepts. however, we must be able to verify that these explanations are faithful to how the llm actually reasons and behaves. what criteria should we use when assessing the faithfulness of natural language explanations? without a clear answer to this question, we run the risk of adopting incorrect (but perhaps intuitive and appealing) explanations, which would have a severe negative impact on all the downstream applications mentioned above. in the current paper, we seek to define criteria for assessing natural language explanations that figure 1 : an overview of our proposed framework. in the observational mode, we evaluate whether a neuron activates on strings picked out by the explanation. in the intervention mode, we assess whether the neuron is a causal mediator of the concept in the explanation. the observational mode only evaluates whether a concept is encoded, as opposed to used @xcite . thus, we propose an intervention mode to evaluate the claim that a is a causally active representation of the concept denoted by e. we construct next token prediction tasks that hinge on the concept and intervene on the neuron a to study whether the neuron is a causal mediator of concepts picked out by e. for example, consider the explanation years between 2000 and 2003 of a neuron a. in the observational mode, we experimentally test which strings the neuron a activates on and quantify how closely this is aligned with the explanation's meaning. in the intervention mode, we can construct a task where the prefix "the year after y is" is given and the model consistently outputs "y + 1". then we can swap the value of a for the value it takes on a different input and observe whether the behavior exhibits the expected change. the success rate of interventions quantifies the extent to which the neuron a is a causal mediator of the concept of years @xcite @xcite . to illustrate the value of this evaluation framework, we report on a detailed audit of the explanation method of @xcite , which uses gpt-4 to generate natural language explanations of neurons in a pretrained gpt-2 xl model. this is, at present, the largest-scale effort to automatically generate explanations of llms: the authors offer explanations for 300k neurons in gpt-2 xl. automatically generating natural language explanations is inherently exciting, but our findings are inauspicious. in the observational mode, we find that even among the top 0.6% of neurons which are considered well-explained by gpt-4's own assessment, the explanation is far from faithful; construed as predictions about neuron activations, gpt-4 generated explanations achieve a precision of 0.64 and a recall of 0.50. in the intervention mode, the picture is more worrisome: we are unable to find evidence that neurons are causal mediators of the concepts denoted by the explanations. while the proposed explanations from the method of @xcite can be useful in exploring hypotheses about model computations, users of the method should have full knowledge of these assessments if they plan to make decisions based off these explanations. 1 we conclude by discussing some of the fundamental issues at hand. first, is natural language a good vehicle for model explanations? it seems appealingly accessible and expressive, but its ambiguity, vagueness, and context dependence are substantial problems if we want to use these explanations to guide technical decision making. second, are neurons appropriate units to analyze? there may be useful signals in individual neurons, but it seems likely that the important structure will be stored in more abstract and distributed ways @xcite @xcite @xcite . however, automated generation poses challenges for evaluation. the faithfulness of natural language explanation is inherently hard to evaluate @xcite . existing automated metrics are mostly neuron-level @xcite . only a few measure model behaviors via ablation or editing @xcite , which is critical for distinguishing encoded vs. used information in neuron analysis @xcite . besides concerns in faithfulness, recent work on distributed representations @xcite and superposition phenomena @xcite suggests individual neurons may not provide the most interpretable structure. intervention-based methods interpretability methods that use interventions to create counterfactual model states have so far provided the most provably faithful explanations of model behaviors @xcite @xcite @xcite @xcite @xcite @xcite . interventionbased methods are also adopted to measure the faithfulness of explanations @xcite @xcite . our evaluation is a causal mediation analysis @xcite , a special case of causal abstraction analysis @xcite . ## observation-based evaluation we now define a framework for evaluating claims that a natural language text e explains a neuron a in a model m using direct observational data. ## intervention-based evaluation the goal of intervention-based evaluation is to assess the claim that a neuron a is a causal mediator of the concept denoted by e. intervention-based evaluation allows us to distinguish concepts that are used vs. encoded in a model @xcite , which is tightly connected to applications that require control and manipulation of the model, such as model editing. if we would like to use the explanation to inform us about where a concept is stored, we need explanations that pass the intervention-based assessment. otherwise, modifying neurons associated with the explanation will have no effect on model behaviors. ## conclusion we developed a framework for rigorously evaluating natural language explanations of neurons. our observational mode of analysis directly tests explanations against sets of relevant inputs, and our intervention mode assesses whether explanations have causal efficacy. when we applied this framework to the method of @xcite , we saw low f1 scores in the observational mode and little or no evidence for causal effects in the intervention mode. finally, we confronted what seem to us to be deep limitations of (i) using natural language to explain model behavior and (ii) focusing on neurons as the primary unit of analysis. overall, we are more optimistic about approaches to model explanation that are grounded in structured formalisms (e.g., programming languages) and seek to explain how groups of neurons act in concert to represent examples and shape input-output behaviors.