FADE : Why Bad Descriptions Happen to Good Features (2025)

Abstract: Recent advances in mechanistic interpretability have highlighted the potential of automating interpretability pipelines in analyzing the latent representations within LLMs. While this may enhance our understanding of internal mechanisms, the field lacks standardized evaluation methods for assessing the validity of discovered features. We attempt to bridge this gap by introducing **FADE**: Feature Alignment to Description Evaluation, a scalable model-agnostic framework for automatically evaluating feature-to-description alignment. **FADE** evaluates alignment across four key metrics – *Clarity, Responsiveness, Purity, and Faithfulness* – and systematically quantifies the causes of the misalignment between features and their descriptions. We apply **FADE** to analyze existing open-source feature descriptions and assess key components of automated interpretability pipelines, aiming to enhance the quality of descriptions. Our findings highlight fundamental challenges in generating feature descriptions, particularly for SAEs compared to MLP neurons, providing insights into the limitations and future directions of automated interpretability. We release **FADE** as an open-source package at: [github.com/brunibrun/FADE](https://github.com/brunibrun/FADE).

## introduction understanding the latent features of machine learning models and aligning their descriptions with human-comprehensible concepts remains a crucial challenge in ai interpretability research. recent advances have made significant strides in this direction, by introducing automated interpretability methods @xcite @xcite that leverage larger language-capable models to describe the latent representations of smaller models @xcite @xcite . * these authors contributed equally. this facilitates inspection of ml models, enabling a deeper understanding of models' behavior which enhances our ability to identify or mitigate harmful responses and biases @xcite . a key insight from these investigations is the highly polysemantic nature of individual neurons -they rarely correspond to single, clear concepts. this discovery has led to the development and adoption of sparse autoencoders (saes) @xcite @xcite , which are intended to decompose polysemantic representations by separating neuron activations into more interpretable components. while saes offer a promising approach for feature decomposition, their reliability remains an open question. recent research reveals significant variability in the way saes capture the underlying learned features @xcite , thus highlighting the need for a holistic framework for the evaluation of feature-to-description alignment. to the best of our knowledge, there is an absence of widely accepted quantitative metrics for evaluating the quality and effectiveness of open-vocabulary feature descriptions. different methodologies rely on custom evaluation criteria which makes it challenging to conduct meaningful, generalizable comparisons across techniques. additionally, existing evaluation approaches typically optimize for a single metric @xcite which may not capture the full complexity of a feature's behavior and leaves open questions about whether the model truly encodes the hypothesized concept rather than simply correlating with the measured feature. with our work, we contribute as follows: ## related work evaluating the alignment between features and their descriptions has become increasingly important with the rise of automated interpretability approaches. while manually inspecting highly activating examples remains a common method to validate interpretations and demonstrate automated interpretability techniques @xcite , more scalable tools are needed for thorough quantitative evaluation. many automated or semi-automated approaches have been proposed, generally falling into activation-centric and output-centric methods. activation-centric methods focus on measuring how well a feature's activations correspond to its assigned description. one prominent approach is a simulation-based scoring, where an llm predicts feature activations based on the description and input data, and the correlation between predicted and real activations of a feature is measured @xcite @xcite . while elegant, this approach can be computationally expensive and tends to favor broad, high-level explanations. a related and conceptually more straightforward way to measure how well the description explains a feature's behavior is to try to directly generate synthetic samples using the description and compare the resulting activations between concept and non-concept samples @xcite @xcite . however, generated datasets are typically small (on the order of 5-20 samples @xcite ) and often constrained to rigid syntactic structures or focus only on the occurrence of particular tokens, making them less effective for evaluating abstract or open-ended language concepts @xcite . another strategy is rating individual samples from a natural dataset for how strongly they express a concept and compare those ratings to the feature's activations @xcite @xcite . a common limitation of activation-centric methods is that they primarily assume activations to be positively correlated with the concept, thus ignoring negatively encoded features @xcite . output-centric methods instead assess how feature activations influence model behavior. some approaches measure the general decrease in performance of the model after ablating the feature @xcite , while others use steering-based interventions, where an increase in generated outputs containing the concept is used as a proxy for feature alignment @xcite . there is a growing need for frameworks that integrate multiple perspectives to provide a comprehensive assessment of feature-to-description alignment across different interpretability methods. for instance, prior work @xcite @xcite has shown that while activation-centric and output-centric measures often correlate, they do not necessarily imply a causal relationship. some studies focus exclusively on saes @xcite , while others analyze mlp neurons @xcite . developing an architecture-agnostic framework for featuredescription evaluation is essential for enabling robust quantitative comparisons across interpretability approaches. although efforts have been made to integrate multiple evaluation perspectives @xcite , these remain fragmented and are often too narrowly scoped to handle open-ended language descriptions. our work addresses these limitations by introducing a more comprehensive evaluation framework that combines activation-and output-centric metrics while explicitly considering interpretability for open-ended language descriptions. ## evaluating feature explanations our primary objective is to establish a comprehensive framework that automatically evaluates feature descriptions across a variety of feature types without human intervention. our framework encompasses four distinct metrics: clarity, responsiveness, purity, and faithfulness, which we consider necessary and sufficient for assessing the alignment between a feature and its description. in our opinion, such a comprehensive evaluation framework is necessary to ensure that features encode the ascribed concept in a robust way. as feature descriptions are often generated by optimizing for a single metric, such as maximizing the activations of specific neurons, they do not necessarily generalize well to other quantifiable aspects, such as faithfulness @xcite . we base our approach on four key assumptions. first, we adopt a 1 ## experiments in this section, we apply fade to assess the alignment of features and their descriptions generated via various state-of-the-art automated interpretability methods @xcite . our goal is to demonstrate that the proposed framework provides a robust, multidimensional measure of feature-to-description alignment. experimental setup as a natural dataset for the evaluations we use samples drawn from the test partition of the pile dataset @xcite , preprocessed as shown in appendix b, giving us approximately 5 million samples. as evaluating llm we use the openai model gpt-4o-mini-2024-07-18 unless stated otherwise. prompts for the evaluating llm as well as details on the hyperparameters can be found in appendix d.2. we run the experiments on 10 3 randomly chosen features from a single layer of a model: layer 20 for gemma-2-2b @xcite , layer 20 of gemma scope saes @xcite , and layer 19 of llama-3.1-8b-instruct @xcite (see appendix c.1 for details). the evaluation results have a high variance, which is caused by both the inherent difficulty of interpreting some features as well as the quality of the ascribed feature descriptions. therefore the mean values are reported only where they help interpret the distribution of the metrics. for all of the presented tables we demonstrate the full distributions as kernel-density estimations with bandwidth adjustment factor 0.3 in appendix e.1. automated interpretability approach feature descriptions, which we refer to as maxact*, are generated based on samples of the train partition of the pile dataset, that demonstrate maximum activation on the feature, similarly to methods utilized in @xcite @xcite . the minor differences between maxact and maxact* are prompts, optimized on qualitative analysis provided via fade, and preprocessing steps of the dataset. the automated interpretability pipeline is described in appendix c.1. ## conclusion in this work, we presented fade, a new automated evaluation framework designed to rigorously evaluate the alignment between features and their open-vocabulary feature descriptions. by combining four complementary metrics clarity, responsiveness, purity, and faithfulness, our approach gives a comprehensive assessment of how a feature reacts to instances of the described concept, an evaluation of the description itself as well as the feature's causal role in the model's outputs. through extensive experiments across different feature types, layers, and description generation mechanisms, we demonstrated that methods relying on a single metric (e.g., simulation-based approaches) often give incomplete or misleading feature descriptions. our framework can be used to highlight both the strengths and weaknesses of existing methods, while it also helps in debugging and improving these methods. we highlighted multiple results for improving the quality of feature explanations, such as using larger, more capable llms for the explainer and including more examples in the prompt. we hope that the open-source implementation of fade will drive further research in automated interpretability and help make language models more transparent and safe to use.