Analyzing (In)Abilities of SAE s via Formal Languages (2025)

Abstract: Autoencoders have been used for finding interpretable and disentangled features underlying neural network representations in both image and text domains. While the efficacy and pitfalls of such methods are well-studied in vision, there is a lack of corresponding results, both qualitative and quantitative, for the text domain. We aim to address this gap by training sparse autoencoders (SAEs) on a synthetic testbed of formal languages. Specifically, we train SAEs on the hidden representations of models trained on formal languages (Dyck-2, Expr, and English PCFG) under a wide variety of hyperparameter settings, finding interpretable latents often emerge in the features learned by our SAEs. However, similar to vision, we find performance turns out to be highly sensitive to inductive biases of the training pipeline. Moreover, we show latents correlating to certain features of the input do not always induce a causal impact on model’s computation. We thus argue that causality has to become a central target in SAE training: learning of causal features should be incentivized from the ground-up. Motivated by this, we propose and perform preliminary investigations for an approach that promotes learning of causally relevant features in our formal language setting.

## introduction in recent years, mechanistic interpretability has gained currency as an approach towards understanding the functioning of language models (lms) @xcite @xcite . a mechanistic interpretability paradigm that has seen remarkable progress especially is sparse autoencoders (saes), which aims to disentangle model representations into independent, interpretable components (lieberum et al., 2024; bricken et al., 2023; cunningham et al., 2023; engels et al., 2024; gao * these authors contributed equally. figure 1 : the autoencoder paradigm for interpretability. autoencoders have formed the basis of approaches to disentanglement in the vision domain @xcite . utilizing synthetic testbeds, prior work has shown several limitations in this general pipeline @xcite . while saes have similarly been used to disentangle hidden representations of language models for interpretability, we aim to perform a similar study as ones in vision to understand the (in)abilities of saes. @xcite @xcite (see app. a for related work). similar interpretability approaches based on autoencoders have been extensively studied in the vision domain (see fig. 1 ), from both empirical and theoretical viewpoints @xcite @xcite @xcite . specifically, using synthetic testbeds where a ground-truth set of latents underlying the data-generating process can be listed @xcite , these studies have established results on the (non-)identifiability of disentangled representations via autoencoders and argued for the importance of inductive biases, which motivate different regularization terms. for instance, @xcite prove that for any given dataset (modeled as a multivariate distribution), there are infinitely many generative models, all entangled with each other, and remark that inductive biases are necessary to "select" from among these distributions. @xcite show that in a correlated dataset (which real-world datasets tend to be), a perfectly disentangled model would lead to suboptimal log-likelihood, and therefore would not be learned without significant inductive biases. we argue a similar study is needed for understanding the (in)abilities of saes for interpreting lm representations. ## experimental setup our experiments consist of training saes (of various paradigms) on the intermediate representation of transformer models trained on formal languages. we explain the data generating process, model architectures, and training paradigms next. ## results qualitatively, we observe that top-k-regularized saes tend to have more interpretable features than l 1 -regularized ones. the former results in interpretable features across all languages-we find features representing fundamental aspects of the corresponding grammars, as discussed next briefly. see app. c for a description of our feature identification pipeline, and further results, including other features we are able to identify and how strongly features correlate with their claimed explanations. to the closing bracket's (right). we note that the depth and opening activation are sufficient to determine the closing activation, and that the opening and closing activations are sufficient to determine the depth. in the case of dyck-2, we expect the depth (the number of brackets yet to be closed) to be represented. we find a feature that thresholds the depth of tokens, i.e., it activates on tokens with depth above a certain threshold depth d. usually, d is greater than the mean; for instance, when mean depth is 8.3, we find @xmath0. we also find features that match corresponding pairs of opening and closing brackets, usually at lower depths (fig. 2 ). these features take values from only a few small ranges, and their values at an opening bracket determine those they take at the matching closing bracket. these features, even though they are binary, indicate that the token representations do maintain some form of counter (which aligns with claims from prior work, like @xcite ). in expr, the analogue of depth is a counter that indicates how many more expressions are needed to complete the sequence (app. b). we find a feature that activates exactly when this counter's value is 1, i.e., when exactly one expression is required (fig. 3 ). this provides strong evidence of a counter process being implemented; in fact, we note that generation without this process would be rather convoluted (algorithm 1 ). an inference we can make from the expr features is that there is an implicit "type" feature in the representations, distinguishing operators of different valences. a natural place to look for this is the parts of speech in our english grammar, and we find that k-regularized saes do contain features corresponding to each part of speech. for example, we illustrate the 'adjective' feature in fig. 4 . further, we note a scaling relation in the performance of the top-k-regularized saes with hidden dimension size. the reconstruction loss they achieve (after a fixed number of iterations) decreases according to a power law with the size of the autoencoder's hidden layer (see app. f); similar results were seen by @xcite , relating the reconstruction loss to the l1 penalty coefficient, indicating our setup captures part of the phenomenology observed with complex settings. ## incentivizing causality our results above indicate that though saes are competent at identifying features correlated with semantically meaningful concepts, these features may not have the expected (or any) causal effects. this mirrors similar observations made in vision, where it has been noted that mere data reconstruction pipelines (e.g., based on autoencoders) can fail to disentangle the latent factors of a generative model @xcite . however, later work on the topic demonstrated that correlations in the data-generating process (e.g., correlation in nearby frames of a video) can be leveraged to obtain disentangled representations @xcite . as this method relies on supervision via other data samples, rather than a ground truth, it is described as weak supervision. a warranted question then is whether weak supervision can be elicited in language modeling scenarios too. motivated by the above, we take inspiration from klindt et al. (2020)'s use of temporal correlations in video data as weak supervision. we argue the temporal and sequential nature of language can be exploited in a similar manner as well: more precisely, we can consider tokens belonging to the same sequence to share latent factors, and therefore pair up tokens within sequences in a similar way as image pairs in vision. we next operationalize this idea in our setup (sec. 4.1) and then analyze the features obtained by saes trained with this method (sec. 4.2). we discuss in sec. 4.3 the connection between the inductive biases of the proposed pipeline and the nature of identified features. we note that the proposed protocol and results should be deemed a proof-of-concept: our goal is to make a stronger connection to prior literature on autoencoder-based approaches to interpretability, hence eliciting (in)abilities of such protocols. ## conclusion inspired by studies in vision @xcite , we propose a minimalistic setup to assess challenges in the use of saes for model interpretability. we validate our setup by identifying semantically meaningful features and demonstrating the sensitivity of said features' extraction to inductive biases. we further demonstrate a lack of causality, i.e., interventions on these features do not yield intended effects. we expect these results to bear out at scale as well, e.g., we find it likely that features identified by saes will not always be causally relevant to model computation. while previous works, like @xcite , have successfully built upon sae features to identify circuits, we believe our results demonstrate the importance of embedding causality into the feature identification process as a first-class citizen. as a step towards this integration, we propose a modified objective for training saes that exploits the structure of our data, and find that it succeeds in identifying features with predictable causal function. we deem these results as a proof-of-concept corroborating our broader arguments, and hypothesize that designing a singular protocol that works well for all tasks and modalities may be difficult.