Linear Steerability in Language Models: When It Emerges and How It Evolves (2025)

Abstract: Language models can be steered by modifying their internal representations to control concepts such as emotion, style, or truthfulness in generation. However, the conditions for an effective intervention remain unclear and are often validated through heuristics and trial-and-error. To fill this gap, we demonstrate that intervention efficacy, measured by linear steerability (i.e., the ability to adjust output via linear transformations of hidden states), emerges during intermediate stages of training. Moreover, even closely related concepts (e.g., anger and sadness) exhibit steerability emergence at distinct stages of training*.To better interpret the dynamics of steerability during training, we adapt existing intervention techniques into a unified framework, referred to as the “Intervention Detector” (ID), which is designed to reveal how linear steerability evolves over the course of training through hidden state and representation analysis. ID reveals that concepts become increasingly linearly separable in the hidden space as training progresses, which strongly correlates with the emergence of linear steerability. We further introduce ID-based metrics, such as heatmaps, entropy trends, and cosine similarity, to help interpret how linear steerability evolves throughout training. In addition, we apply ID across different model families to ensure the generality of our findings on steerability dynamics.

## introduction transformer-based language models have achieved considerable success, demonstrating great potential to enhance human productivity @xcite . to align the outputs of these models more closely with desired outcomes, a common approach is to fine-tune such models with carefully curated datasets @xcite @xcite . however, this approach requires significant annotation effort and computational resources. probing studies have shown that neural networks' hidden states encode meaningful concept representations, offering an alternative to expensive finetuning @xcite @xcite . by manipulating these representations, various methods can steer model output, including steering vectors and contrast-consistent search @xcite @xcite . among these, linear steering is notable for being adjustable and minimally invasive, demonstrating effectiveness in output control @xcite @xcite @xcite @xcite . however, despite the empirical success of these methods, the underlying mechanisms that govern intervention effectiveness remain poorly understood. as a result, while interventions can manipulate model outputs, we cannot reliably predict their effects or ensure that the representations being modified align with human-interpretable concepts. for intervention to be robust and trustworthy, it is essential to understand when and why such techniques work. inspired by the methods summarized in our contributions are as follows. this method could serve as a cost-effective monitoring tool for applications that require linear steering, such as language model agents and ai chatbots. our work presents the first longitudinal study to examine linear steerability across a language model's training lifecycle. motivation: as scaling laws yield diminishing returns, post-training enhancements gain traction. for example, test-time computing @xcite improves llms via augmented inference resources. moreover, representation steering also shows promise. however, most linear steering studies @xcite @xcite focus on fully trained models, overlooking interventions during the training stage. pinpointing when steerability emerges can optimize training by identifying ideal stopping points and prioritizing relevant concepts. while methods like sparse autoencoders and linear probing can extract useful representations for intervention, they require additional training and are thus incompatible with our cost-effective monitoring framework. we leave their exploration to future work. ## related work recent research has predominantly focused on well-trained language models, studying representations in neurons, layers, and circuits @xcite @xcite @xcite while recent research on intervention techniques has been conducted exclusively on well-trained language models, little is known about why such interventions become effective or when during training they begin to work. a systematic study of intervention effects across the full training trajectory would provide insights into the reliability and generalizability of these techniques. currently, only @xcite , pythia @xcite , map @xcite , and llm360 @xcite provide opensource pre-training checkpoints. future directions include examining how different pre-training corpora influence intervention outcomes, and exploring alternative steering methods beyond pca or k-means. ## methodology inspired by existing intervention methods, we adapt them into the intervention detector (id) (figure 3 ) and apply it to two downstream tasks: an unsupervised detection task (focused on emotion concepts without ground truth data) and a supervised detection task (targeting factuality and commonsense reasoning concepts with ground truth annotations). we first fine-tuned a series of base model checkpoints with minimal instruction-following data, similar to the cold start approach employed in training the deepseek-r1 model @xcite . details of the fine-tuning process, including its necessity and its impact on experimental results, are provided in appendix d.2 to enhance the reproducibility of our method. unsupervised detecting task: for this task, the concepts to be extracted do not have ground truth in the data. we construct a set of prompt stimuli, pass them onto the model, and obtain hidden states of the last token. we then apply linear decomposition to get representation vectors that align with human understanding on six emotions -anger, fear, happiness, sadness, surprise and disgust. supervised detecting task: for the task with ground truth data -such as the commonsense reasoning task with multiple choice questions @xcite ) -we extract the difference in the hidden states of the model when correct versus incorrect answers are applied as stimuli. applying linear decomposition to the difference of hidden states of a large enough set of such positive and negative pairs yields concept vectors that align with commonsense reasoning ability. details of the dataset can be found in appendix b. id method involves the following steps: let h @xmath0. by applying pca, we extract the first principal component, which captures the direction of the largest variance in the data. the resulting principal com- in this case, the original data matrix has the shape h @xmath1. we obtain a vector @xmath2, representing the direction of the largest variance in the data. alternatively, we can obtain representation vectors by applying k-means for @xmath3. for a layer l, this vector v l is linked to a specific concept. since pca identifies the direction of maximum variance in the data and k-means can partition data into distinct clusters (e.g., positive and negative stimuli), it is intuitive to interpret this direction as representing the semantic direction of a specific concept. 3. calculate id score: by computing the inner product of the representation vectors from a layer l with the hidden states when passing stimulus s i from s test , we obtain a number which we refer to as the id score i l i for the specific layer l: a lower id score suggests that the concept cannot be effectively captured by linear methods such as pca, meaning that the extracted representation is noisy and is not likely to produce effective interventions. analyzing representation vectors: in this study, we adopt the concept of signal-to-noise ratio (snr) from signal processing to evaluate the effectiveness of representation vectors. in the early stages of training, the representation vectors derived from linear models such as pca are dominated by noise, leading to low snr and poor alignment with human-understandable concepts. as training progresses, the noise decreases and the vectors are better at capturing semantic representations, resulting in more effective interventions. appendix f shows that the proportion of first principal component increases over time, highlighting the growing effectiveness of linear models in capturing conceptual representations. a common way to analyze vectors is through cosine similarity. for a series of checkpoints @xmath4, c 2 , . . . , c n }, on layer l, the cosine similarity between two checkpoints c i and c j can be computed as: analyzing id scores across layers: to analyze how model representations evolve during training, we compute the entropy e l to quantify the distribution of id scores across different checkpoints, and use inter-layer id score differences within each checkpoint to capture the alignment dynamics at a given training stage. let @xmath5, where n is the number of checkpoints and l is the number of layers. for a given checkpoint c, the layer-wise scores i c,l (where @xmath6, l]) are normalized as: ĩc,l log ĩc,l . by calculating the difference in id scores between each layer and its preceding layer, we obtain a curve that characterizes how alignment changes across the model depth. larger values on this curve indicate sharper transitions in alignment strength between adjacent layers. this layer-wise differential signal helps reveal where significant shifts in internal representation structure occur during pretraining. in later sections, we empirically examine whether the checkpoint at which this difference reaches its maximum correlates with the emergence of linear steerability (see figure 9 and during training, id scores and related metrics-such as inter-layer id differences, entropy, and abrupt changes in the cosine similarity of the "steering vector" between adjacent checkpoints-can act as early signals that suggest when linear steerability begins to emerge, and are best interpreted as heuristic cues informed by empirical observation. see ## experiments in our experiments, we saved a checkpoint every 15k steps and fine-tuned each one on the same dialogue dataset as crystalchat, using one-tenth of the data for a single epoch. (crystalchat is the fully fine-tuned model from the final checkpoint.) the details of the model can be found in appendix d, and the experiment settings can be found in appendix f. we also plotted the heatmap of the id scores using amber @xcite , another open source model with pre-train checkpoints and results can be found in appendix i. experimental results using a finer-grained data set -constructed with token-level stimulus pairs similar to caacan be found in appendix o, figure 17 . ## conclusion our findings show that linear steerability does not emerge uniformly, but develops over the course of training and varies between concepts. this emergence aligns with increasingly structured internal representations, making concepts more linearly separable in low-dimensional space. by analyzing steerability-related metrics across checkpoints and layers, we can roughly identify when a model becomes steerable for a given concept. these observations hold across different model families, suggesting a general pattern in the dynamics of steerability during training.