T ense L o C : Tense Localization and Control in a Multilingual LLM (2025)

Abstract: Multilingual language models excel across languages, yet how they internally encode grammatical tense remains largely unclear. We investigate how decoder-only transformers represent, transfer, and control tense across eight typologically diverse languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai. We construct a synthetic tense-annotated dataset and combine probing, causal analysis, feature disentanglement, and model steering to LLaMA-3.1 8B. We show that tense emerges as a distinct signal from early layers and transfers most strongly within the same language family. Causal tracing reveals that attention outputs around layer 16 consistently carry cross-lingually transferable tense information. Leveraging sparse autoencoders in this subspace, we isolate and steer English tense-related features, improving target-tense prediction accuracy by up to 11%% in a downstream cloze task.

## introduction recent transformer-based large language models (llms) learn high-dimensional contextual embeddings that yield state-of-the-art performance on multilingual tasks. however, these vectors conflate multiple linguistic features @xcite @xcite , and as yet it remains unclear how these models represent tense internally. grammatical tense, how languages mark past, present, and future, is fundamental to accurate human communication, reasoning and natural language processing (nlp) alike. linguistic theories, from reichenbach's tripartite model of event, reference, and speech time @xcite to later typological surveys, show that languages employ varied morphological and syntactic strategies, morphological inflections (e.g., "-ed"), auxiliaries (e.g., "will"), or adverbial cues (e.g., "yesterday"), to situate events temporally. despite this foundational importance, the ways in which multilingual llms internally encode tense remain largely unchartered. prior probing work @xcite shows that morphological cues can predict tense in cross-lingual settings (i.e., french and chinese), and large-scale studies report that multilingual encoders reliably encode morphological information including tense @xcite . yet, these studies rely on correlation and cannot show whether the identified subspaces are functionally used by the model. along a related line of research, sparse autoencoders (saes) have been proposed to disentangle monosemantic features, hidden dimensions aligned with human-interpretable concepts @xcite . if successful, saes offer not only interpretability but also control, enabling researchers to steer model outputs by scaling these features @xcite . however, their application to grammatical tense remains under-explored. in particular, it is still unknown whether sparse tense features identified by saes, if they exist, are functionally necessary or sufficient for influencing model predictions. in this work, we present a comprehensive analysis of llama-3.1 8b to examine how it encodes and uses grammatical tense across typologically diverse languages, and to determine whether these encodings are causally necessary, sufficient, and manipulable via sparse-feature interventions. we show that targeted interventions on identified subspaces produce predictable changes in generation accuracy, providing a functional (not merely correlational) account of tense representation. we combine probing, causal tracing, pre-trained saes, and targeted residual-stream interventions grounded in mechanistic interpretability to (i) locate tense subspaces, (ii) identify tense-carrying streams and layers (section 3), (iii) disentangle human-interpretable, monosemantic tense features (section 4), and (iv) test controllability via feature scaling (section 5). we show our main findings in figure 1 . our contributions are fourfold: ## identification and isolation of tense representation we systematically probe how tense is encoded in llama-3.1 8b, examining which layers and com-ponents represent tense and to what extent. to complement this, we use causal tracing to identify which layers are functionally responsible for carrying and applying tense signals during generation. in the body of the paper, we mainly focus on causal tracing, with additional detailed results on probing reported in appendix d. causal tracing of tense signals. as a preliminary experiment to ascertain if the tense representation is linearly decodable, we perform linear probing @xcite @xcite and find that tense representation resides throughout all the layers emerging from early layers and most robust in later layers (figure 7 in appendix d). however, probing alone only shows where information is encoded and thus demonstrates correlation rather than causal influence; to address this, we test causality by intervening in intermediate activations to verify that the representations in question directly drive syntactic tense production. we adopt the causal tracing method introduced by @xcite , implementing layerwise intervention and patching in our target model using the pyvene library @xcite . in causal tracing, we care about the activations (i.e., hidden signals) as they travel through the network, which in our case, are tense signals. prompting. each of our trials uses a one-shot prompt consisting of (i) a full sentence in the target tense and (ii) a truncated version of that sentence ending just before the verb: @xmath0 ## identifying human-interpretable tense features. having identified critical layers ℓ * for tense representation, we next ask whether features extracted by saes can be used to steer the model's outputs. specifically, we test whether activating or inhibiting these features systematically shifts the predicted verb tense. this allows us to evaluate not only the interpretability of sae features but also their causal influence on generation. to this end, we use saes to discover latent features in the model's hidden states that align with grammatical tense in english. our goals are twofold: (i) to validate that sae-derived features are consistent with the probing and causal-tracing results (figures 2 in section 3 and 7 in appendix d), and (ii) to identify monosemantic tense features that reliably map to tense labels and localize to the critical transformer layers ℓ * identified earlier. experiments are conducted on our curated datasets no_temp and with_temp. hidden-state extraction. each sentence is fed individually through the original llama model via the hookedtransformer interface in the sae_lens library @xcite . we capture the hidden activations at the attention, mlp, and residual output streams for critical layers 15-31 (figure 2 in section 3). we extract the activation vector corresponding to the main verb token in each sentence. ## steering tense generation after identifying tense-sensitive features, we test whether these features can be used directly to control model behavior. following an adapter-based steering paradigm @xcite , we integrate sae-derived "tense axes" into the residual stream and scale them during generation (mcgrath et al., 2024; o'brien et al., 2024; härle et al., 2024). this mirrors prior steering work on syntactic and factual pathways and enables fine-grained tense control without retraining. specifically, we evaluate the causal impact of tense features by steering llama-3.1 8b's hidden activations in a cloze task, which is explained below. using feature scaling, we multiply selected latent dimensions by a factor α at critical layers ℓ * , with α > 1 for excitation and α < 1 for inhibition on the cloze task. cloze task. we use a cloze fill-in-the-blank evaluation. each prompt consists of a temporal cue (e.g., "yesterday"), a sentence with a missing verb, three verb-form options (one per tense), and an answer placeholder. for instance: example yesterday, the dog ___ at the mailman. a) barks b) barked c) will bark answer: the model must output the correct option ("a", "b", or "c"). for each tense, we construct a balanced development set of 30 prompts and a test set of 500. prompts pair diverse subjects/objects (e.g., "i", "we", "the mailman") with base verbs conjugated (including irregulars) into target tenses using patternlite (smedt and daelemans, 2012); correct option order is randomized. we run the task in english and german and report accuracy. steering procedure. we compare three baselines and multiple steering configurations ( ## conclusion we present a four-phase diagnostic pipeline: probing, causal tracing, sae disentanglement, and steering that links where tense information is linearly readable in latent representations to where it is functionally necessary and controllable. lin-ear probes show that llama-3.1 8b (meta, 2024) internally represents simple past, present, and future tenses in low-dimensional subspaces that are detectable across layers; with crosslingual transfer peaking in layers 20 to 30, suggesting a language agnostic encoding. causal interventions @xcite localize a functionally necessary subspace at around layers 15-16, primarily within the attention stream (with contributions from mlp activations and outputs), and restoring this small subspace recovers tense probability. applying saes @xcite @xcite to activations at layers 15-16 yields monosemantic tense features that align with human-readable tense markers. scaling these features in the residual stream systematically shifts cloze completions toward the target tense, improving correct-tense accuracy by up to 11% points with modest degradation. crucially, the effect persists even without temporal adverbs (for example, "yesterday"), showing that the model internally encodes tense rather than relying on surface cues. english derived features transfer to german past and present but not future tense, suggesting that the model captures an abstract crosslingual temporal structure, though some future constructions may remain language specific or data limited. to our knowledge, this is the first evidence in a multilingual llm of a causally active, language agnostic tense subspace whose disentangled, interpretable features can steer generation. the finding holds across eight languages for simple tense forms, but broader generalization to richer aspectual patterns, other model families, and naturalistic contexts remains open. future work should extend this framework to more complex temporal systems and finer grained circuit level analyses of cross-lingual temporal representation.