Temporal Alignment of Time Sensitive Facts with Activation Engineering (2025)

Abstract: Large Language Models (LLMs) are trained on diverse and often conflicting knowledge spanning multiple domains and time periods. Some of this knowledge is only valid within specific temporal contexts, such as answering the question, “Who is the President of the United States in 2022?” Ensuring LLMs generate time-appropriate responses is crucial for maintaining relevance and accuracy. In this work we explore activation engineering as a method for temporally aligning LLMs to improve factual recall without any training. Activation engineering has predominantly been used to steer subjective and qualitative outcomes such as toxicity or behavior. Our research is one of few that uncovers the bounds of activation engineering on objective outcomes. We explore an activation engineering technique to anchor LLaMA 2, LLaMA 3.1, Qwen 2 and Gemma 2 to specific points in time and examine the effects of varying injection layers and prompting strategies. Our experiments demonstrate up to a 44% and 16% improvement in relative and explicit prompting respectively, achieving comparable performance to the fine-tuning method proposed by Zhao et al. (2024). Notably, for LLaMA 2 and LLaMA 3.1 our approach achieves similar results to the fine-tuning baseline while being significantly more computationally efficient and requiring no pre-aligned datasets.

## introduction large language models (llms) encode and train on a large corpus of information that the end user can query @xcite @xcite . their training sets can span a large timeframe leading to overlapping and conflicting answers for time sensitive queries such as "who is the president of the united states of america?". questions like these can have different answers throughout time with the correct answer (joe biden in 2024 and donald trump in 2025) being temporally sensitive; relevant to the time it is asked @xcite @xcite . temporally sensitive questions require llms to correctly understand the time they are answering for. without temporal alignment llms are recalling facts based on training distributions leading to a chaotic sense of time for factual recall @xcite . this leads to errors such as gpt2-xl recalling that the current prime minister of australia is malcolm turnbull. this is at odds with the knowledge the llm has on hand. the model, if prompted carefully can recall that scott morrison is the serving prime minister of australia in 2022, a more recent and temporally relevant answer. the confusion about whom the current prime minister of australia demonstrates how temporal misalignment can lead to erroneous factual recall @xcite . conflicting answers throughout time can lead to a range of recall errors. to overcome these errors many methods aim to overwrite the subject, relationship, object (sro) facts held within an llm @xcite @xcite @xcite ; for example, overwriting the object associated with the president of the united states of america and altering the probability of the output tokens to preference don-ald trump over joe biden. these methods fall under the categories of continual learning @xcite @xcite , knowledge editing @xcite @xcite @xcite , and retrieval augmented generation (rag) @xcite , which all strive to provide correct information to the end user. however, overwriting sro fact sets ignores the opportunity to realign llms temporally to recall the correct facts already known by the model. building on the temporal alignment work of @xcite we explore the capability of injecting vectors into the residual stream during inference (activation engineering) as a technique to temporally align models within their existing knowledge cut-off timeframe and correct for relative temporal statements such as "who is the current prime minister of japan?". figure 1 demonstrates that when llama2-7b (which has a knowledge cutoff date of september 2022) is asked this question the preferred response is yoshida suga (who served as prime minister between 2020 and 2021). however, when temporally aligned to the year 2022, using activation engineering the preferred response is fumio kishida; who was the current serving prime minister of japan in 2022. the distinction within this scenario is that we are aligning to knowledge the model already possesses. while zhao et al. (2024) uses fine-tuning to temporally align, we explore the effectiveness of activation engineering (ae); reducing the computational requirements to temporally align, increases the flexibility and responsiveness for end users, and requires less prealigned data to reference and train. in this work we hypothesise that ae is a more efficient method to temporally align models compared to fine-tuning, reducing the amount of training and data required, whilst providing similar outcomes to temporal alignment via fine-tuning. we experiment with the activation methods of we apply ae over two datasets, head of governments (hog) and temporal alignment question answer (taqa) @xcite . the hog dataset was created as part of this research to be a small and domain specific dataset. the taqa dataset @xcite on the other hand was selected to test the effectiveness of ae on a larger, more diverse dataset; exploring the generality of ae on temporal alignment and providing a benchmark to compare against. we run sweep tests throughout llama2-7b, 13b and 70b models varying the layers of activation and phrases injected into the residual stream. next, we test the effect of ae on a single layer and then compound the effect by applying ae to multiple layers. we then compare the results of ae against explicit prompting (e.g., in 2022 the president of the united states of america is?), relative prompting (e.g., the current president of the united states of america is?) and fine-tuning; following the methodology of @xcite . we take our findings from the smaller hog and taqa-1000 experiments and scale them to the entire taqa-9000 testing set where we compare our summary taqa results to @xcite demonstrating a similar alignment to a specific year, but with less computational overhead. ## datasets a variety of datasets have been developed to examine the properties and methods pertaining to llms. recall, biases and reasoning @xcite are a few categories of datasets constructed to address research in these areas. these datasets typically look at subject relationship object (sro) mappings, ignoring time as a factor for those relationships. for temporal research our dataset requires a record of changes over time for an sro fact set, expanding the dataset to one that maps srot: subject, relationship, object and time. specifically, we require a dataset that contains consistent time-exclusive answers, where the answer a is only valid between @xmath0, t e ], and a has a consistent set of answers through time. to the best of our knowledge, there are only two datasets that meet this requirement @xcite . other datasets such as atoke @xcite , mquake @xcite , chronoedit @xcite contain question-answer sets through time but do not continuously record the change of answers over time. these datasets are focussed on knowledge editing for time sensitive questions typically exploring one hop knowledge editing effects. for our experiments we have chosen to first create a smaller head of government (hog) dataset to provide an easier measure and more efficient experimentation feedback loop. then, to validate our findings are domain agnostic, we test against temporal alignment question answer (taqa) @xcite which contains a much wider knowledge domain. we chose taqa over @xcite as taqa is originally benchmarked against a fine-tuning alignment approach which provides a fair baseline for us to compare against. ## methods as outlined in algorithm 1 and figure 2 , during the prediction of an answer for a temporally sensitive question p u such as, "who is the current president of the united states of america?", we inject a pre-defined steering vector ae that represents the specific year in to layers l of the model. these steering vectors are created from running temporal phrases {p, } such as a year number through the model and extracting the activation vectors h before a selected layer. using h, we apply a positive or negative coefficient c to phrase's vector increasing or reducing the influence of the phrase, and producing h a . negative coefficients are used to reduce the influence of a phrase, whilst positive coefficients are used to increase the influence of a phrase upon the output of a model. having developed a set of vectors h that have been multiplied by a coefficient, we sum the set of phrase vectors together to produce a single vector ae that encapsulates a time period we aim to align to. we add the alignment vector ae to the residual stream of query prompt p u at the same layer l the original activation vectors h were extracted from. ultimately, this small vector "nudge" can change the probability of output tokens, steering the model to recall information from a specific period of time. ## conclusion temporal alignment can improve the factual recall accuracy of models but can currently only be achieved through fine-tuning, a computational and time intensive process. we have demonstrated similar results to the fine-tuning process using activation engineering (ae) on llama2-7b, 13b and 70b models. ae can produce similar results to fine-tuning with less data preparation and less upfront computation and time requirements. our ablation studies have shown that ae effects a model's perception of time through specific tweaks in the residual stream of lower layers in an llm. generalisation of ae to llama3.1, qwen2 and gemma2 models has shown mixed results, with ae being effective for llama3.1. these results could be the result of differences in architecture or training data.