Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations (2025)

Abstract: This paper explores the robustness of language models (LMs) to variations in the temporal context within factual knowledge. It examines whether LMs can correctly associate a temporal context with a past fact valid over a defined period, by asking them to differentiate correct from incorrect contexts. The LMs’ ability to distinguish is analyzed along two dimensions: the distance of the incorrect context from the validity period and the granularity of the context. To this end, a dataset called TimeStress is introduced, enabling the evaluation of 18 diverse LMs. Results reveal that the best LM achieves a perfect distinction for only 11% of the studied facts, with errors, certainly rare, but critical that humans would not make. This work highlights the limitations of current LMs in temporal representation.

## introduction when a language model (lm) completes the textual prompt "the capital of france is" with "paris", it demonstrates that it has stored this fact somewhere in its parameters. however, as shown by numerous studies @xcite @xcite , this type of factual knowledge is not necessarily robust to certain variations in the prompt (use of paraphrases, aliases, typographical errors, negations, etc.). among these variability factors, the temporal dimension of factual knowledge has been less studied. thus, in this paper, we study the robustness of lms' factual knowledge in the face of simple variations in the temporal context. while the state of the art has demonstrated certain biases in lms related to the temporal distribution of their training data or their weaknesses in reasoning with temporal concepts, our work aims to quantify how well lms can correctly associate a temporal context (e.g., a year or a date, such as "in 2018, . . . ", "on november 5, 2022, . . . ") with ## related work this section presents related work to ours, focusing on the study of factual knowledge in lms, the consideration of their temporal aspect, and their temporal reasoning abilities. ## problem statement and dataset the goal of this paper is to measure how robust a language model (lm) is to the temporal context associated with a fact. to achieve this, the proposed experimental protocol involves analyzing the lm's preferences when faced with correct or incorrect contexts for the same fact. this section first formalizes this problem and then presents the timestress dataset, which instantiates it. ## experimentation this section details our experiments on the timestress dataset. as a reminder, our objectives are, in order, to measure the ability of models to distinguish correct and incorrect temporal contexts, analyze their robustness, and search for anomalies in this task when incorrect contexts are closer to or farther from the validity interval, and as the granularity of contexts becomes finer. numerous models from different families and sizes were tested: mistral-nemo-base-2407, mistral-7b-v0.3 (jiang et al., 2023); openeml-{450m, @xcite ; gemma-2-{2b, 9b, 27b} @xcite ; llama-3.1-{8b, 70b} @xcite . for each, both pretrained and instruction-tuned versions were considered, resulting in a total of 18 studied lms. all models were sourced from huggingface.co. in the first series of experiments, the statements were passed to the models as raw text rather than as instructions to enable the comparison between pretrained and instruction-tuned models. the use of an "instruction/message" format is explored in a second phase. ## experimental protocol: motivations there are seemingly more "natural" approaches for probing factual knowledge in language models, such as the evaluation protocols used in lama @xcite , @xcite , kamel (kalo and fichtel, 2022), and bear @xcite . instead of comparing probabilities across several temporal contexts, one could ask the lm to answer temporally contextual questions such as "in 2011, who was the president of the us?", and evaluate the lm based on the generated answers. however, our experimental protocol was preferred for several reasons. first, our setup-where the lm must distinguish between statements with correct and incorrect temporal contexts by assigning probabilities-allows to target specific facts without ambiguity, even in the case of non-functional relations, such as "shares a border with," where a subject-relation pair can have multiple valid objects. in generation-based settings, an lm may produce one or several correct answers, or even off-topic outputs, making evaluation less reliable and direct comparison across lms more difficult. this is especially true given that classical generation-based metrics, such as rouge @xcite , can underestimate performance. sometimes, the set of all correct answers is difficult to enumerate due to the vagueness of the relation (e.g., does asking for the borders of a country include continents and oceans?) and due to the sometimes large number of ways of expressing an answer. additionally, our evaluation protocol is efficient and scalable, as it does not require generation or answer validation. given the imperfections of other evaluation protocols, it would have been difficult to defend our claims-especially those involving sensitive metrics like robustness and the study of rare lm errors-if our results could be attributed to limitations of the evaluation method itself. ## conclusion this study examined the robustness of lms to simple temporal variations in factual knowledge. it assessed their ability to distinguish correct from incorrect temporal contexts based on two factors: the distance of contexts from the validity period of facts and their granularity. to facilitate this, the timestress dataset was introduced, featuring high-quality statements on popular temporal facts from wikidata (according to a popularity index) and enabling the evaluation of 18 lms of varying sizes and families. the results revealed that the best-performing lm was robust for only 11% of the studied facts, exhibiting errors, certainly rare, but critical that are uncommon to humans, which we frame as anomalies. these errors consist of a susceptibility to easy incorrect contexts and imperfect knowledge generalization across granularities. notably, these findings held true regardless of whether the lm was pretrained or instructiontuned, and whether the statements were presented in an instruction or raw format. this highlights the limits of current lms in temporal representation. it is worth noting that since the studied temporal facts are relatively popular, these results likely represent an upper bound of lms' performance on the general population of facts, given the strong link between knowledge popularity and its likelihood of being learned by lms @xcite .