A Study into Investigating Temporal Robustness of LLM s (2025)

Abstract: Large Language Models (LLMs) encapsulate a surprising amount of factual world knowledge. However, their performance on temporal questions and historical knowledge is limited because they often cannot understand temporal scope and orientation or neglect the temporal aspect altogether.In this study, we aim to measure precisely how robust LLMs are for question answering based on their ability to process temporal information and perform tasks requiring temporal reasoning and temporal factual knowledge. Specifically, we design eight time-sensitiverobustness tests for factual information to check the sensitivity of six popular LLMs in the zero-shot setting.Overall, we find LLMs lacking temporal robustness, especially to temporal reformulations and the use of different granularities of temporal references. We show how a selection of these eight tests can be used automatically to judge a model’s temporal robustness for user questions on the fly. Finally, we apply the findings of this study to improve the temporal QA performance by up to 55%.

## introduction despite the strong zero-and few-shot performance of llms, it has been recently pointed out that llms suffer from a partial or imprecise understanding of the temporal scope, orientation, and reasoning expressed in text @xcite @xcite . the inaccurate understanding of temporal orientation and grounding raises concerns regarding the effectiveness of llms over a range of tasks involving temporal reasoning and intents like questionanswering and search over historical sources @xcite , qa over legal and personal temporal collections @xcite @xcite , or fact checking @xcite . moreover, questions with temporal aspects are relatively rare in many current qa datasets and may thus go undetected user llm yoichiro nambu received which award in 2008 ? nobel prize in physics user the answer correct? can i trust that this answer is correct? does the model robustly understand the temporal fact? ## study details we use several data sources to test factual knowledge with time-scoped questions for assessing llms' robustness in handling temporal references: wikidata (time-sensitive qa @xcite ), the historical new york times news archive (archivalqa @xcite ), temporalquestions @xcite , major world events from wikipedia (event dating/ordering), and factchecked temporal claims crawled from various websites (temporal claims @xcite ). an overview is given in temporal reversal. we use wikidata information (similar to @xcite ) and interpret these factual statements as quadruples (subject, relation, object, time). in other studies, these quadruples have been used to construct questions such as "who was the american president in 2012?" answer: "obama," asking for the subject or the object of the quadruple. we hypothesize that a thorough, actual understanding of the question's temporal aspect would result in the model being able to answer both the normal (forward) question as well as a reformulation of this question that queries for the time (e.g., "when was obama president of the usa?" answer: "2009-2017"). we utilize 10k examples from wikidata for this test since it has quadruples with individual years and the required intervals in which the relation was true. we apply the set of relations used by @xcite and write templates for the reformulations. ## testing temporal robustness in the upcoming sections, we investigate different classes of temporal questions and problems and how our models react to them. for convenience, we show an overview of all models and temporal robustness results in ## automatic robustnesss testing as with all test suites, our tests for temporal robustness, as discussed in the previous sections, are built a priori and require that we know the answers to the questions. however, many of our tests are querycentric reformulations that can be applied to almost generic questions (ending in a temporal reference like "in 2007?"). thus, we wonder whether we could use these query-centric tests to better judge if the model correctly processed and understood the time component of the question. in this setting, we are not required to know the ground-truth answer, which allows us to use this automatic test suite on the fly. this can be beneficial either when hosting the chat/question-answering system to understand when the models might output wrong answers to users or by directly showing the results so that users may use this additional information as contextualization to decide whether they want to trust the model answer or not (example in given the lack of robustness in the relativization, positioning, and reversal tests, one might question any model's temporal literacy. note that the model might still produce the correct answer, even if the predictions and their consistency might lead us to believe it is not sure about this. however, if the model produces the correct answer, it is more likely due to shortcuts or chance and not because it understood the temporal factual relation. next, we show that the query-centric reformulations can be used to estimate whether llm answers are correct. we use 3, 000 questions for which we have ground-truth answers, creating automatic question reformulations and getting the model predictions for all questions. we evaluate the consistency between the original and test question predictions, resulting in four values of 0 to 1. we find these four consistency scores to be predictive of the actual correctness of the answer, outperforming a majority classifier on a balanced test set by 14.9% (llama 3.1). this result emphasizes that besides better understanding and calibrating user trust in the predictions, these scores can help to evaluate the correctness of model predictions automatically. ## discussion & conclusion the temporal robustness tests presented in this work offer a first suite to benchmark temporal processing abilities of llms. besides actionable insights -like avoiding relative time references for most models or starting temporal questions with a time reference -we offer tests that help investigate what temporal understanding is present in models. the temporal robustness tests may be used in addition to the typical task performance as predeployment checks to evaluate models' abilities and problems better. for example, one might take gpt 4 and jamba 1.5, which perform very similarly on temporal qa (absolute values in in this study, we examined the temporal robustness of llms. we tested a variety of llms using a suite of eight tests assessing different kinds of temporal abilities and robustness to natural paraphrases of questions. while we generally observe higher qa performance for bigger models such as gpt 4, we did not find these models robust to our temporal robustness tests. this study serves as the inaugural benchmark for llms' temporal robustness, providing valuable insights into correct temporal information processing and model failures. further, we found our temporal robustness tests applicable along the entire model lifecycle: 1) for developers to benchmark their models and understand which abilities need improvement. 2) as pre-deployment checks to understand the differences between similarly performing models. 3) by using our automatic tests to help users gauge whether or not to trust the model's predictions. 4) by guiding question reformulations for improved qa performance. we believe this set of tests to help study the temporal robustness of llms.