ETRQA : A Comprehensive Benchmark for Evaluating Event Temporal Reasoning Abilities of Large Language Models (2025)

Abstract: Event temporal reasoning (ETR) aims to model and reason about the relationships between events and time, as well as between events in the real world. Proficiency in ETR is a significant indicator that a large language model (LLM) truly understands the physical world. Previous question-answering datasets available for evaluating the ETR ability lack a systematic taxonomy and pay limited attention to compound questions. In this paper, we propose a unified taxonomy for event temporal questions and construct a comprehensive benchmark ETRQA, to evaluate the ETR abilities of LLMs based on this taxonomy. ETRQA not only inherits and expands the evaluation content of existing datasets but also contains multiple categories of compound questions. We evaluate two leading LLM series, Llama and Qwen, on ETRQA across various settings. Our experimental results indicate that large-scale LLMs exhibit certain ETR abilities. Yet they do not perform well in solving specific types of reasoning tasks, including reasoning involving time spans, reasoning for compound questions, and reasoning with fine temporal granularity. Additionally, we hope ETRQA can benefit the temporal reasoning research community for future studies.

## introduction recently, large language models (llms) have demonstrated impressive performance in numerous reasoning tasks @xcite @xcite . however, it has been observed that they still underperform in a fundamental aspect of human cognition, namely temporal reasoning @xcite . temporal reasoning is typically categorized into three levels: symbolic, commonsense, and event temporal reasoning @xcite . event temporal reasoning (etr) is the most comprehensive of these levels, as it not only encompasses the understanding and application of the first two types of reasoning but also involves modeling and reasoning about the relationships between events and time, as well as between events in the real world @xcite . the proficient etr ability is an important indicator that an llm truly grasps the physical world. therefore, we focus on the task of etr. most question-answering datasets available for evaluating the etr ability are constructed using temporal knowledge graphs. they generally generate questions based on facts that evolve over time @xcite @xcite @xcite . for example, the question "what team did lebron james play for in 2009?" is based on the facts that lebron james has played for different teams over various periods. each fact can be regarded as a temporal event, and we refer to such questions as event temporal questions in this paper. to answer such questions, llms need to perform etr. however, there exist two issues in these previous datasets: (1) due to the lack of a systematic taxonomy, existing datasets are often designed with several empirically defined question types, which results in incomplete evaluations, as shown in to address the above issues, we first establish a unified taxonomy based on previous studies. this taxonomy categorizes event temporal questions by question composition and answer type, providing a standardized reference that can be widely adopted. based on this taxonomy, we propose a question-answering dataset for etr, named etrqa. etrqa inherits the evaluation content of existing datasets while expanding upon them through a more comprehensive coverage and examination of question composition and answer type. specifically, etrqa contains multiple types of compound questions, which make the dataset more complex and challenging. for instance, to solve the compound question "what was the team that lebron james played for the longest time from 2009 to 2015?", llms need to not only understand the temporal constraint "from 2009 to 2015" but also compare the durations of events to find "for the longest time" within that range. it is clear that such compound questions are well-suited for evaluating the comprehensive etr abilities of llms. to evaluate the etr abilities of llms, we adopt two industry-leading llm series: the llama series @xcite and the qwen series @xcite on etrqa. the latest models, llama-3.3-70b-instruct and qwen2.5-72b-instruct, have demonstrated competitive performance, matching or even surpassing gpt-4o @xcite on many benchmarks, making them highly representative. we show the effects of prompting strategies, instruction tuning, model sizes, and general reasoning distillation in the task of etr via llms. furthermore, we analyze and discuss the difficulties and challenges llms face. the experimental results show that: (1) large-scale llms demonstrate a certain level of etr abilities compared with small-scale llms, but there is still room for improvement; (2) careful thought before answering and post-training techniques (e.g., instruction tuning and general reasoning distillation) are crucial for enhancing the models' etr abilities; (3) llms struggle with reasoning involving time spans and show a decline in performance when reasoning for compound questions and reasoning with fine temporal granularity. our contributions can be summarized as follows: ## a taxonomy of event temporal questions the proposed taxonomy of event temporal questions, shown in ## related work early temporal nlp research primarily focuses on temporal information extraction tasks, leading to the development of early evaluation benchmarks such as tempeval @xcite . these efforts are mainly centered around extracting events, times, and temporal relations from text. however, with advancements in language models, particularly large language models, attention gradually shifts to the more challenging task of temporal reasoning. this shift leads to the emergence of numerous temporal reasoning datasets. among them, comprehensive benchmarks such as tram @xcite and timebench @xcite represent a major leap forward in temporal reasoning evaluation. these benchmarks consolidate various existing temporal reasoning datasets to facilitate a thorough evaluation of temporal reasoning abilities. timebench categorizes temporal reasoning into three levels: symbolic @xcite , commonsense @xcite @xcite , and event temporal reasoning. our work focuses on the third category, which involves not only understanding and applying the first two types of reasoning but also modeling and reasoning about the relationships between events and time, as well as between events. many existing datasets can be used to evaluate event temporal reasoning abilities. some of these datasets are specifically designed to evaluate the temporal reasoning abilities of language models. timeqa @xcite is an early dataset used to evaluate a model's ability to understand and reason about time-evolving facts. tempreason @xcite evaluates temporal reasoning across three levels: time-time, time-event, and event-event relations. its follow-up work, complex-tr @xcite , emphasizes the importance of multi-answer and multi-hop (i.e., relative time constraints) temporal reasoning. these three datasets share a common limitation: the question types are limited, and the complexity of reasoning primarily arises from external contexts (e.g., wikipedia). this makes it difficult to pinpoint the specific challenges that models face in event temporal reasoning. tgqa @xcite addresses potential data leakage in evaluation by anonymizing data through replacing entities with fictional ones. however, tot @xcite suggests that this method may introduce incorrect claims and adopts a simpler, non-semantic anonymization approach, while extending the context to more complex graph structures. despite this, both datasets define question types empirically, lacking a systematic taxonomy. complextempqa @xcite divides temporal questions into three categories: attribute, comparison, and counting questions. however, some of the question types it defines involve mathematical reasoning, which makes it unsuitable for independently evaluating temporal reasoning. additionally, several time question-answering datasets built for temporal knowledge graph question answering (tkgqa) @xcite @xcite @xcite , table question answering @xcite , or heterogeneous source question answering @xcite can also be used for evaluation, but they share similar limitations as the aforementioned datasets. to address the aforementioned limitations, we conduct an evaluation based on event temporal questions and establish a unified taxonomy. furthermore, we introduce more challenging compound questions to further increase the complexity and diversity of reasoning. based on this taxonomy, we propose the etrqa benchmark to comprehensively evaluate event temporal reasoning abilities. ## potential impact we believe that enhancing event temporal reasoning abilities is crucial for applying large language models (llms) to various real-world tasks related to temporal events, such as event timeline summa-rization @xcite . our work will have an impact on temporal reasoning in the following areas: dataset for the unified taxonomy of event temporal questions that we propose, we use it to guide the creation of more comprehensive question templates and the process of question generation. in practice, datasets can also be constructed through crowdsourcing or llm synthesis, and the taxonomy we present will serve as a key guide for both of these approaches. model training and evaluation our experimental results show that llms still do not perform well in solving certain types of reasoning tasks. therefore, our dataset can be introduced during the model's pre-training or post-training phases to enhance its learning of event temporal reasoning. furthermore, our dataset can not only be used for a comprehensive evaluation of llms' event temporal reasoning abilities but also to assess the impact of specific ability enhancements (e.g., mathematical reasoning) on the model's performance in event temporal reasoning tasks. retrieval-enhanced generation applying event temporal reasoning to real-world scenarios often requires integrating context retrieval. our paper presents a novel approach for retrieval-enhanced event temporal reasoning research, specifically decoupling reasoning from retrieval. our experimental results provide the upper bound for the event temporal reasoning abilities of some mainstream models, and when these models are combined with retrieval systems, the key objective will be to optimize the retrieval system to help the models achieve this upper bound. ## conclusion to comprehensively evaluate the event temporal reasoning abilities of llms, we develop a unified taxonomy for event temporal questions and propose a comprehensive question-answering benchmark named etrqa, by inheriting and expanding the evaluation content of existing datasets, as well as designing various types of compound questions. we conduct a thorough evaluation of two industryleading llm series on etrqa. the experimental results indicate that the etr abilities of llms still have room for improvement, particularly in reasoning involving time spans, reasoning for compound questions, and reasoning with fine temporal granularity.