Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning? (2024)

Abstract: Temporal reasoning is fundamental for large language models (LLMs) to comprehend the world. Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections. In this paper, we introduce CoTempQA, a comprehensive co-temporal Question Answering (QA) benchmark containing four co-temporal scenarios (Equal, Overlap, During, Mix) with 4,748 samples for evaluating the co-temporal comprehension and reasoning abilities of LLMs. Our extensive experiments reveal a significant gap between the performance of current LLMs and human-level reasoning on CoTempQA tasks. Even when enhanced with Chain of Thought (CoT) methodologies, models consistently struggle with our task. In our preliminary exploration, we discovered that mathematical reasoning plays a significant role in handling co-temporal events and proposed a strategy to boost LLMs’ co-temporal reasoning from a mathematical perspective. We hope that our CoTempQA datasets will encourage further advancements in improving the co-temporal reasoning capabilities of LLMs.

## introduction recent advanced large language models (llms) like @xcite have shown impressive capabilities in understanding, generating, and reasoning about natural language @xcite @xcite . despite their advancements, these models fall short in mastering temporal reasoning @xcite , which is fundamental for humans to comprehend the world and distinguish daily events @xcite @xcite , requiring a complex integration of capabilities, involving implicit arithmetic calculations @xcite , understanding logical implications @xcite , and leveraging extensive world knowledge @xcite . current studies in temporal reasoning mainly focus on time-sensitive question-answering (tsqa). @xcite first introduced the timeqa datasets, constructing time-evolving facts for a given subject and formulating questions based on the specific timestamp within the evolutionary facts. templama @xcite extracted structured facts from the wikidata knowledge base @xcite for closedbook tsqa. furthermore, tempreason @xcite translated explicit temporal expressions into the implicit event information within questions, offering a more comprehensive evaluinterpretation relation the datasets mentioned above provide a straightforward way to evaluate llms' capabilities in temporal reasoning. however, as llms evolve, there is an urgent need to evaluate their proficiency in more realistic scenarios. as shown in figure 1 , the reality might present a more intricate and multifaceted nature, involving concurrent events and complex temporal interconnections over time @xcite . current datasets mainly question single or isolated events and might not fully reflect the realistic temporal characteristics. therefore, we create the co-temporal qa (cotemp-qa) datasets to complement existing corpora by focusing on the concurrent nature of time and cotemporal relations in real-world situations. experiments conducted on both closed-book and openbook qa settings across 14 large language models reveal that even the advanced model gpt-4 is well below a satisfactory co-temporal reasoning performance. specifically, gpt-4 achieves an overall score of 54.7, and the best open-source llm is 30.1, which significantly falls behind the human performance of 92.8. we also observe that the representative reasoning enhancement strategies, e.g., chain-of-thought (cot) @xcite , fail to consistently improve and even reduce the temporal reasoning capabilities of llms in some scenarios. throughout the investigation on our cotemp-qa, we observed that mathematical reasoning plays a crucial role in handling co-temporal events. building on this insight, we propose a simple but effective math-reasoning cot (mr-cot) strategy to boost the co-temporal reasoning capability of llms, achieving a remarkable 10.8 point improvement over existing baselines. however, it is important to note that there remains a nonnegligible gap between the performance of our proposed mr-cot and human-level reasoning in handling complex, concurrent temporal relations. we hope our research could inspire more great works to improve the co-temporal ability of llms. ## potential impact we believe that it is crucial to enable llms to understand events that occur simultaneously or overlap in time. the potential impact of improved cotemporal reasoning on downstream applications includes the following: ## conclusion in this paper, we propose the cotempqa datasets to facilitate the investigation of under-explored cotemporal reasoning problems for large language models. extensive experiments have shown a significant gap between existing advanced llms and human-level performance, even with the enhancement of reasoning approaches. we also discover that mathematical reasoning is crucial for understanding co-temporal events and propose a mathbased strategy to improve llms' co-temporal reasoning. reasoning on concurrent and intricate temporal relations remains an open research question, and we hope more enhancement to develop upon our cotempqa datasets.