T ime B ench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models (2024)

Abstract: Grasping the concept of time is a fundamental facet of human cognition, indispensable for truly comprehending the intricacies of the world.Previous studies typically focus on specific aspects of time, lacking a comprehensive temporal reasoning benchmark.To address this, we propose TimeBench, a comprehensive hierarchical temporal reasoning benchmark that covers a broad spectrum of temporal reasoning phenomena.TimeBench provides a thorough evaluation for investigating the temporal reasoning capabilities of large language models.We conduct extensive experiments on GPT-4, LLaMA2, and other popular LLMs under various settings.Our experimental results indicate a significant performance gap between the state-of-the-art LLMs and humans, highlighting that there is still a considerable distance to cover in temporal reasoning.Besides, LLMs exhibit capability discrepancies across different reasoning categories.Furthermore, we thoroughly analyze the impact of multiple aspects on temporal reasoning and emphasize the associated challenges.We aspire for TimeBench to serve as a comprehensive benchmark, fostering research in temporal reasoning.Code and data are available at https://github.com/zchuz/TimeBench.

## introduction time flies over us, but leaves its shadow behind. understanding time is a crucial part of human comprehension of the world. envision the blossoming of flowers, and you'll associate it with the arrival of spring. the ponder within it encompasses the intricate interplay of world knowledge, causality, and event temporal relationships. temporal reasoning, in contrast to reasoning of a singular nature, comes with inherent complexity, encompassing implicit arithmetic, logical implications, and world knowledge. it is a form of integrated reasoning built upon foundational reasoning like mathematical and logical reasoning @xcite @xcite . recently, large language models (llms) have demonstrated remarkable performance in complex reasoning @xcite @xcite @xcite , but their performance in temporal reasoning has not yet been extensively explored. recent research for temporal reasoning typically focuses only on a few aspects, such as temporal commonsense or temporal question answering @xcite @xcite . due to the inherent complexity of temporal reasoning, it is challenging to accurately measure models' temporal reasoning capabilities based on limited aspects. to address this issue, we propose timebench, a comprehensive and hierarchical temporal reasoning benchmark. specifically, drawing inspiration from the human cognitive process of transitioning from abstraction and concreteness to integration @xcite , we categorize temporal reasoning into three levels: symbolic temporal reasoning, commonsense temporal reasoning, and event temporal reasoning. these levels respectively represent understanding abstract time expression, grasping concrete world knowledge, and integrating and applying this knowledge in real-world scenarios. timebench comprises 10 tasks with 16 sub-tasks, covering a broad spectrum of temporal reasoning phenomena. besides, prior work typically features only a single task form, too simplistic to capture the model's performance. in contrast, we incorporate four distinct task forms, offering a more realistic simulation of challenges. to quantify the temporal reasoning capabilities of contemporary llms, we extensively assess widely-used llms, including proprietary models such as chatgpt @xcite and @xcite , as well as open-source like llama2 @xcite , vicuna-1.5 @xcite , mistral @xcite , baichuan2 @xcite , chatglm3 @xcite and @xcite . we conduct experiments under zero-shot and fewshot settings, combining commonly used reasoning techniques, chain-of-thought prompting @xcite . the experimental results suggest that gpt-4 outperforms other models, showcasing strong temporal reasoning capabilities, as shown in figure 1 . nevertheless, there is still a considerable gap between the strongest models and humans. on the contrary, open-source models show inferior performance in temporal reasoning, attributed to shortcomings in abstract time understanding, temporal relations modeling, and a lack of temporal commonsense. in addition, we also observe that chain-of-thought prompting does not yield a consistent improvement in performance. these findings indicate that there is still significant room for improvement in models' temporal reasoning capabilities. moreover, we have conducted a thorough analysis of the deficiencies and obstacles faced by models in temporal reasoning. we aspire for temporal reasoning to garner increased attention within the research community. our contributions can be summarized as follows: ## methodology we perform evaluations using the prompt-based approach, including standard prompting and chainof-thought prompting. experiments are conducted under both zero-shot and few-shot settings. ## conclusion temporal reasoning entails inherent diversity and complexity. the lack of a comprehensive benchmark makes it challenging to quantify llms' temporal reasoning capabilities. in this work, we present timebench, a comprehensive and hierarchical benchmark for llm temporal reasoning, tailored to mirror temporal reasoning in complex scenarios. we conduct extensive experiments on state-of-the-art llms to investigate their temporal reasoning capabilities. our findings indicate a substantial gap between state-of-the-art llms and human performance, emphasizing the need for further research in this area. moreover, we provide a meticulous analysis and discussion, outlining the current challenges that models face and suggesting potential directions for improvement.