Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models (2023)

Abstract: Temporal reasoning represents a vital component of human communication and understanding, yet remains an underexplored area within the context of Large Language Models (LLMs). Despite LLMs demonstrating significant proficiency in a range of tasks, a comprehensive, large-scale analysis of their temporal reasoning capabilities is missing. Our paper addresses this gap, presenting the first extensive benchmarking of LLMs on temporal reasoning tasks. We critically evaluate 8 different LLMs across 6 datasets using 3 distinct prompting strategies. Additionally, we broaden the scope of our evaluation by including in our analysis 2 Code Generation LMs. Beyond broad benchmarking of models and prompts, we also conduct a fine-grained investigation of performance across different categories of temporal tasks. We further analyze the LLMs on varying temporal aspects, offering insights into their proficiency in understanding and predicting the continuity, sequence, and progression of events over time. Our findings reveal a nuanced depiction of the capabilities and limitations of the models within temporal reasoning, offering a comprehensive reference for future research in this pivotal domain.

## introduction temporal reasoning @xcite stands as a fundamental pillar of human communication and understanding, acting as a guiding force in our interpretation of events and narratives. comprehending natural language involves a deep understanding of time and its facets, which include the duration, sequence, and frequency of events. within the realm of natural language understanding (nlu), the ability to reason with temporal information has emerged as a significant area of research. this exploration has witnessed substantial strides in recent years, with numerous researchers contributing to this growing body of knowledge @xcite . temporal reasoning extends beyond the mere awareness of time -it requires a nuanced understanding of time's relation to events and actions. for instance, humans inherently know that a vacation usually lasts longer than a walk and occurs less frequently. therefore, the quest to enhance temporal reasoning within ai systems is of paramount importance. a language model with a robust understanding of temporal context is primed to perform better on downstream natural language processing (nlp) tasks such as storytelling @xcite , natural language inference @xcite , timeline understanding @xcite , and user status tracking @xcite . the integration of temporal reasoning not only enhances the nuances of these applications but also holds the potential to significantly improve the overall performance of ai systems. large language models (llms) @xcite have demonstrated remarkable capabilities in a variety of tasks, ranging from commonsense reasoning @xcite to arithmetic problemsolving @xcite . despite the abundance of studies benchmarking llms on these tasks, there is a conspicuous absence of comprehensive, large-scale analysis focusing on benchmarking the models on temporal reasoning. since temporal reasoning represents a crucial aspect of human comprehension, influencing our interpretation and response to a myriad of scenarios, hence the lack of large-scale benchmarking for llms on temporal reasoning tasks is a significant gap in our understanding of these models' capabilities. acknowledging this considerable gap, we have undertaken the first extensive benchmarking of llms on temporal reasoning tasks. (1) our comprehensive analysis encompasses 6 datasets, leveraging 8 different language models. the language models have been tested through 3 different prompting strategies, aiming to explore the breadth and depth of their temporal reasoning proficiency. moreover, we have also included 2 code generation lms, further broadening the spectrum of our analysis. we further analyze the llms on varying temporal aspects, offering insight into their proficiency in understanding and predicting the continuity, sequence, and progression of events over time. in particular, we investigate the following research questions: ## benchmark setup in the following sections, we provide the details of the datasets, tasks, prompting techniques, and language models used in our research study. the datasets and tasks primarily pertain to temporal reasoning tasks, requiring the models to display an understanding and reasoning in time-sensitive contexts and situations. in terms of language models, we examine a diverse set of models pretrained with different strategies, including both standard and code generation lms. ## results in the following subsection, we initiate our discussion by providing a succinct overview of the performance of various llms, code generation lms, and diverse prompting techniques across multiple datasets. we then advance to a more in-depth exploration in sec. 4.2, scrutinizing the performance of different llms in various temporal tasks. finally, we turn our attention to analyzing different temporal characteristics of the best-performing llms from our previous assessments (sec. 4.3). ## conclusion in this paper, we aimed to bridge a critical knowledge gap by conducting a comprehensive benchmarking of llms on temporal reasoning tasks. our thorough analysis has shed light on certain limitations in the ability of llms to reason temporally. specifically, we have identified areas where llms struggle, such as comprehending the temporal states of events, accurately reasoning over precise timings, managing multiple temporal events, and predicting future events. by highlighting these