TRAM : Benchmarking Temporal Reasoning for Large Language Models (2024)

Abstract: Reasoning about time is essential for understanding the nuances of events described in natural language. Previous research on this topic has been limited in scope, characterized by a lack of standardized benchmarks that would allow for consistent evaluations across different studies. In this paper, we introduce TRAM, a temporal reasoning benchmark composed of ten datasets, encompassing various temporal aspects of events such as order, arithmetic, frequency, and duration, designed to facilitate a comprehensive evaluation of the TeR capabilities of large language models (LLMs). We evaluate popular LLMs like GPT-4 and Llama2 in zero-shot and few-shot scenarios, and establish baselines with BERT-based and domain-specific models. Our findings indicate that the best-performing model lags significantly behind human performance. It is our aspiration that TRAM will spur further progress in enhancing the TeR capabilities of LLMs.

## introduction temporal reasoning is fundamental for humans to understand the world and distinguish between everyday events. for instance, when given the activities "watching a movie" and "watching a sunset", we intuitively recognize that, though both are time-bound, a movie typically lasts longer than a sunset. moreover, while movies can be watched repeatedly, sunsets transpire once a day. such innate comprehension is not just about sequencing events or understanding durations; it extends to more intricate aspects of time, allowing us to make sense of complex narratives and the causality of events. despite advancements in natural language processing (nlp) and the advent of large language models @xcite @xcite , mastering temporal reasoning remains a significant challenge due to its intricate nature, the variability of temporal expressions, and the need for contextual understanding. recent work in temporal reasoning (ter) has primarily focused on time-sensitive questionanswering @xcite @xcite , demonstrating that despite significant advancements in nlp, current language models have yet to reach humanlevel performance in this domain. furthermore, these studies, while addressing explicit temporal elements such as order, duration, and time-event relations, overlook more complex aspects of ter, like temporal narratives and causality. importantly, the establishment of a unified framework including broad facets of ter has not yet been achieved. to facilitate research in this direction, we present the temporal reasoning for large language model benchmark (or tram for short), a collection of ten temporal reasoning tasks. these tasks range from foundational understanding (e.g., duration, frequency) to advanced temporal interpretations and computations (e.g., arithmetic, causality). each task consists of one or more subtasks, all of which are specifically crafted to assess a model's ter capabilities across varying levels of understanding and difficulty. in total, our benchmark includes 38 distinct subtasks, comprising a total of 526.7k ques-tions. answers have been derived through a combination of expert annotations and programmatic generation. diverging from prior ter research and in line with @xcite , our questions are formatted as straightforward multiple-choice tests rather than generative tasks, thereby more appropriately evaluating llms. example questions in tram are shown in figure 1 . to gain deeper insight into the ter challenges posed by tram, we extensively evaluate several prominent language models, including bert (kenton and toutanova, 2019), roberta @xcite , the domain-specific ter model rst @xcite , and recent llms including llama2 @xcite , gemini pro @xcite , @xcite . we use limited training data to fine-tune bertstyle and rst models. llms are evaluated using standard and chain-of-thought prompting under zero-shot and few-shot learning paradigms. our results indicate that gpt-4 excels in most tasks, achieving an average accuracy of up to 84.4%. moreover, we observe notable performance disparities across tasks among the models. despite the impressive performance of gpt-4, it falls short of human proficiency by over 10%, highlighting significant room for llms to improve their ter capabilities. manual error analysis shows that models struggle with nuanced understanding and interpreting implicit cues across all task categories. in summary, our contributions are threefold: our proposal for a comprehensive ter benchmark builds on the evolution of datasets in this domain while addressing the lack of a unified system for evaluation. the modern nlp landscape sets the stage for a nuanced evaluation of both pretrained models and llm paradigms. temporal reasoning benchmarks in the realm of ter, several datasets have emerged to address distinct challenges. early benchmarks, such as time-bank @xcite , predominantly focused on temporal relations. tempeval-3 (uz- @xcite broadened the scope by introducing multiple tasks, including temporal entity extraction and temporal relation extraction. recently, there has been a surge in the development of time-sensitive question-answering datasets like mctaco @xcite , time-sensitive qa @xcite , templama @xcite , tempreason @xcite , and menatqa @xcite . however, these datasets often specialize in narrower aspects of ter, such as duration, frequency, or event-time relations. in contrast, our benchmark offers a comprehensive scope of ter, addressing diverse levels and dimensions of understanding about time, aiming to provide a more complete representation of ter challenges than previously available datasets. ## experiments in our evaluation, we compare the performance of prevalent llms across all datasets and analyze the mistakes they make. we report the best results after multiple runs for each experimental setting. ## discussion we introduce tram, a comprehensive benchmark spanning ten diverse tasks, to evaluate the temporal reasoning of llms. the contrasting performances across models emphasize the significance of experimental strategies and shed light on the intrinsic challenges. this benchmark serves as a tool for researchers to identify model limitations and guide further advancements in this domain. implications of tram the introduction of tram establishes a new paradigm for probing the temporal reasoning capabilities of llms. unlike previous benchmarks, which often offered fragmented insights into temporal tasks, tram provides a comprehensive system. this allows for a unified evaluation of how models comprehend both rudimentary temporal concepts and complex temporal narratives. the differentiation in task complexity within tram elucidates the various stages of temporal understanding. in particular, tram underscores challenges like decoding implicit temporal cues and navigating intricate temporal relationships, providing a roadmap for future improvements in llms in this area. future directions tram has initiated a step towards evaluating llms' temporal reasoning capabilities, but there are further avenues to explore. going forward, we will experiment with more test data and refine tailored prompting techniques for each task through iterative testing. moreover, we plan to expand the benchmark to include varied question formats. for generative tasks, this might encompass short answers and summarization. even within mcqs, we intend to incorporate questions that may have one or more correct answers, allowing for a more comprehensive evaluation. we also plan to fine-tune leading open-source llms, such as llama3, phi-3, and gemma, on these tasks to develop domain-specific expert models. these efforts aim to create tailored llms that can more effectively understand and reason about time across various contexts. ## limitations while tram sets a holistic standard for ter assessment, we acknowledge its limitations. one primary concern is the subset evaluation of the test set, which may not reflect the full spectrum of llms' ter capabilities. furthermore, the mcq format may allow llms to guess randomly, skewing performance evaluations. moreover, textual questions may not capture the entire complexity of ter tasks, as real-world scenarios often integrate multi-modal cues such as images and videos.