Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives (2024)

Abstract: Reasoning about time and temporal relations is an integral aspect of human cognition, essential for perceiving the world and navigating our experiences. Though large language models (LLMs) have demonstrated impressive performance in many reasoning tasks, temporal reasoning remains challenging due to its intrinsic complexity. In this work, we first study an essential task of temporal reasoning—temporal graph generation, to unveil LLMs’ inherent, global reasoning capabilities. We show that this task presents great challenges even for the most powerful LLMs, such as GPT-3.5/4. We also notice a significant performance gap by small models (< 10B) that lag behind LLMs by 50%. Next, we study how to close this gap with a budget constraint, e.g., not using model finetuning. We propose a new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), that first converts the events set to a Python class, then prompts a small model to generate a temporally grounded narrative, guiding the final generation of a temporal graph. Extensive experiments showcase the efficacy of NoT in improving various metrics. Notably, NoT attains the highest F1 on the Schema-11 evaluation set, while securing an overall F1 on par with GPT-3.5. NoT also achieves the best structural similarity across the board, even compared with GPT-3.5/4.

## introduction temporal reasoning is essential for humans to perceive the world, understand daily communications, and interpret the temporal aspects of experiences @xcite . the recent advent of large language models (llms) has garnered substantial attention to their impressive performance in various reasoning tasks, such as arithmetic reasoning (cobbe et al., 2021; zhong 1 our code is available at https://github.com/ launchnlp/not . @xcite , where the input is a goal and a set of unordered events. in this work, to better unleash the pre-training power of llms trained with a mixture of text and code, we cast tgg as a code completion task. @xcite and commonsense reasoning @xcite . nonetheless, few llms exist to handle temporal reasoning well @xcite @xcite , due to the task's inherent complexity, mingled with implicit logical inference and the necessity for profound world knowledge. to gain deeper insights, the research community mainly focuses on two ends along the spectrum: either a simple relation extraction task that orders a pair of events @xcite , or a perplexing commonsense understanding task demanding multifaceted reasoning skills beyond the mere temporal aspect @xcite @xcite . worse still, the former is limited to a local scope spanning two adjacent sentences only and fails to account for the significance of global temporal relations, leading to overly optimistic results @xcite . therefore, neither setup provides a clear understanding of llms' true temporal reasoning abilities. in this work, we aim to unveil the inherent, global temporal reasoning capabilities of llms, evaluating them in isolation free from confounding factors, and addressing the limitations of previous studies which only focused on local contexts. we first introduce a task of temporal graph generation (tgg; figure 1 ): given a high-level goal foot_0 t (e.g., business change) and a set of events v, the objective is to produce a temporal graph g(v, e) where a directed edge in e reveals the temporal order between events. though this specific notion of tgg is new, many of its applications are not. in this work, we specifically study tgg in order to evaluate and improve the temporal reasoning capability, since tgg is deemed a major bottleneck when llms perform temporal reasoning. with tgg, we put forth the first research question. rq1: what is the temporal reasoning capability of popular llms? prior work @xcite shows a huge gap between ai systems and human performance on various temporal understanding tasks. additionally, there is a notable performance disparity between proprietary llms (e.g., gpt-4) and open-weights llms, particularly those with fewer than 10 billion parameters (henceforth, small llms). our study on temporal reasoning reveals a similar trend and identifies the existence of both gaps, as demonstrated in extensive experiments across three evaluation benchmarks of diverse genres reveal six interesting findings: 1) small llms struggle with temporal reasoning even with few-shot examples; 2) cot is also ineffective at temporal reasoning, in line with existing finding @xcite ; 3) gpt-4 sometimes falls off the throne due to alignment, when answering sensitive queries; 4) not is a powerful tool to assist small llms to catch up with or even surpass gpt-3.5, and presents strong compatibility with various base llms; 5) the temporally grounded narratives are significant in improving llms' temporal reasoning process; 6) ai systems are far from mastering temporal reasoning, trailing the human baseline by 30 f1 points. we also analyze the impact of shot numbers and perform a holistic evaluation of reference narratives in few-shot examples. 5-shot is found to be the sweet spot for temporal reasoning, after which the performance plateaus, likely due to long-context challenge. we identify three key characteristics of reference narratives for them to avail small llms most: conciseness, simplicity, and factuality. ## experiment in this work, we focus on temporal graph generation (tgg), an essential task of temporal reasoning. here, we discuss datasets, experimental setup, baselines, and evaluation metrics. we provide additional implementation details in appendix a. ## conclusion in this paper, we assess the inherent, global temporal reasoning capabilities of llms, by studying the core challenge of temporal reasoningtemporal graph generation (tgg). to this end, we propose narrative-of-thought (not), a novel prompting technique tailored for temporal reasoning. concretely, with few-show narrativeaware demonstrations as references, not prompts llms to first generate a temporally grounded narrative and then sort the input events topologically into a temporal graph, by manipulating the generation in code space. extensive experiments showcase not's effectiveness, demonstrated by its superior performance over gpt-3.5 on multiple metrics, as well as the compatibility of not with various llms.