Large Language Models Can Learn Temporal Reasoning (2024)

Abstract: While large language models (LLMs) have demonstrated remarkable reasoning capabilities, they are not without their flaws and inaccuracies. Recent studies have introduced various methods to mitigate these limitations. Temporal reasoning (TR), in particular, presents a significant challenge for LLMs due to its reliance on diverse temporal concepts and intricate temporal logic. In this paper, we propose TG-LLM, a novel framework towards language-based TR. Instead of reasoning over the original context, we adopt a latent representation, temporal graph (TG) that enhances the learning of TR. A synthetic dataset (TGQA), which is fully controllable and requires minimal supervision, is constructed for fine-tuning LLMs on this text-to-TG translation task. We confirmed in experiments that the capability of TG translation learned on our dataset can be transferred to other TR tasks and benchmarks. On top of that, we teach LLM to perform deliberate reasoning over the TGs via Chain-of-Thought (CoT) bootstrapping and graph data augmentation. We observed that those strategies, which maintain a balance between usefulness and diversity, bring more reliable CoTs and final results than the vanilla CoT distillation.

## introduction as one of the fundamental abilities, temporal reasoning (tr) plays an important role in human perception. it is not just about understanding basic concepts such as ordering or duration; it extends to more intricate aspects, e.g., task planning or causal relation discovery. recently, large language models (llms) @xcite @xcite have emerged with some reasoning capabilities @xcite . however, there is observation that they still can not perform tr sufficiently well (wang and zhao, 2023; @xcite , preventing their applications from solving complex real-world problems. in particular, tr requires a combination of various skills including mathematical and logical reasoning as well as commonsense knowledge @xcite @xcite . recent works mainly adopt general approaches to investigate and improve the tr capability of llms. for example, @xcite @xcite benchmark the leading llms on different tr tasks with standard input/output prompt, few-shot in-context learning (icl) and chain-of-thought (cot) reasoning @xcite . similarly, @xcite designs several specific types of prompts as prompt tuning. @xcite introduce predefined python programs/rule-based templates to perform supervised fine-tuning (sft). in addition, @xcite adopt some extra strategies, which include specific pre-training, instruction tuning and reinforcement learning. despite the effectiveness of such methods, they either ignore or not explicitly involve the intrinsic nature of tr. humans perform complex tr on a timeline of events which are aligned with the entities and relations. these temporal concepts (e.g., ordering, duration, frequency, typical time) are then rigorously defined based on the timeline information. in other words, the aligned timeline (more generally, the temporal graph, tg) serves as a latent representation to help humans understand the patterns in tr. however, due to the lack of ground truth, the high-quality tg translation is a challenging task for most tr benchmarks. to solve this problem, we propose a synthetic dataset (tgqa), which is fully controllable and requires minimal supervision. we demonstrate the capability of tg translation learned on our dataset can be transferred to other tr tasks and benchmarks. given a reliable tg, the key challenges of teaching tr to llms include: (1) how can one introduce the necessary arithmetic and commonsense knowledge involved in tr? prior work @xcite shows that explicitly introducing knowledge into context enhances the performance of llms. in this paper, we first identify all the valid time expressions, and then generate related knowledge (e.g., temporal relation and time gap between the timestamps, and the relative order of the gaps). to be specific, our contributions are summarized as follows: temporal graph [sub; rel; obj; start/end; time]: [1] (john thompson was born in weston) starts at 1921; [2] (john thompson owned pearl network) starts at 1942; [3] (sophia parker was married to john thompson) starts at 1947; [4] (john thompson was married to sophia parker) starts at 1947; [5] (sophia parker was married to john thompson) ends at 1953; [6] (john thompson was married to sophia parker) ends at 1953; [7] (john thompson owned pearl network) ends at 1967; [8] (john thompson died in riverside) starts at 1988; [9] (sophia parker died in lancaster) starts at 1995. graph-based story (from gpt-3.5): once upon a time in the quaint town of weston, a baby boy named john thompson was brought into the world in the year 1921. growing up, he had a vibrant spirit and an adventurous soul. • • • graph-based qas (from rule-based python script): q1: which event started first, (john thompson owned pearl network) or (john thompson was married to sophia parker)? a1: (john thompson owned pearl network). q2: true or false: event (john thompson owned pearl network) was longer in duration than event (sophia parker was married to john thompson)? a2: true. ## tg-llm motivated by human perception, we propose a new paradigm called tg-llm. we first translate the text into a temporal graph (tg), and then guide the llm to perform deliberate reasoning on it. ## experiments we aim to answer the following research questions in our experiments: (1) can our strategies (cot bootstrapping and graph data augmentation) bring more reliable reasoning over tgs? (2) can our twostep framework lead to better tr performance? (3) do these learned capabilities of tr generalize to other tasks? ## conclusion tg-llm, a novel framework for lms, has been proposed to improve their performance on temporal reasoning. to produce reliable final answers, our framework equips llms with the temporal graph and intermediate reasoning steps. extensive experiments indicate that tg-llm achieves better performance than existing pipelines. an interesting direction for future work is to extend it to more complex applications such as inductive and abductive reasoning. due to the graph structure and capability of deliberate reasoning, it is promising to improve the model performance on these tasks as well.