Consistent Discourse-level Temporal Relation Extraction Using Large Language Models (2025)

Abstract: Understanding temporal relations between events in a text is essential for determining its temporal structure. Recent advancements in large language models (LLMs) have spurred research on temporal relation extraction. However, LLMs perform poorly in zero-shot and few-shot settings, often underperforming smaller fine-tuned models. Despite these limitations, little attention has been given to improving LLMs in temporal structure extraction tasks. This study systematically examines LLMs’ ability to extract and infer discourse-level temporal relations, identifying factors influencing their reasoning and extraction capabilities, including input context, reasoning process and ensuring consistency. We propose a three-step framework to improve LLMs’ temporal relation extraction capabilities: context selection, prompts inspired by Allen’s interval algebra (Allen, 1983), and reflection-based consistency learning (Shinn et al., 2024). Our results show the effectiveness of our method in guiding LLMs towards structured processing of temporal structure in discourse.

## introduction temporal relations describe the interaction between events along the temporal dimension, forming a crucial aspect of natural language understanding. humans can encode temporal information in language, using various linguistic expressions to convey time-related concepts. beyond mere extraction of explicit temporal markers such as before and after, humans leverage their linguistic competence and cognitive reasoning to interpret time in language, constructing a coherent mental timeline of events. this allows them to infer implicit temporal relations, even when direct temporal cues are absent or ambiguous @xcite @xcite . in contrast, llms encode textual information in a high-dimensional latent space, capturing intricate semantic patterns without an explicitly defined mechanism for understanding time. extracting temporal structures is essential in a wide range of nlp applications, including document summarisation @xcite , storyline construction @xcite , and reading comprehension @xcite . in recent years, llms' impressive text generation and processing abilities have attracted significant attention, directing extensive research into their capabilities in various ranges of tasks. relation extraction has emerged as a common task for evaluating llms, with studies showing that llms often underperform smaller fine-tuned models in zero-shot or few-shot settings @xcite @xcite . this issue is particularly pronounced in temporal relation extraction, where llms struggle with zero-shot and few-shot settings @xcite @xcite . three critical challenges in discourse-level temporal relation tasks remain underexplored: (1) context selection: recent studies leveraging llms often rely on two common strategies-feeding the entire document or using sliding windows to extract context-both of which introduce significant noise. (2) performance: even in the fine-tuning setting, llms still exhibit a notable performance gap compared to the methods based on smaller-scale pre-trained language models @xcite , such as bert @xcite and roberta @xcite . (3) consistency: while most recent work focuses on achieving high f1 scores, limited attention has been given to evaluating and improving model consistency. this study, motivated by prior work, is dedicated to tackling the above-mentioned challenges. as noted by @xcite , extracting complete temporal structures from discourse-level texts is complex, labour-intensive, and costly for humans. enhanced llm performance in this task can support corpus annotation, significantly reducing time and financial costs while advancing downstream nlp applications. we propose a three-step framework to improve llms' temporal relation extraction capabilities: context selection, prompts inspired by allen's interval algebra @xcite , and reflection-based consistency learning @xcite . context selection minimises noise by focusing on relevant text and addressing llms' challenges with long contexts and distant event relations. prompt engineering inspired by allen's interval algebra enables structured reasoning about temporal attributes like start and end times. reflection-based consistency learning iteratively identifies and corrects inconsistencies, teaching llms temporal coherence. the ablation study demonstrates that our context selection strategy is effective and computationally efficient. besides, our novel self-reflection strategy improves the consistency of model predictions and offers a new perspective for addressing consistency challenges within the field. to the best of our knowledge, we are the first to use self-reflection to ensure consistency in the discourse-level temporal relation extraction task. our method enhances the temporal relation extraction ability of llms and achieves performance that surpasses the state-ofthe-art models. ## related work before the introduction of tddiscourse (tdd) @xcite , most studies focused on locallevel corpora like timebank @xcite and timebank-dense @xcite . ## method to improve llms' performance in temporal relation extraction, we propose a three-step framework: (1) context selection, (2) instruction and chain-of-thought (cot) @xcite prompt design and (3) consistency learning. figure 2 shows an overview of our method.