Can LLM s Help Encoder Models Maintain Both High Accuracy and Consistency in Temporal Relation Classification? (2025)

Abstract: Temporal relation classification (TRC) demands both accuracy and temporal consistency in event timeline extraction. Encoder-based models achieve high accuracy but introduce inconsistencies because they rely on pairwise classification, while LLMs leverage global context to generate temporal graphs, improving consistency at the cost of accuracy. We assess LLM prompting strategies for TRC and their effectiveness in assisting encoder models with cycle resolution. Results show that while LLMs improve consistency, they struggle with accuracy and do not outperform a simple confidence-based cycle resolution approach. Our code is publicly available at: https://github.com/MatufA/timeline-extraction .

## introduction extracting event timelines from text is a key natural language processing (nlp) task, organizing events chronologically based on their relative occurrence rather than absolute timestamps. a broader definition by @xcite describes a timeline as a data structure that arranges events and times in a total order. timelines have a wide range of practical applications, even when considering events alone. for instance, @xcite demonstrated how timelines can be used to process government decision letters, extracting and organizing events for improved understanding. another example is in the medical domain @xcite : given a patient's textual medical record-or a collection of such records-it becomes valuable to extract a timeline of relevant medical events to summarize and visualize their journey. timeline extraction typically involves five steps: (1) event detection, identifying relevant events, often treating all verbs as events; (2) anchoring, selecting events for comparison; (3) temporal relation classification (trc), assigning relations to pairs; (4) graph construc-tion, combining pairwise relations into a temporal graph; and (5) timeline extraction, deriving a timeline from the graph. various methods have been proposed for extracting timelines from temporal graphs @xcite @xcite . @xcite proposed a method for extracting event timelines from documents annotated with the full timeml scheme @xcite , which defines 13 temporal relation types. however, modeling all 13 relations is complex and often results in temporal inconsistencies. to address the complexity of timeml's full relation set, several datasets focus on simplified subsets. a widely used resource for trc is matres @xcite , which reduces the relation types to three deterministic labels-before, after, and equals-along with a vague category for uncertain cases. these labels are assigned to a subset of all possible event pairs, a design choice intended to improve annotation consistency and reduce ambiguity. despite this simplification, temporal inconsistencies can still arise, particularly with models following a pairwise approach: predicting relations independently for each event pair without considering previously predicted labels. for example, a model might predict: a before b, b before c, and mistakenly, a after c. the last relation contradicts the others and creates a temporal cycle, which complicates efforts to derive a consistent, linear event timeline. a real instance of such a cycle, predicted on a matres document, is illustrated in figure 1 . large language models (llms) have achieved state-of-the-art performance across many nlp tasks. however, previous studies @xcite have shown that generative llms underperform compared to encoder-based models on the trc task as defined in matres. the advantage of llms lies in their ability to encode document-wide information flexibly, which enables them to generate an entire temporal graph in a single step. this capability, recently termed global trc, offers the potential to reduce temporal inconsistencies by considering all event pairs jointly. building on prior work in trc, non-fine-tuned generative llms still lag behind smaller supervised models that follow the pairwise approach. however, llms' ability to generate the entire temporal graph in a single inference step offers a key advantage: the potential to reduce temporal inconsistencies, a common issue in pairwise models. therefore, in this work we make two main contributions: ## related work temporal relation classification has primarily been addressed using fine-tuned, relatively small encoder-based language models, typically following a pairwise approach in which each event pair is labeled independently. a key limitation of the pairwise approach is its tendency to produce globally inconsistent outputs. since these models make independent predictions for each pair of events, they do not take previously predicted labels into account during inference. this lack of global awareness can result in contradictions, such as temporal cycles, which undermine the coherence of the predicted temporal structure and ultimately hinder accurate timeline construction. despite this limitation, numerous well-established encoder-based methods have been proposed to tackle pairwise trc. these include approaches that leverage contextualized representations and joint inference strategies to improve local and global consistency @xcite @xcite @xcite @xcite @xcite @xcite . while these models have contributed significantly to the field, the challenge of maintaining globally coherent temporal graphs remains a central concern in temporal relation classification. early efforts such as @xcite introduced a structured framework for trc by refining the task with better contextual representations and curated evaluation protocols. subsequent work expanded this by incorporating global constraints, as in @xcite , which applied joint inference to enforce temporal consistency across event graphs. similarly, @xcite proposed econet, which leveraged event graph structures and global coherence to improve document-level temporal reasoning. domain-specific applications have also driven innovation in trc, particularly in the clinical domain. @xcite addressed the challenges of trc in clinical texts, which often involve fragmented or incomplete narratives. their work demonstrated that specialized models and annotation schemes are necessary to adapt general trc methods to the clinical setting. @xcite , reframed trc as a boolean question answering task. by training a roberta model on yes/no questions formulated based on the annotation guidelines, they effectively simulated the human annotation process and achieved state-of-the-art results on the matres dataset. more recent work by @xcite introduced contempo, a large-scale benchmark for document-level temporal reasoning, combining distant supervision and llm-based annotation to enhance the breadth and realism of training data. several recent studies have also focused on extracting temporal structures beyond pairwise relations. @xcite and @xcite explored the prediction of document creation times (dct) and global temporal graphs, respectively, highlighting the importance of temporal anchoring in narrative understanding. @xcite and @xcite also tackled full timeline construction, proposing models that jointly identify events and infer their temporal relationships, often integrating external knowledge or reasoning modules. with llms becoming state-of-the-art in many tasks and offering more flexible input handling in a zero-shot setting, recent studies have explored different ways to use them for trc, both in pairwise and global settings. jain et al. (2023) evaluated a variety of llms (including standard and code-generation models) recent studies have begun exploring the use of zero-shot llms for trc, though most efforts have adhered to the traditional pairwise prediction framework @xcite @xcite . a more recent study @xcite proposes enhancing global consistency by prompting a strong generative llm to produce the entire graph of temporal relations in a single step. to address potential contradictions and instability of llms in generating consistent output, this approach incorporates a post-processing step based on the linear programming optimization framework introduced by @xcite , which enforces global coherence by resolving inconsistencies in the predicted temporal graph. together, these works form a comprehensive foundation for understanding the evolution of trc, from pairwise classification to global timeline construction, and from specialized supervised models to llm-based generalization. building on these efforts, we compare zero-shot llm accuracy and consistency across prompts and propose a simple, effective cycle-breaking method for encoders while maintaining accuracy. ## datasets our investigation of the trade-off between pairwise accuracy and global consistency is grounded in experiments on two datasets that represent distinct annotation paradigms. @xcite matres is a widely-used benchmark for trc which simplifies timeml's relations into four labels: before, after, equal, and vague. it employs a sparse annotation strategy, providing gold labels primarily for event pairs in close proximity (i.e., within two-sentence contexts) to enhance interannotation agreement. this sparsity directly impacts our evaluation: accuracy is computed using only the gold subset, while temporal consistency is measured over all generated relations. ## conclusions our study highlights both the promise and the current limitations of using llms for timeline extraction through trc. across extensive experiments, we observe a recurring inverse relationship between accuracy and consistency: as llms are pushed toward higher accuracy through prompt design and reasoning strategies, they tend to generate more globally inconsistent temporal graphs-often resulting in cycles. this trade-off mirrors classic precision-recall tensions in traditional machine learning and highlights the challenge of achieving both local accuracy and global coherence in zeroor few-shot generative settings. we also find that current llms struggle to generate complete and accurate temporal graphs in a single pass, even when using compact representations and chainof-thought reasoning. while supervised encoderbased models remain more accurate on annotated pairs, their pairwise prediction structure inherently introduces global inconsistencies unless followed by post-hoc constraints. interestingly, when evaluating llm-generated graphs with existing cycle resolution strategies, a simple confidence-based encoder model remained among the most effective for enforcing consistency-highlighting the value of integrating structured reasoning modules into otherwise generative workflows. our results motivate future research directions focused on hybrid approaches that combine the strengths of encoderbased models-particularly their structured rea-soning and consistency enforcement-with the generalization and contextual understanding of llms. we believe that with targeted improvements in prompt engineering, structural guidance, and consistency-aware inference, llms can play a central role in advancing temporal relation extraction beyond current limitations.