L ex T ime: A Benchmark for Temporal Ordering of Legal Events (2025)

Abstract: Understanding temporal relationships and accurately reconstructing the event timeline is important for case law analysis, compliance monitoring, and legal summarization. However, existing benchmarks lack specialized language evaluation, leaving a gap in understanding how LLMs handle event ordering in legal contexts. We introduce LexTime, a dataset designed to evaluate LLMs’ event ordering capabilities in legal language, consisting of 512 instances from U.S. Federal Complaints with annotated event pairs and their temporal relations. Our findings show that (1) LLMs are more accurate on legal event ordering than on narrative texts (up to +10.5%); (2) longer input contexts and implicit events boost accuracy, reaching 80.8% for implicit-explicit event pairs; (3) legal linguistic complexities and nested clauses remain a challenge. While performance is promising, specific features of legal texts remain a bottleneck for legal temporal event reasoning, and we propose concrete modeling directions to better address them.

## introduction understanding the temporal relationships between events is fundamental for natural language understanding. in the legal domain, temporal reasoning is important for tasks such as case law analysis, contract interpretation, and compliance monitoring. accurately determining the timeline of events helps assess liability, obligations, and procedural validity. however, the unique linguistic characteristics of legal texts raise questions about the effectiveness of current temporal reasoning approaches. language models (lms) face notable challenges with temporal reasoning due to the complex logic and ambiguities involved, and the necessity to process long contextual dependencies @xcite . while several benchmarks exist for temporal reasoning with large language models (llms), on november 25, 2020, the district court granted the unions summary judgment motion and denied companyx. companyx subsequently appealed the matter to the united states court of appeals for the sixth circuit (no. 20-xxxx). on september 17, 2021, however, the company and the union entered a settlement agreement that resolved the appeal. none focus on specialized languages. most current research on event extraction and ordering focuses on narratives, such as the event storyline corpus @xcite , the fine grained temporal relations dataset @xcite , and the tracie dataset @xcite . as a result, lms' proficiency in expert types of language, such as legal language, is largely unknown. how do lms address the unique features of legal language for event ordering? in this work, we conduct the first study targeting temporal event ordering within legal language. besides its implications for legal work, event ordering is a foundation for downstream legal nlp tasks, such as summarization, drafting, and question-answering. we start by assembling lex-time, a novel dataset composed of 512 instances constructed from labor-related u.s. federal complaints ( §2). we extract events relevant to both the case and its procedural timeline, providing a struc-tured resource for studying temporal reasoning in legal texts. an example instance from lextime is shown in fig. 1 . we show that legal language used in lextime exhibits specific lexical, syntactic, and discourse characteristics, making it a distinct genre ( §3). since event temporal ordering has been extensively studied in narrative contexts, we use tracie as a baseline for comparison due to its similarity in task design and focus on event ordering. next, we conduct a systematic evaluation of lms' ability to perform event ordering, defined as identifying temporal relations between event pairs ( §4, 5). our analysis focuses on key aspects relevant to the legal domain, such as the impact of context length, the distinction between explicit and implicit events, and a comparative study of narrative versus legal language. following @xcite , we define an implicit event as one whose trigger is not explicitly stated in the text but can be inferred from the surrounding context, such as negotiated in fig. 1 . a key indicator of the implicit event is that it can be fully entailed from and properly implied by the following explicitly stated events. our experiments assess chain-of-thought (cot) prompting @xcite , a method previously successful in reasoning over complex and long inputs. however, we find that cot prompting is ineffective for event ordering, leading to a 3.4% accuracy drop for lextime compared to standard few-shot prompting, which yields the best accuracy. overall, we have three core findings: (1) llms show improved performance on legal event ordering in lextime compared to narrative, achieving enhancements of up to +10.5% (gpt-4o, few-shot). this suggests that legal texts offer clearer temporal structures for models to use. (2) llms perform better with longer input contexts (+2.3% for gpt-4 turbo, few-shot) and with event pairs that include implicit event types. the highest accuracy of 80.8% is achieved when queries contain a pair with one explicit and one implicit event (gpt-4 turbo, few-shot), surpassing pairs of two explicit events by 2.6% and the overall dataset average by 3.2% ( §5). (3) however, linguistic complexities commonly found in legal language contribute to model errors ( §6). features typical of the legal genre are overrepresented in misclassified instances, such as paraphrases (+89% in error samples) and events occurring in subordinate clauses (+84%). in sum, we make the following contributions: ## specific features of legal events this section examines the linguistic and structural characteristics that differentiate legal language from narrative, using lextime and tracie as representative datasets. we show that legal language constitutes a distinct genre characterized by specialized event structures and temporal representations. prior work has shown that legal language follows distinct linguistic conventions @xcite , including lengthy nominalized expressions, frequent passive forms, and specialized vocabulary, making its syntactic structure more complex than that of general prose or news texts @xcite . in an empirical study of the text in the u.s. code, martinez et al. (2024) found higher rates of complex syntactic structures relative to six baseline genres of english. similarly, @xcite noted that u.s. case law contains center-embedded clauses at double the rate of other text genres. guided by the features identified by @xcite , we divide our analysis into: (1) a lexical analysis and (2) a syntax and discourse analysis (fig. 3 ). we employ a combination of methods, each selected for its suitability to a specific feature of interest: (1) named-entity recognition to extract dates; ## benchmark methodology we conduct evaluations using two different promptbased approaches: standard prompting and chainof-thought (cot) prompting. we perform experiments in zero-shot (zs), one-shot (1s), and few-shot (fs) settings (3 examples). prompts and examples can be found in appendix d. ## experimental results gpt-4o, gpt-4 turbo outperform mistral, llama, and flan models across all evaluated prompts and settings. experimental results show that llms perform better on lextime than on tracie, suggesting their ability to tackle some linguistic complexities of legal language. however, §6 will show that these complexities still limit performance. across the five studied subsets, gpt-4 turbo achieves the highest accuracy on lextime in four cases (up to 80.8% accuracy for explicit-implicit pairs of events), including three in an fs setting and one in a 1s cot setting (all data). the best accuracy for the short-context subset is obtained by mistral 123b in a fs setting with 77.8%. overall, cot appears among the top three results only three times across all categories (all data, long context, and explicit-implicit), suggesting its limited impact on performance in this task. compared to gpt-4o, mistral 123b exhibits lower accuracy across all prompts, with performance differences ranging from 0.2% to 8%. llama 3.1 70b also shows reduced accuracy, with a performance gap between 1% and 3.7%. flan-t5 780m demonstrates a significant decline, with accuracy differences ranging from 10.4% to 21.9%. when considering the other llama models collectively, their accuracy is lower by 14.4% to 28.7%. compared to gpt-4 turbo, mistral 123b shows lower accuracy across all prompts, with differences ranging from 1.9% to 9.8%. llama 3.1 70b exhibits a decline in accuracy ranging from 1.3% to 11.5%, while flan-t5 780m shows a more significant drop, with differences between 13% and 21.7%. the other llama models collectively show accuracy reductions ranging from 14.6% to 29.2%. cot prompting does not improve accuracy for event ordering. ## errors analysis building on the linguistic analysis in §3, we manually analyze 100 erroneous predictions by gpt-4, gpt-4 turbo, and llama 3.1 70b . visualization of errors is shown in fig. 7 . ## conclusion we present lextime, the first benchmark for evaluating temporal event ordering in legal language, and systematically evaluate llms' ability to reason over legal temporal structures. our findings show that while llm performance on lextime is superior to their performance on a narrative text benchmark, the complexities of legal discourse, such as paraphrasing, constrain accuracy. we show that longer input contexts and implicit-explicit event pairs improve model accuracy, offering practical insights for optimizing llm-based legal nlp applications. experimental results suggest that further performance improvements could rely on exploiting identified errors and linguistic structure to align models with specialized domain language.