E vent R el B ench: A Comprehensive Benchmark for Evaluating Event Relation Understanding in Large Language Models (2025)

Abstract: Understanding event relationships is critical for tasks such as narrative comprehension, information extraction, and reasoning in natural language processing. Despite the remarkable advancements of large language models (LLMs) across diverse NLP tasks, current studies have not systematically evaluated their ability to capture the complex of event relations. To this end, we aim to assess LLMs on event relationship extraction (ERE) by designing the benchmark EventRelBench. EventRelBench comprises 35K diverse event relation questions covering four key categories—coreference, temporal, causal, and supersub relations. These questions are provided at two levels of granularity: document-level and sentence-level. Extensive experiments on different sizes and types of LLMs show that existing LLMs still fall short in accurately extracting and understanding event relationships. To address this gap, we introduce EventRelInst, a 48K instruction fine‐tuning dataset in the event relation extraction domain. Experimental results not only highlight the shortcomings of current general-purpose LLMs in extracting event relationships but also demonstrate the effectiveness of EventRelInst. Both EventRelBench and EventRelBench will be publicly available.

## introduction event relations are crucial for numerous downstream nlp applications, including narrative comprehension, information extraction, question answering, and commonsense reasoning @xcite @xcite @xcite @xcite . event relation extraction (ere) facilitates the construction of structured event graphs and the inference of implicit dependencies among events. while early work on event relation extraction relied on featureengineered classifiers and structured prediction 1 *corresponding author. frameworks @xcite @xcite @xcite , the advent of large language models (llms) has opened new opportunities for capturing rich contextual cues and implicit knowledge. recent llms @xcite @xcite @xcite @xcite have demonstrated impressive zero-shot and few-shot performance across a wide spectrum of benchmarks. these benchmarks range from reasoning-oriented tasks such as gsm8k @xcite to knowledgeintensive challenges like factual question answering @xcite . however, systematic evaluation of llms' capacity to understand and extract complex event relationships remains lacking. existing llm benchmarks typically focus on individual relation types in isolation (e.g., temporal relation) or operate at a single granularity (e.g., sentence-level). this leaves open questions about llms' capacity to comprehend event relations across diverse categories and text scopes. without a unified evaluation benchmark, it is difficult to pinpoint specific weaknesses of general-purpose llms or to develop targeted solutions. to bridge this gap, we introduce eventrel-bench, a comprehensive benchmark designed to evaluate llms on four crucial event relation categories-temporal, causal, coreference, and supersub. eventrelbench consists of 35k multiple-choice questions drawn from diverse domains, ensuring rich variety in linguistic constructions and knowledge demands. two granularity levels are included: document-level and sentencelevel. as shown in figure 1 , llm predicts four event relationships in the given document. event(2: killed) has a coreference event relationship with event(3: died). event(2: killed) and event(5: affecting) have a temporal relationship of "before". event(1: storm) has a causal relationship of "cause" with event(4: canceled). the subevent of event(1: storm) is event(3: died). through extensive experiments on eventrel-bench with several open-source and leading limited-access llms, we have observed several key findings: 1) existing llms exhibit poor performance on event relation extraction tasks, with some models (e.g., aquilachat-7b and gemma-7b-it @xcite ) even underperforming random guessing, highlighting substantial potential for improvement. 2) llms perform better on document-level tasks than on sentence-level tasks. this contrasts with the typical behavior observed in conventional models. we believe this may be attributed to the ability of llms to better capture and understand contextual information over long distances. 3) performance varies markedly across llms for different event relation extraction tasks, reflecting each model's distinct strengths and limitations. motivated by observed shortcomings, we further introduce eventrelinst, a 48k example instruction fine-tuning dataset tailored to the event relation extraction task. eventrelinst converts domainspecific data into explicit instruction-response pairs. this format enables models to better grasp the distinct demands of each relation type and the corresponding contextual granularity. our experiments show that fine-tuning with eventrelinst can lead to significant performance gains compared to benchmark models. this outcome validates the effectiveness and potential of eventrelinst. in summary, our main contributions are as follows: ## experiments in this section, we present and analyze the performance of various llms on eventrelbench, which evaluates models' capabilities across four prompting paradigms: zero-shot, zero-shot with cot, few-shot, and few-shot with cot. we also include results for fine-tuned variants (ft) of several models using eventrelinst based on lora.