Adaptive Schema-aware Event Extraction with Retrieval-Augmented Generation (2025)

Abstract: Event extraction (EE) is a fundamental task in natural language processing (NLP) that involves identifying and extracting event information from unstructured text. Effective EE in real-world scenarios requires two key steps: selecting appropriate schemas from hundreds of candidates and executing the extraction process.Existing research exhibits two critical gaps: (1) the rigid schema fixation in existing pipeline systems, and (2) the absence of benchmarks for evaluating joint schema matching and extraction.Although large language models (LLMs) offer potential solutions, their schema hallucination tendencies and context window limitations pose challenges for practical deployment. In response, we propose \textbf{A} daptive \textbf{S} chema-aware \textbf{E} vent \textbf{E} xtraction ( \textbf{ASEE} ), a novel paradigm combining schema paraphrasing with schema retrieval-augmented generation. ASEE adeptly retrieves paraphrased schemas and accurately generates targeted structures.To facilitate rigorous evaluation, we construct the \textbf{M} ulti- \textbf{D} imensional \textbf{S} chema-aware \textbf{E} vent \textbf{E} xtraction ( \textbf{MD-SEE} ) benchmark, which systematically consolidates 12 datasets across diverse domains, complexity levels, and language settings.Extensive evaluations on MD-SEE show that our proposed ASEE demonstrates strong adaptability across various scenarios, significantly improving the accuracy of event extraction. Our codes and datasets are available at https://github.com/USTC-StarTeam/ASEE.git

## introduction event extraction (ee) is an essential task in information extraction (ie) @xcite that aims to identify event triggers (i.e., event detection (ed)) and their associated arguments (i.e., event arguments extraction (eae)) from unstructured text, thereby transforming raw * corresponding author text into structured event representations @xcite . event extraction plays an important role in various natural language processing applications, such as knowledge graph construction, question answering, information retrieval, and event prediction, by providing structured representations of real-world occurrences @xcite . while existing ee studies are effective, they typically operate under the assumption of a small, predefined set of event schemas. however, real-world ee deployments, especially in broad-domain applications, present a more complex challenge: practitioners must first select the appropriate schema from hundreds or even thousands of domainspecific candidates before executing the extraction. this dynamic, two-stage paradigm is critical in interactive scenarios. for instance, in building an ondevice personalized assistant @xcite , the system needs to construct a user's profile by extracting diverse information (e.g., interests, appointments, contacts) from a long-term conversational history. in such cases, a conventional approach with a rigid, fixed schema is incompatible with the dynamic nature of user interactions. this operational gap between academic research and industrial needs exposes two fundamental limitations: one in system design and the other in evaluation benchmarks. first, in terms of system design, existing approaches lack adaptability. traditional pipeline systems are inherently rigid: they are either hardcoded for a fixed schema, which fails in crossdomain scenarios, or handle multiple schemas via naive concatenation, leading to conflicts and error propagation. while large language models (llms) seem to resolve this with their generalization abilities @xcite , their practical application reveals a new dilemma. a common llm-based strategy involves loading all candidate schemas into the context window. even as context windows expand, this "brute-force" method is suboptimal, incurring significant computational costs and suffering from performance decay due to effects like "lost in the middle" @xcite , where models struggle to utilize information within long inputs @xcite . furthermore, llms can exhibit schema hallucination, inventing event types not specified in the provided schemas @xcite . retrieval-augmented generation (rag) @xcite presents a more targeted alternative. however, its effectiveness is often hampered because current implementations rely on oversimplified schema definitions that lack the semantic depth required for precise retrieval and reliable schema-guided extraction. second, the absence of joint evaluation benchmarks creates an academic-industrial disconnect. while recent studies explore llm-based extraction @xcite , existing datasets (lai, 2022) presuppose perfect schema selection-an assumption invalid in real-world scenarios where schema retrieval constitutes a mission-critical prerequisite. this discrepancy leaves the combined capability of schema retrieval and event extraction unevaluated. to address the first limitation, we propose adaptive schema-aware event extraction (asee), a novel paradigm integrating schema paraphrasing with schema retrieval-augmented generation, by decomposing the event extraction task into schema retrieval and schema-aware extraction. figure 1 shows an example of asee in broad domains. in particular, asee extensively builds event extraction schemas, adaptively retrieves specific schemas, automatically assembles event extraction prompts, and accurately generates targeted structures. to resolve the second limitation, we construct the multi-dimensional schema-aware event extraction (md-see) benchmark by systemati-cally consolidating 12 datasets, enabling rigorous joint evaluation on schema retrieval and extraction accuracy across various domains, complexities, and language settings, addressing the current evaluation vacuum. our principal contributions are trifold: we propose asee, the first framework that jointly enhances the event extraction capability through llm-based schema paraphrasing and schema-retrieval augmented generation. information extraction task information extraction (ie) aims to automatically extract structured knowledge from unstructured texts, typically involving three core tasks: (1) named entity recognition (ner) for identifying entity boundaries and types @xcite , (2) relation extraction (re) for detecting semantic relationships between entities @xcite , and (3) event extraction (ee) for recognizing event triggers and their associated arguments @xcite . these tasks are conventionally categorized as closed ie (with predefined schemas) or open ie (schema-agnostic). closed ie relies on predefined schemas specifying target entities, relations, or event structures, enabling precise extraction in constrained domains @xcite . open ie systems, conversely, extract open-domain triples without schema constraints, sacrificing precision for broader coverage @xcite . recent approaches in oie have explored learning from syntactic structures or chunking to improve extraction quality without predefined schemas @xcite . existing event extraction methods predominantly follow the closed ie paradigm, requiring predefined schemas that specify event types, roles, and constraints. though effective in controlled settings, such rigid schemas face two key challenges: (1) oversimplification -most schemas abstract away domain-specific nuances (e.g., using generic role labels like "participant"), making them insufficient for guiding llms in real-world extraction; (2) semantic gap -predefined schemas often mismatch actual context semantics, especially when processing cross-domain or emerging event types. this paper proposes adaptive schema-aware event extraction (asee) to address real-world ee scenarios requiring schema retrieval before extraction. unlike traditional closed ee with rigid predefined schemas, our framework dynamically adapts schemas through : (1) schema paraphrasing that rewrites schema elements using contextual knowledge, and (2) schema-retrieval augmented generation guided by retrieving the relevant schemas and extracting events with the matched schemas. event extraction with llms recently, a significant amount of work has focused on utilizing large language models (llms) to implement and enhance the performance of information extraction tasks. these works can be primarily divided into four aspects: (1) data augmentation: leveraging llms to generate synthetic data or augment existing datasets for information extraction tasks, avoiding the introduction of unrealistic, misleading, and offset patterns @xcite . (2) prompt engineering: crafting effective prompts for llms to guide them in extracting relevant information from text @xcite . (3) code llms: utilizing code-based llms to automate and enhance the information extraction process @xcite . (4) llm fine-tuning: finetuning llms on domain-specific data to optimize their performance on targeted information extraction tasks @xcite . however, when using llms for event extraction (ee), they tend to experience significant hallucinations due to insufficient task-specific optimization, and their limited context window makes it impractical to rely solely on llms to handle event extraction tasks across diverse scenarios. retrieval-augmented generation (rag) methods have emerged as a promising solution, enhancing extraction accuracy by incorporating external knowledge @xcite . for instance, @xcite decompose event extraction into two subtasks: event detection (ed), which retrieves relevant event examples, and event argument extraction (eae), which extracts events based on the retrieved examples. in this paper, we aim to mitigate the hallucination issue of llms in ee by adopting schemaretrieval augmented generation, while also addressing the issue of llm context length limitations. ## benchmark construction we construct the multi-dimensional schemaaware event extraction (md-see) benchmark to facilitate rigorous evaluation, which systematically consolidates 12 datasets across diverse domains, complexity levels, and language settings. due to space constraints, additional details are provided in the appendix a. ## conclusion in this paper, we introduced adaptive schemaaware event extraction (asee), a novel framework designed to enhance event extraction across diverse domains using large language models. by decomposing the extraction process into schema paraphrasing and schema retrieval-augmented extraction, asee effectively mitigates challenges such as hallucinations and context length limitations inherent in llm-based approaches. we develop the md-see dataset, which consists of high-quality schemas across various dimensions, providing a comprehensive resource for evaluating event extraction systems. we conducted extensive experiments on multiple datasets including our newly developed md-see dataset, demonstrating asee's adaptability and superior extraction performance. the framework's ability to build and leverage a comprehensive schema pool enables more precise and scalable event extraction, making it a robust solution for a wide range of real-world applications. future work will explore integrating relation extraction and named entity recognition, fine-tuning larger llms, and extending asee to more complex multilingual and cross-lingual scenarios.