Instruction-Tuning LLM s for Event Extraction with Annotation Guidelines (2025)

Abstract: In this work, we study the effect of annotation guidelines–textual descriptions of event types and arguments, when instruction-tuning large language models for event extraction. We conducted a series of experiments with both human-provided and machine-generated guidelines in both full- and low-data settings. Our results demonstrate the promise of annotation guidelines when there is a decent amount of training data and highlight its effectiveness in improving cross-schema generalization and low-frequency event-type performance.

## introduction event extraction (ee) aims to identify and structure what, who, when, where, and how of realworld events from given textual resources @xcite @xcite . translating this abstraction requires complex schema specifications that define event types, argument roles, and their interrelationships, yet being able to precisely capture the language nuances and distinguish between event types and argument roles, which posits the task as an inherently challenging problem. recently, large language models (llms) have transformed nlp research and practices dramatically, owing to the rich knowledge and other capabilities (e.g., reasoning) they have obtained from extensive pre-training @xcite @xcite . this transformation has similarly impacted the broader research field of information extraction @xcite . existing applications of llms to ie can be categorized into two lines. the prompt engineering-based approaches, often based on proprietary llms, consider an llm as a black box, querying it with task specifications via zero-or few-shot prompting and relying on its latent knowledge to extract interested information @xcite @xcite . however, these approaches not only lead to inferior performance but also incur prohibitive costs, especially when the task is complex. our work will thus focus on the second line of approach, namely, instruction-tuning open-weight llms. this line of approach adapts an llm to specific ie tasks and schemas by directly training it to follow the task instructions, which offers a promising yet cost-effective solution. for example, @xcite leverages natural language instructions to guide large language models for ie tasks; @xcite proposed a two-phase learning framework that enhances schema understanding and following ability via automatically annotated data. more recently, @xcite instruction-tuned llama @xcite on multiple ie datasets and discovered annotation guidelines-textual descriptions of an event type and its argument roles used by human annotators when collecting the dataset, as effective components of an ie task's instruction. despite the promise of the existing explorations, however, most of them have focused on the relatively simpler task of named entity recognition, yet how to properly instruction-tune llms for the structured ee task is still understudied. to fill this gap, we study instruction-tuning llms for ee, with a focus on the role of annotation guidelines in task instructions (fig. 1 ). we conduct a systematic analysis using llama-3.1-8b-instruct on two ee datasets (ace05 @xcite and richere @xcite ) under varied training settings. our key findings are organized around four themes: finally, we validated the consistency of our findings across both model scale and diversity. using the smaller llama-3.2-1b-instruct model, we observed that annotation guidelines retain their benefits, reducing common ee errors and supporting both frequent and rare event types. extending beyond the llama family, experiments with qwen2.5-coder-1.5b-instruct @xcite , a code-oriented model with a distinct pertaining objective, revealed similarly robust gains. we also evaluated on speed++ @xcite , a domain-shifted dataset of informal epidemic-related tweets, where guidelines continued to significantly improve performance. together, these results confirm the broad utility of machine-generated guidelines across model architectures, data regimes, and textual domains. ## related work llms for ie and ee with the growing capabilities of llms, recent efforts have explored their potential in ie @xcite and studied ee as an auxiliary task. existing llm-based ie methods generally fall into two categories: in-context learning (icl) and supervised fine-tuning (sft). icl-based approaches @xcite @xcite rely on providing a few-shot context within prompts, enabling llms to infer structured information without explicit parameter updates. while being data-efficient, they were found to misinterpret the task specifications @xcite and suffer from brittle sensitivity to prompt phrasing and example ordering @xcite . in addition, they also incur prohibitive costs due to the lengthy reasoning chains especially for complex tasks. in contrast, sft-based methods @xcite @xcite @xcite fine-tune llms on annotated datasets, which can significantly improve their ee performance. our work deepens this line of research and particularly explores the 0 20 40 60 80 100 f-score (%) 0 5 10 15 20 25 event index +8.6% +7.7% +7.3% +9.8% +6.2% +13.2% -2.3% +10.0% +16.7% +2.9% +30.5% +9.3% +4.7% +16.2% -3.0% +10.2% +30.9% +16.7% +40.0% +1.7% +42.9% -28.6% 0.0% 0.0% +30.0% -33.3% +100.0% 0.0% 0.0% 0.0% 29.73% 35.2% ace w/o ns 0 20 40 60 80 100 f-score (%) 0 5 10 15 20 25 event index +4.0% -3.1% +6.6% +2.8% +15.5% +14.3% +9.1% -5.9% -5.5% -1.7% +1.7% -3.3% +7.9% +33.3% +27.0% 0.0% +14.2% -16.5% +13.3% 0.0% -4.8% 0.0% -19.0% +33.3% -7.3% +33.3% 0.0% 0.0% -20.0% +100.0% 25.32% 27.18% richere w/o ns inclusion of annotation guidelines in instructions. while there have been existing works on similar topics, they did not focus on ee @xcite or instruction tuning @xcite . code prompts for ee while ee tasks are typically represented in texts, code-based prompting has emerged as a promising alternative, leveraging structured representations to enhance schema adherence. early works have applied code-style prompts to event argument extraction @xcite and other ie tasks @xcite , demonstrating potential but often underperforming compared to sft-based models due to the absence of fine-tuning. eventrl @xcite utilizes outcome supervision with specific reward functions to reduce information mismatch and hallucination. knowcoder @xcite addresses this limitation by introducing a comprehensive schema representation in code format, integrating taxonomies, constraints, and structured definitions. complementary to these works, we study generating annotation guidelines to enhance the instruction tuning of llms for code-formatted ee and demonstrate their effectiveness. ## conclusion we demonstrate that incorporating structured annotation guidelines improves the instruction-tuning of llms for ee, bridges the data gap when only a limited amount of training data is available, and enhances the model's cross-schema generalization. our explorations of guideline generation also highlight the promise of automatically generating effective instructions.