ULTRA : Unleash LLM s’ Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Self-Refinement (2024)

Abstract: Structural extraction of events within discourse is critical since it avails a deeper understanding of communication patterns and behavior trends. Event argument extraction (EAE), at the core of event-centric understanding, is the task of identifying role-specific text spans (i.e., arguments) for a given event. Document-level EAE (DocEAE) focuses on arguments that are scattered across an entire document. In this work, we explore open-source Large Language Models (LLMs) for DocEAE, and propose ULTRA, a hierarchical framework that extracts event arguments more cost-effectively. Further, it alleviates the positional bias issue intrinsic to LLMs. ULTRA sequentially reads text chunks of a document to generate a candidate argument set, upon which non-pertinent candidates are dropped through self-refinement. We introduce LEAFER to address the challenge LLMs face in locating the exact boundary of an argument. ULTRA outperforms strong baselines, including strong supervised models and ChatGPT, by 9.8% when evaluated by Exact Match (EM).

## introduction event extraction, a long-standing and prominent information extraction task, aims to extract event structures consisting of core information elements (e.g., "who" did "what" to "whom", "when", "where", and "why") from unstructured texts @xcite @xcite . event-centric understanding is of great importance, not only in its inherent merits, but also due to its role as an information-rich representation for downstream tasks like summarization @xcite , recommendation @xcite , and news narrative understanding @xcite @xcite . event argument extraction (eae), a crucial and challenging step in event extraction, is the task of identifying news title: drought puts 2.1 million kenyans at risk of starvation news body: [0] national disaster declared as crops fail after poor rains and locusts, while ethnic conflicts add to crisis last modified on wed 15 sep 2021 07.02 bst. ## methodology taking as input a news article d, ultra first reads text chunks of the article d sequentially to generate a candidate argument set {a} ( §2.1), upon which ultra drops less-pertinent candidates through self-refinement and returns {a f } ( §2.3). a leafer module, learning from errors ( §2.2), is introduced to tackle llms' incapability of locating exact boundaries of argument spans, and yield {a ′ }. ultra+ is a variant augmented with extractions by a document-level extractor to capture information that requires full-article discourse analysis (e.g., extracting "why"-type arguments; §2.4). figure 1 depicts the overall framework of ultra+. putting it all together, we produce two versions: ultra-base and ultra-long, which consume 5-sentence and 15-sentence windows in layer-1, respectively. instead of conducting costly prompt hacking @xcite , we adopt an existing instruction from niv2 @xcite and tailor it to our use case, named aligned instruction. we show designed task instructions {i} in table a4 . ## experiments we conduct experiments on the docee dataset @xcite , which contains 27,485 news articles, classified into 59 event types, and 356 argument roles. we use the cross-domain setting in our experiments since it only contains a minimally annotated target training set (i.e., 50 articles) which can best assess models' generalizability in the wild. specifically, its test set contains 1, 955 news articles covering 10 different event types, and each article is annotated with ∼ 6.5 arguments. we use the same data split and processed texts as in the original doceae dataset for a fair comparison. in terms of evaluation metrics, we follow the literature on document-level event argument extraction @xcite , and adopt exact match (em) and head noun phrase match (hm) as evaluation metrics. em assesses if an extracted argument exactly matches a reference, while hm is a relaxed metric that concerns if there is an overlap of head words of noun phrases between extractions and references. baselines. we compare ultra against three model families to comprehensively study extraction performance and monetary costs. the first model family, supervised ml, characterized as using human annotations as the supervision signal to train small-scale lms, consists of eeqa @xcite and ontology qa @xcite . ontology qa, an extension of eeqa, incorporates argument ontology knowledge, which achieves the sota performance for doceae. second, we compare with chatgpt using different prompting techniques, given its popularity and impressive capability. we follow @xcite and prompt chatgpt to extract spans for all argument roles in one pass. for single-question variant, we modify the original prompt to instruct chatgpt to extract span(s) for only one argument role at a time. motivated by @xcite , which generates a chain-of-thought rationale before summarizing an article, we build the cot-chatgpt variant by replacing the summarizer in @xcite with an argument extractor. the last family involves prompting a document-level extractor a3 . with different instructions, utilizing flan-ul2 as its backbone for a fair comparison. this serves two purposes: test sensitivity to different custom instructions that are designed from scratch; and illuminate the effectiveness of aligned instructions. ## results and analysis our proposed ultra achieves the best f1 scores across the board ( ## conclusion in this study, we present ultra, a cost-effective event argument extraction framework built upon an open-source llm. concretely, ultra reads a sequence of text chunks from an article, the outputs of which are refined through self-refinement. with minimal annotation efforts, a leafer module is implemented to improve argument span boundary identification. our results show the superiority of ultra in comparison to supervised ml and closed llms. we further showcase the customizability of ultra to cater to different extraction criteria.