Debate as Optimization: Adaptive Conformal Prediction and Diverse Retrieval for Event Extraction (2024)

Abstract: We propose a multi-agent debate as optimization (DAO) system for event extraction, where the primary objective is to iteratively refine the large language models (LLMs) outputs through debating without parameter tuning. In DAO, we introduce two novel modules: the Diverse-RAG (DRAG) module and the Adaptive Conformal Prediction (AdaCP) module. DRAG systematically retrieves supporting information that best fits the debate discussion, while AdaCP enhances the accuracy and reliability of event extraction by effectively rejecting less promising answers. Experimental results demonstrate a significant reduction in the performance gap between supervised approaches and tuning-free LLM-based methods by 18.1% and 17.8% on ACE05 and 17.9% and 15.2% on CASIE for event detection and argument extraction respectively.

## introduction event extraction (ee) @xcite @xcite involves identifying and categorizing event mentions, expressed through trigger tokens and participants in natural language text. recent studies show that leveraging large language models (llms) has led to remarkable advancements in numerous applications @xcite @xcite . their potent natural language understanding capabilities are generic and adaptable to nearly any open domain. however, a significant gap remains for event extraction between advanced tuning-based approaches @xcite @xcite @xcite and approaches without tuning @xcite @xcite . llms struggle to match the performance of tuning-based approaches due to several challenges. first, the inherent ambiguities and variations in event mentions present significant obstacles in accurately identifying them. for instance, in the phrase "pay the fines", two potential questions arise: whether the event type should be classified as a transfer-money or fine event and whether the event trigger should be "pay" or "fines". second, existing solutions fail to efficiently incorporate domain-specific knowledge, such as extensive event schemas. while a common solution is to enumerate event schemas into the prompt @xcite , llms can struggle to fully comprehend and utilize this information. lastly, unlike tuning-based methods that can leverage annotated data, such as ace05 @xcite and ere @xcite , to learn implicit statistical features and resolve nuanced semantic differences, llms are difficult to tune, even with small amounts of data, particularly without access to the model checkpoint. to address these challenges, we introduce a tuning-free multi-agent debating-as-optimization (dao) framework. this approach demonstrates that event extraction answers can be gradually optimized through debates among llm agents without domain-specific fine-tuning, allowing the system to adapt effortlessly to new domains or ontologies. to optimize the initial solution, we propose two novel modules: the diverse retrieval augmented module (drag) and the adaptive conformal prediction module (adacp). the drag module dynamically retrieves domain-specific data entries that best fit the current points of disagreement. the adacp model employs an adaptive conformal prediction policy to progressively reject less convincing answers based on the retrieved knowledge. the event extraction answer is gradually refined through more precise retrieval of domain-specific knowledge and the application of stricter rejection rules. our aim is to demonstrate that the significant performance gap can be narrowed with the proposed multi-agent debate framework. ## approach in event extraction (ee), two sub-tasks are involved: event detection (ed) and event argument extraction (eae). the proposed debating as optimization (dao) framework tackles both ed and eae through a unified debating process, employing distinct task-specific prompts for each sub-task. detailed agent prompts are in appendix b. ## experimental setup dataset and evaluation metrics we conducted experiments on two public benchmark datasets, ace05-e (automatic content extraction, ace05) foot_1 and casie @xcite . for the ace05, we reported evaluation results on the test set using the same test split as in @xcite . for the casie, we used the same test split as in @xcite . the evaluation is focused on three sub-tasks: ed, eae where the ground truth trigger is given, and ee where ed and eae are performed jointly. we only report argument extraction performance for ee following previous work @xcite . for the ace05 dataset, we followed previous work @xcite and used the exact match f1 score for evaluating ed and the argument head f1 score for evaluating eae and ee. for the casie dataset, we adhered to the evaluation standards established in previous studies @xcite , employing the types metric for all three sub-tasks. baselines we consider the following baselines that utilize zero-shot or in-context learning capabilities of llms: (1) chatgpt-14 @xcite , the first work that systematically analyzes the chatgpt's performance on information extraction (ie) tasks utilizing its zero-shot capabilities. (2) chatgpt-ie @xcite , which highlights that chatgpt often generates longer trigger or argument spans, contributing to the evaluation gap between chatgpt and tuning-based approaches. a soft-matching strategy is proposed to mitigate this evaluation gap, thereby providing a more accurate reflection of chatgpt's performance. (3) chatie @xcite , a multi-turn questionanswering framework for zero-shot ie, wherein the first stage collects all the possible event types and in the second stage it performs information extraction for each event type. (4) g-ptlm @xcite regularize the event argument predictions by explicitly expressing argument constraints with prompts. (5) code4struct @xcite formulate event extraction as a code generation problem, and represents event ontology in python code expression. (6) code4uie @xcite , another code generation-based approach, utilizing additional m annotations retrieved from the training corpus with the highest similarity to the input sentence. the retrieved examples are used as icl examples. in addition to the zeroshot or in-context learning based approaches, we include three supervised fine-tuning (sft) based approaches with relatively smaller lms as baselines, including degree @xcite , in-structuie @xcite , and rexuie @xcite . implementation details the proposed system is flexible, allowing any llm to serve in any arbitrary agent role defined within the framework. in our experiments, we employ three llms: llama-3-8b-instruct (llama3), gemini-pro (gemini), and gpt-3.5-turbo (gpt). the results are presented under two distinct settings: (a) gemini-gpt: in this setting, two debaters are powered by gemini and gpt, respectively. the critic agent is powered by gemini, while the judge agent is powered by gpt. (b) llama3-gpt: here, one debater uses llama-3-8b-instruct (llama3), and the other uses gpt-3.5-turbo (gpt). both the critic and judge agents are powered by gemini. we set the temperature of all agents to 0 to ensure reproducibility. the llama checkpoint is accessible at the huggingface (ai@meta, 2024) under llama 3 community license agreement. we use official api to access gemini foot_2 and gpt foot_3 under commercial license. additionally, the calibration model is flant5-xxl foot_4 . no tuning is involved for any of the llms. all the experiments are run with one nvidia a40. we use spacy for argument head detection. the initial conformal generation risk threshold is determined by a randomly sampled calibration set from the training set. and the conformal calibration is conducted by a frozen flan-t5-xxl. for ed, the initial conformal generation risk q0 is set to 1, with a decay rate β of 0.5. for eae, the initial conformal generation risk q0 is set to 3, also with a decay rate of 0.5. all debates are capped at a maximum of three rounds. the initial cluster radius µ 0 is constantly set to 1.35, and the radius decay factor λ is 0.9. ## conclusion this work introduces a novel multi-agent debate paradigm that resembles the optimization process. this debate model is conceptualized as an optimization mechanism wherein supporting information is systematically retrieved to regulate the distribution of risk. the evolution of risk distribution throughout the debating process illustrates how the integration of the adaptive conformal prediction module and the diverse rag module can progressively steer the risk distribution towards more confident answers. through this framework, the debate process becomes not just a discourse but a strategic endeavor aimed at achieving optimal outcomes.