General Collaborative Framework between Large Language Model and Experts for Universal Information Extraction (2024)

Abstract: Recently, unified information extraction has garnered widespread attention from the NLP community, which aims to use a unified paradigm to perform various information extraction tasks. However, prevalent unified IE approaches inevitably encounter challenges such as noise interference, abstract label semantics, and diverse span granularity. In this paper, we first present three problematic assumptions regarding the capabilities of unified information extraction model. Furthermore, we propose the General Collaborative Information Extraction (GCIE) framework to address these challenges in universal information extraction tasks. Specifically, GCIE consists of a general Recognizer as well as multiple task-specific Experts for recognizing predefined types and extracting spans respectively. The Recognizer is a large language model, while the Experts comprise a series of smaller language models. Together, they collaborate in a two-stage pipeline to perform unified information extraction. Extensive empirical experiments on 6 IE tasks and several datasets, validate the effectiveness and generality of our approach.

## introduction information extraction (ie) endeavors to derive structured information from unstructured text @xcite , which involves a series of tasks, including named entity recognition, relation extraction, entity linking, aspect-based sentiment analysis, and event extraction @xcite . given its diverse objectives @xcite and heterogeneous structures @xcite , traditional ie methods often necessitate task-specific architectures and processes, entailing elaborate manual design @xcite . despite some success, task-specific approaches impede rapid unified architectural development. consequently, an alternative avenue of ie research focuses on addressing multiple subtasks using unified modeling architectures, as exemplified in recent works @xcite @xcite . however, these prospective unified ie methods still grapple with several unresolved issues. one prominent challenge involves the noise interference introduced by negative samples during model training and prediction. unlike traditional nlp tasks, there are usually long-tail data distributions in information extraction tasks that demonstrate imbalanced label quantities across various types, with a larger number of negative samples compared to positive ones @xcite @xcite . how to bridge label with output is also a challenge. other than generative unified modeling architectures, @xcite ; @xcite ; @xcite employ extractive models to achieve unified information extraction through heterogeneous decoding processes across different subtasks. to capitalize on the knowledge acquired during the pretraining stage, many generative and extractive methods represent label types using natural language words. however, unlike context-based large language models such as gpt-3, palm, llama, etc. @xcite @xcite , the efficacy of smaller language models @xcite @xcite in comprehending abstract labels remains questioned. for instance, "attack" is an event type hard to understand by a single word in ace05-evt, representing a range of conflict events such as wars, coups, strikes, terrorist attacks, etc., not merely its literal meaning. witnessing the remarkable performance of massively large language models in extensive nlp tasks, several llm-based methods for information extraction have been proposed @xcite @xcite @xcite . however, there is still no optimal solution regarding the tradeoff between effectiveness and efficiency, primarily due to the poor performance without fine-tuning in ie tasks @xcite and the overhead associated with training llms. in this paper, we are dedicated to analysing these key problems and devising solutions. through our investigation, we sum up three primary factors influencing the capabilities of unified ie models: 1) noisy imbalanced data: a large number of negative samples and long-tail data distribution. 2) abstract label type: obscure type words pose a challenge for understanding by lms. 3) diverse span granularity: annotated data from different sources has various criteria for identifying spans. consequently, we posit that the primary capabilities of unified information extraction models revolve around anti-interference, label understanding, and span identification, addressing the aforementioned challenges. to tackle these issues, we propose the collaborative framework that consists of a recognizer and multiple experts. the recognizer, an llm proficient in anti-interference and label understanding, is tasked with recognizing label types and filtering negative samples. on the other hand, expert utilize type indication as prompt to generate structured text, which are fine-tuned on low noise data distribution for a specific ie task. the recognizer and experts operate in a two-stage pipeline to produce general schemas for universal ie tasks, as illustrated in figure 1 . different from previous research, our approach focuses more on solving the aforementioned problems and achieving performance improvement by simultaneously utilizing the potential advantages of llm and slm. to validate the effectiveness and generality of gcie, we conduct extensive experiments, encompassing 6 ie subtasks across various datasets. the experimental results demonstrate the rationality of key capabilities for unified ie and excellent performance under the supervised and few-shot settings. these findings collectively suggest that the integration of slm and llm yields enhanced information extraction capabilities. in conclusion, the main contributions are summarized as follows: ## key capabilities for unified information extraction in this section, we outline the essential prerequisites for tackling the challenges inherent in universal information extraction tasks, delineating them into three key capabilities. we then elucidate the significance of these capabilities, underscoring why a robust ie model should incorporate all three. while our investigation is approached from a unified ie perspective, it is also applicable to numerous task-specific methodologies. the similar topic is discussed by @xcite . ## general collaborative framework in this section, we introduce a two-stage (recognition & filtering and identification) general col-laborative framework combining llm and slm to acquire capabilities of anti-interference, labelunderstanding and span-identification for universal information extraction tasks. ## error analysis in this section, we discuss the reason why our method can achieve remarkable performance improvements through quantitative error analysis. we report the results of gcie and the baseline on several types shown in to llm not specializing in some professional fields and recognizing unspecific types. we summarize the main points of error analysis as follows: ## related work from the perspective of the target tasks, we primarily present research works about various paradigms for information extraction. many works focus on single specific ie task, such as entity and relation extraction @xcite @xcite @xcite @xcite @xcite @xcite , event detection and argument extraction @xcite @xcite @xcite @xcite @xcite and aspectbased sentiment analysis @xcite @xcite @xcite @xcite . some of these works are based on few-shot settings. with the development of deep language models and the increasing demand of heterogeneous information processing, more and more ie models are designed in the unified paradigm to address various ie tasks. early unified ie models typically employ multi-task joint training to enable the model to adapt various information extraction tasks with different objectives and schemas @xcite @xcite . @xcite has utilized unified semantic matching to achieve state-of-the-art performance on multiple datasets. some recent research efforts @xcite @xcite aim to introduce novel methods to adapt universal ie tasks rather than unified modeling. however, the most closely related approaches to our work are the unified structured generation paradigm for a range of ie tasks @xcite @xcite . since the advent of chatgpt and other llms, more and more researchers take efforts to unlock the potential of llms and bridge the performance gap with sota results in ie tasks @xcite @xcite @xcite . we also regard this as a prospective research direction of unified information extraction. ## conclusion in this study, we analyze the important factors for information extraction and introduce three core capabilities. these capabilities, typically not concurrently possessed by existing ie models, are identified through a series of exploration experiments. our findings suggest that context-based llm is proficient in identifying negative samples and recognizing predefined types. building upon this in-sight, we propose gcie for unified information extraction, which combines the strengths of llm and experts to encompass both of these capabilities. extensive experiments validate that, compared to existing llm-only and slm-only methods, gcie exhibits excellent performance across many ie tasks. all of these indicate a prospective unified ie research direction to take advantages of llm and fine-tuned slm.