CROSSAGENTIE : Cross-Type and Cross-Task Multi-Agent LLM Collaboration for Zero-Shot Information Extraction (2025)

Abstract: Large language models (LLMs) excel in generating unstructured text. However, they struggle with producing structured output while maintaining accuracy in zero-shot information extraction (IE), such as named entity recognition (NER) and relation extraction (RE). To address these challenges, we propose CROSSAGENTIE, a multi-agent framework that enhances zero-shot IE through multi-agent LLM collaboration. CROSSAGENTIE refines LLM predictions iteratively through two mechanisms: intra-group cross-type debate, which resolves entity-label conflicts through context-based evidence and confidence aggregation, and inter-group cross-task debate, where NER and RE mutually refine outputs via bidirectional feedback. Furthermore, we introduce template fine-tuning, distilling high-confidence multi-agent outputs into a single model, significantly reducing inference cost while preserving accuracy. Experiments across five NER and five RE datasets show that CROSSAGENTIE significantly outperforms state-of-the-art zero-shot baselines by a large margin. CROSSAGENTIE effectively addresses LLMs limitations in structured prediction with an effective and efficient approach for zero-shot information extraction.

## introduction information extraction (ie) is a fundamental task in natural language processing (nlp) that aims to extract structured information from unstructured or semi-structured text @xcite . it includes subtasks such as named entities recognition (ner) and relation extraction @xcite . traditional supervised ie methods typically follow a "pre-@xmath0, where a pre-trained language model is adapted to a labeled dataset with extensive supervision signals (devlin * corresponding author et al., 2019; raffel et al., 2023; zhuang et al., 2021). while effective, these methods suffer from high annotation costs and limited generalization, making them impractical for low-resource scenarios and rapidly evolving domains. given these limitations, recent research has explored zero-shot ie as a promising direction @xcite . recent advances in large language models (llms) @xcite have enabled more effective zero-shot ie methods to overcome the shortcomings of traditional supervised models. the llms' strong language understanding capabilities, gained through extensive pre-training, allow them to perform ie tasks effectively. llm-based approaches for zero-shot ie include direct prompting @xcite @xcite , in-context learning @xcite , synthetic data generation @xcite , and pseudolabeling for fine-tuning @xcite @xcite . these methods reduce reliance on annotated data and enhance adaptability, making llms a promising solution for zero-shot ie. despite advancements, llms still encounter critical challenges that limit their performance in zeroshot ie. first, llms struggle to generate structured outputs that adhere to predefined labeling schemas in ie. unlike traditional models optimized for structured representations, llms predominantly generate free-form text. although prompting techniques, such as using symbols @xcite , lists @xcite , and tables @xcite , have been explored, inconsistencies persist in structured output generation. second, entity-label conflicts arise when identical entities receive inconsistent categorizations (e.g., "washington" might be labeled as both location and person). existing approaches @xcite tackle this issue through weak supervision, either by fine-tuning smaller models on pseudo-labeled data or by transferring knowledge from limited annotations. however, they rely on external supervision rather than leveraging the intrinsic reasoning embedded in llms' representations, which limits the generalization of these methods to broader scenarios. third, llms struggle with domain adaptation, failing to internalize domain-specific knowledge despite task instructions. while prompt engineering can create role-specialized agents @xcite , these methods require extensive tuning and lack cross-domain generalization. as a result of the above three challenges, current llmbased methods struggle with achieving high performance in zero-shot ie @xcite @xcite . for example, direct prompting with gpt-3.5 achieves only 45% f1 on conll03 @xcite and 34% on ontonotes4 @xcite for ner. to address the above challenges, we propose crossagentie, a multi-agent llm collaboration framework that enhances zero-shot ner and re performance through structured debate and bidirectional refinement. first, intra-group crosstype debate resolves entity-label conflicts by verifying classifications (e.g., distinguishing "washington" as "location" or "person") through contextbased reasoning. second, inter-group cross-task debate refines ner and re predictions by integrating relation-based feedback, enhancing contextual grounding, and entity accuracy through bidirectional knowledge exchange. third, to enhance domain adaptation, crossagentie equips type agents with domain-specific metadata, leveraging entity-type knowledge and ontology constraints for schema-aligned classification. finally, to improve inference efficiency, crossagentie introduces template fine-tuning that distills the multi-agent outputs into a single model. this process reduces computational cost while ensuring cross-domain consistency, greatly enhancing the efficiency of crossagentie. experiments across different datasets show that crossagentie significantly outperforms state-of-the-art zero-shot baselines by a large margin. llms for ner several approaches enhance ner with llms. gpr-ner @xcite reformulates ner as text generation with entity markers and self-verification, reducing overpredictions via few-shot and in-context learning. universalner @xcite distills chatgptgenerated data into a smaller llama-based model through instruction tuning. verifiner @xcite integrates llms with external knowledge bases for post-hoc verification, refining entity boundaries and types. decomposed-qa @xcite improves ner via task decomposition, syntactic augmentation, and self-consistency voting with chatgpt. progen @xcite uses step-by-step generation and self-reflection to enhance few-shot ner dataset construction and entity attribute refinement. ## crossagentie framework this section introduces crossagentie, a multistage framework for structured information extraction using collaborative agents. we first formalize the problem (sec. 3.1), followed by type-agent setup (sec. 3.2), intra-group cross-type discussion (sec. 3.3), inter-group cross-task discussion (sec. 3.4), and finally template fine-tuning (sec. 3.5). figure 1 illustrates the overall framework, with detailed prompts provided in appendix d. ## experiments we evaluate crossagentie on ner and re benchmarks using strict full-matching criteria, comparing them with state-of-the-art baselines. as prior work (section 2) has applied llms in various settings, we select the most relevant state-of-the-art (sota) zero-shot approaches. see appendix b for methods comparison. for fair comparison, we use gpt-3.5 as the backbone, aligning with existing baselines, and additionally test our approach on gpt-4o for evaluation on a more advanced llm. please see section 4.3 for details.