D oc IE @ XLLM 25: Z ero S emble - Robust and Efficient Zero-Shot Document Information Extraction with Heterogeneous Large Language Model Ensembles (2025)

Abstract: The schematization of knowledge, including the extraction of entities and relations from documents, poses significant challenges to traditional approaches because of the document’s ambiguity, heterogeneity, and high cost domain-specific training. Although Large Language Models (LLMs) allow for extraction without prior training on the dataset, the requirement of fine-tuning along with low precision, especially in relation extraction, serves as an obstacle. In absence of domain-specific training, we present a new zero-shot ensemble approach using DeepSeek-R1-Distill-Llama-70B, Llama-3.3-70B, and Qwen-2.5-32B. Our key innovation is a two-stage pipeline that first consolidates high-confidence entities through ensemble techniques, then leverages Qwen-2.5-32B with engineered prompts to generate precise semantic triples. This approach effectively resolves the low precision problem typically encountered in relation extraction. Experiments demonstrate significant gains in both accuracy and efficiency across diverse domains, with our method ranking in the top 2 on the official leaderboard in Shared Task-IV of The 1st Joint Workshop on Large Language Models and Structure Modeling. This competitive performance validates our approach as a compelling solution for practitioners seeking robust document-level information extraction without the burden of task-specific fine-tuning. Our code can be found at https://github.com/dinhthienan33/ZeroSemble.

## introduction automatically extracting information from unstructured text is critical for knowledge discovery and management. more specifically, the shared task-iv of the 1st joint workshop on large language models and structure modeling -document-level information extraction (docie) challenge focuses * equal contributions. on retrieving not only entities and their types, but also all entity mention's corresponding semantic relations (relation triples) within long unstructured documents. this task covers 34 domains, which is a lot, showing how complex and generalized the solutions need to be. most information extraction systems have difficulty with the documentlevel linguistic ambiguity, heterogeneity, coreference, and cross-sentence relations. not to mention, they tend to be overly reliant on richly annotated datasets from single domains, stitching domainspecific training, which forms a significant barrier to rapid adaptation across the range of domains included in the docie dataset. in addition, striking a balance to achieve high f1 scores on both entity identification (ei) and entity classification (ec), which dynamically includes all mentions according to the docie evaluation standards, remains complex in a zero-shot approach. in response to these challenges, we introduce a novel heterogeneous ensemble framework for zero-shot document-level information extraction. our approach eliminates the need for domain-specific training by strategically combining three state-of-the-art llms with complementary strengths: deepseek-r1-distill-llama-70b (ai, 2025), llama-3.3-70b-versatile @xcite , and qwen-2.5-32b @xcite . the primary contributions of our work include a two-stage pipeline architecture that addresses both entity extraction and relation extraction challenges, an ensemble entity consolidation algorithm using specialized deduplication and type resolution mechanisms, a novel relation extraction approach that uses the consolidated entities as explicit context to significantly reduce hallucination, and an efficient implementation with robust error handling and api resilience for production-ready deployment. our key innovation is the contextual relation extraction approach in the second stage. rather than naively combining relation outputs from individ-ual models or performing a complete re-extraction, we prompt qwen-2.5-32b with the validated entity set from stage one. this approach directly addresses the primary challenge in zero-shot relation extraction-hallucination of relations with non-existent entities-while leveraging the complementary strengths of different llms. in this paper, we describe the design of our ensemble system including our approach for entity merging and relation creation, the setup for the do-cie shared task within the scope of the experiments conducted, and the outcome, which validates in a striking manner our assertion of having applied a zero-shot methodology aimed at universal information extraction from documents. ## related work significant advancements have been made in document-level information extraction in recent years. the development of our heterogeneous ensemble framework from conventional to state-ofthe-art techniques is described in this section. ## zerosemble: system architecture and implementation without requiring domain-specific training, ze-rosemble uses three cutting-edge large language models to implement a novel zero-shot heterogeneous ensemble approach for document-level information extraction. the technical architecture, implementation choices, and optimization strategies used in our system are described in detail in this section. figure 1 shows the two-stage pipeline architecture used by zerosemble. three different llms-deepseek-r1-distill-llama-70b, llama-3.3-70b, and qwen-2.5-32b-are used in parallel entity extraction in the first stage. the ensemble algorithm, which is implemented in the combine.py module, is then used to consolidate the entities. our deliberate choice of these models drew on their unique architectural advantages as demon-strated by current comparative studies. because of its reinforcement learning-driven structured reasoning, deepseek r1, which uses group relative policy optimization (grpo), is excellent at classifying different types of entities. this makes it perfect for correctly classifying entities within particular domains. thanks to its broad context window, llama-3.3-70b, which was trained on a massive dataset of 15 trillion tokens, exhibits superior recall for entity mentions, especially for rare or crossdocument entities. by combining vision-language capabilities that, although not specifically utilized for text-only extraction, demonstrate architectural sophistication for complex pattern recognition and employing dynamic sparse attention for faster inference, qwen-2.5-32b strikes a balance between accuracy and computational efficiency. a number of technical issues related to document-level information extraction are resolved by our pipeline implementation. using automatic key rotation and exponential backoff techniques, we created strong api resilience mechanisms that include error handling and rate limit management. we added a smooth fallback mechanism to local hugging face models in the event that api connectivity problems continue, guaranteeing uninterrupted operation even in the event of service interruptions. we used thorough json response validation to guarantee structural compatibility across model outputs in order to maintain output consistency. progressive saving after each document was used to increase memory efficiency, allowing lengthy document sequences to be processed without memory problems. ## experimental results the xllm @ acl 2025 shared task-iv: universal document-level information extraction dataset, which consists of 248 documents from various domains, is used in this section to empirically evaluate our zerosemble approach. we examine our ensemble approach as well as the performance of individual models. ## conclusion zerosemble, a novel method for zero-shot document-level information extraction based on heterogeneous llm ensembles, is presented in this paper. we presented a two-step pipeline that uses the high-confidence entity set that is produced to constrain and enhance relation extraction after first combining entity extractions from several cutting-edge llms (deepseek-r1-distill-llama-70b, llama-3.3-70b, and qwen-2.5-32b). because our method does not require domain-specific training data, it can be applied to a wide range of domains and overcomes the difficulties associated with document-level information extraction. thus, with an overall score of 22.49 (mean of four evaluation metrics: entity identification @xmath0.65, entity classification @xmath1.11, re general mode @xmath2.19, and re strict mode @xmath3.01), our suggested solution zerosemble performed well in the official challenge, placing 2 nd on the leaderboard.