AID -Agent: An LLM -Agent for Advanced Extraction and Integration of Documents (2025)

Abstract: Extracting structured information from complex unstructured documents is an essential but challenging task in today’s industrial applications. Complex document content, e.g., irregular table layout, and cross-referencing, can lead to unexpected failures in classical extractors based on Optical Character Recognition (OCR) or Large Language Models (LLMs). In this paper, we propose the AID-agent framework that synergistically integrates OCR with LLMs to enhance text processing capabilities. Specifically, the AID-agent maintains a customizable toolset, which not only provides external processing tools for complex documents but also enables customization for domain and task-specific tool requirements. In the empirical validation on a real-world use case, the proposed AID-agent demonstrates superior performance compared to conventional OCR and LLM-based approaches.

## introduction automatic extraction and integration of document information is an essential task for digitalization. unstructured data in pdf documents are traditionally manually extracted and integrated into a structured schema. the development of advanced ocr techniques leverages the automation of machine extractions in many scenarios; nevertheless, the information recognized by ocr engines remains at a synthetic and symbolic level, which leads to massive failures in complex documents, where the target information is not explicitly extractable. even some commercial tools (e.g., azure document intelligence foot_0 , google document ai foot_1 ) achieve superior ocr performance on general-purpose tasks, they may still fail when applied to domain-specific and complex documents. recently, multi-modal llms support directly prompting pdf documents as images and extracting information in a question-answering fashion. however, two major challenges hinder the usage of such models at scale. firstly, a complex document usually contains unstructured information (e.g., irregular table structure) or cross-references within the document, meaning that feeding the ocr-parsed text into an llm often leads to suboptimal results. although multimodal llms can incorporate layout information and prompt long context, at the current stage, they still result in a significant cost overhead than classical approaches. it is especially inefficient for long documents, where the relevant information only lies on a few pages. secondly, despite recent advances in the long-context feature of llms, holistically resolving target information from multiple spots in complex documents is still a challenging task. furthermore, in some domain-specific tasks, extracting professional entities is also beyond the capability of general-purpose llms, e.g., pictograms, chemical compositions, and technical abbreviations. to this end, we propose using llm-based agents to handle the extraction of complex documents, where external tools with centralized scheduling efficiently optimize the extraction process. agent ai systems enable ai models to solve complex tasks independently. specifically empowered by recent llms, llm-agents have shown superior capability in several application domains @xcite , e.g., gaming, robotics, and nlp. however, in the document information extraction domain, the research is still underexplored. a desired llmagent is supposed to extract document information and integrate it into the target schema in an endto-end fashion. task decomposition, reasoning, and scheduling are key abilities in this scenario. general and task-specific tools can accelerate the effective extraction procedure. finally, an internal validator should ensure consistency and validity based on available knowledge, in either a zero-shot or a few-shot setting. in this paper, we propose the aid-agent (advanced integration of documents-agent), an llmagent specialized in complex document information extraction. our contribution can be summarized as follows: ## aid-agent in this section, we introduce the proposed aidagent, i.e., an llm-agent for advanced extraction and integration of documents. the architecture of aid-agent is visualized in figure 1 . the aid-agent is designed to process complex document extraction tasks end-to-end. it accepts pdf documents and the target extraction schema in json format as inputs and outputs the filled json schema after extraction. internally, aid-agent consists of three major components, an executor, a tool pool, and a validator. the executor is implemented using an llm, which is responsible for task decomposition, scheduling, and tool calling. this can be implemented, for example, using the react @xcite framework for efficient reasoning and action. the tool pool comprises a set of external tools for the executor. depending on the document property, the executor automatically selects tools to solve the extraction task. in the current stage, the automatic tool selection is based on the initial prompt, tool description as well as validator feedback. ultimately, we aim to train the executor using reinforcement learning for an optimal tool selection, refer to section 5 for more discussion. aiming at complex documents that lead to failures of naive llm-based approaches, we introduce three general tools in our tool pool. the table resolver organizes all structured tables in the document, which enables retrieving tabular information in an sql-fashion. the reference resolver targets cross-referenced information in the document and integrates multi- table resolver ple information sources in the document to generate the final answer. it is implemented using keyword matching and semantic embedding comparison. both table resolver and reference resolver return the retrieved relevant information back to the executor for the llm-based extraction. for certain documents with irregular and unstructured content, e.g., irregular tables with attached text descriptions, the vision analysis tool provides a powerful option to identify the bounding box of the target document area and use the vision language model (vlm) for the extraction. since only irregular, complex parts of the document are extracted using vlm, we limit the expense as much as possible. finally, for domain-and task-specific demands, we also leave a standard interface in the tool pool for customized tools, where specific professional tools can be implemented to leverage the llmextraction, e.g., chemical composition translator, and pictogram checklist. in the few-shot setting, a few extraction examples or domain expert guidance can also be included as a customized tool, which provides additional knowledge and regulation for the extraction. lastly, the validator interacts with the executor and provides an assessment of the current extraction quality, which will guide the tool selection action in the next iteration. a rule-based function validates compatibility of extraction formatting w.r.t. the target schema. furthermore, the consistency among similar extraction fields is ensured by using regular expressions. given the target schema and the current extraction, we also use llm-asa-judge to generate feedback to indicate possible improvements for the executor. once all fields are extracted (missing fields marked as none), or the agent reaches the predefined maximum iterations, the extraction procedure terminates. ## case study we present a case study of applying the aid-agent on technical report documents from our industrial partner. specifically, the goal is to extract metadata of the supplier information and chemical compositions of materials from each document. a toy example of the procedue is demonstrated in appendix a. we have collected 44 manually labeled pdf documents for evaluation. as evaluation metrics, we employ accuracy, precision, recall, and f1-score, while true positive is the number of correct extracted fields, false positive is the number of mistaken but non-empty fields and false negative is the number of mistaken empty fields. we evaluate the metrics per document and report the overall average and standard deviation. furthermore, in order to assess the effectiveness of tools employed by the agent, we also conduct multiple ablation studies that exclude one tool at a time, as well as replace the standard gpt-4o model with gpt-4o-mini model as the base model. and we include "ocr+llm" as a baseline solution where no additional tool is available. the evaluation results are summarized in generally, the aid-agent with complete tool access performs the best. with a 0.926 f1-score, the agent is able to extract the majority of desired information. the reasons for a few representative mistakes are poor ocr recognition, misalignment of complex tables, and integration of multiple extraction versions. since we did not include any treatment to rotated content, the performance on one 90 • -rotated document is significantly lower than others, even feeding the cropped image to the vlm did not improve the result. from the various ablation studies, we deter-mined the necessity of the toolset we designed. leaving out any of the tools will result in decreased performance. the most significant performance degradation is caused by replacing the base llm from gpt-4o to gpt-4o-mini. this result indicates that the necessary information is included either in the ocr-parsed text or the cropped images, and a sub-optimal llm or vlm can misuse it in the agent inference process. ## discussion in this paper, we have introduced the aid-agent for complex document extraction and integration. multiple tools and the validator support efficient reasoning and action of the executor during processing. the customizable tool pool also enables extensive extension flexibility for use case-specific challenges. compared to classical llm-based document extraction, which requires enormous effort in prompting engineering, the prompts used by the aid-agent executor and tools are reusable, while task-specific information can be included as a tool. the major improvement we are trying to bring to the aid-agent is the optimization of tool scheduling. at the current stage, the executor applies reasoning and action steps based on the extraction task in the prompts and tool descriptions. the llm of the agent executor makes the scheduling using all textual information within the agent. in the future, we aim to train the executor using a few-shot reinforcement learning approach. specifically, with labeled samples, the executor is supposed to optimize the order of tool calls to minimize the time and api-call costs. for instance, directly reading selectable pdfs is more time-efficient than ocr, and text-based llm extraction is more cost-efficient than image-based vlm extraction. these can be regularized as a reward function. at the current stage, we have conducted a case study using a limited set of proprietary pdf documents from a single domain. as a next step, we plan to evaluate the aid-agent on various public benchmark datasets, such as bigdocs @xcite . our primary focus is to facilitate the automation of industrial document processing tasks, including invoices, purchase orders, and delivery notes. furthermore, we aim to conduct comparative studies with recent document extraction models. specifically, we intend to compare aidagent with pipeline-based extraction tools (e.g., mineru @xcite , docling @xcite ) that perform step-by-step processing-from layout analysis and ocr to text integration-as well as with end-to-end vlms (e.g., got-ocr @xcite , smoldocling (nassar et al., 2025)) that directly produce structured outputs from input documents. finally, we also consider leveraging state-of-the-art llms, such as qwen @xcite and llama @xcite , as potential backbones for aid-agent.