Intelligent Document Parsing: Towards End-to-end Document Parsing via Decoupled Content Parsing and Layout Grounding (2025)

Abstract: In the daily work, vast amounts of documents are stored in pixel-based formats such as images and scanned PDFs, posing challenges for efficient database management and data processing. Existing methods often fragment the parsing process into the pipeline of separated subtasks on the layout element level, resulting in incomplete semantics and error propagation. Even though models based on multi-modal large language models (MLLMs) mitigate the issues to some extent, they also suffer from absent or sub-optimal grounding ability for visual information. To address these challenges, we introduce the Intelligent Document Parsing (IDP) framework, an end-to-end document parsing framework leveraging the vision-language priors of MLLMs, equipped with an elaborately designed document representation and decoding mechanism to decouple the content parsing and layout grounding to fully activate the potential of MLLMs for document parsing. Experimental results demonstrate that the IDP method surpasses existing methods, significantly advancing MLLM-based document parsing.

## introduction in contemporary settings, a substantial volume of information is produced daily and stored within pixel-based representations, such as images and scanned pdfs, rather than arranged in machineunderstandable structured formats, such as json and html. this scenario presents considerable challenges in practice, as structured formats are indispensable for efficient database storage and standardized data processing @xcite , as well as for downstream applications, including information retrieval and natural language processing @xcite @xcite . document parsing @xcite @xcite aims at extracting the content of unstructured documents. the task is challenging due to the diversity of the layout and logical structures @xcite , incorporating a plethora of plain text, visual elements, such as images and icons, and multi-modal information, such as tables and charts. this diversity and complexity require document parsing technologies to possess strong adaptability and generalization capabilities to handle various types of documents effectively @xcite . most existing document parsing methods lever-age the pipelines of separated subtasks, as shown in fig. 1 (a), including document layout analysis @xcite , reading order prediction @xcite , and document hierarchy parsing @xcite . the main drawbacks of the pipeline methods are: 1) by encoding fragmented layout elements independently, these methods fail to capture the holistic information of documents, and thus result in semantic loss. there have also been attempts to apply advanced large language models (llms) to conduct end-toend document parsing @xcite @xcite as an image-to-markdown task, as shown in fig. 1 (b) ; however, the markdown representation of these methods fails to explicitly capture the multi-modal elements such as images, icons and charts. @xcite introduce the layout grounding task to mitigate this issue, as shown in fig. 1 (c ). but the method results in suboptimal performance in both content parsing and layout grounding tasks, due to the distractions between the two heterogeneous tasks @xcite @xcite , significantly affecting the document parsing performance. to address this issue, in this paper, we propose an end-to-end framework, intelligent document parsing (idp), which leverages the powerful visionlanguage priors of mllms to build an effective document parsing method that possesses exceeding effectiveness. as illustrated in fig. 1 (d ), the idp framework decouples the content parsing and layout grounding tasks, allowing different decoder modules to focus on the tasks they excel at. this approach fully unleashes the potential of mllms in the document parsing situation. specifically, on one hand, we developed the idp representation format, akin to html sequences and integrating information of layout elements, their reading order, and logical relationships. on the other hand, we addressed the issue of conflicts between con-tent parsing and layout grounding by introducing the decoupled decoding mechanism. experimental results show that this model outperforms leading mllms and shows advantages compared to pipelines composed of task-specific models. our main contributions can be summarized as follows: ## idp document rerpresentation previous llm-based document parsing methods @xcite often follow the markdown generation paradigm. however, important multi-modal information, such as images and charts, is omitted in this format. therefore, in recent studies @xcite , researchers still prefer using parsing results that include bounding boxes of layout elements. in this paper, we consider the document parsing task as recognizing the layout elements and organizing them in an ordered and hierarchical structure. specifically, the input is given as a document as image i. the output is the extracted layout elements @xmath0, e 2 , ..., e m } in traversal order, along with their hierarchical depth predicted. each layout elements e i is represented by the corresponding bounding boxes b @xmath1, y i1 , x i2 , y i2 ], categories c i , depth l i in the document structure tree and the contents. in order to empower an end-to-end model with comprehensive capabilities for parsing and extracting both text and multi-modal content from documents, we propose the document representation of idp format, as shown in fig. 2 . it decomposes the content-inferred task and the visual grounding task. specifically, the layout elements are uniformly formatted as html tags, which integrates reading orders (sequential order of tags), categories (attribute 'class'), and hierarchical depth (attribute 'level'). the tag contents of textual layout elements are the extracted text, while those of multi-modal elements are corresponding place holders. the positional information of layout elements is stored in an auxiliary data item, which is a list of bounding boxes. the correspondence between layout elements and bounding boxes is maintained through a one-to-one mapping based on unique ids. in this way, the idp format caters better to the prior knowledge of mllms. it addresses the challenge faced by the markdown format in preserving multi-modal information. additionally, it avoids coupling layout grounding results within the prediction sequence, thereby allowing mllm to focus more effectively on document content parsing and organization. ## idp model in order to avoid conflicts among content-inferred and visual-spatial tasks @xcite and fully activate the potential of mllms, we propose a decoupled decoding mechanism to disentangle the task into two simultaneous sequential and spatial decoding processes. the overall architecture of the idp model is depicted in fig. 3 . it consists of 3 parts: (1) an image encoder for encoding the document images; (2) an llm that models the multi-modal inputs and generates doc-ument contents organized in html; (3) a spatial decoder for the layout grounding task. in the decoding process, the layout grounding process is triggered by a generated layout element by the llm, and it predicts grounding results based on layout queries from the llm. the following subsections specify these crucial components and the decoding process, respectively. ## conclusion in this paper, we introduce the intelligent document parsing (idp) framework, an end-to-end document parsing framework leveraging the visionlanguage priors of mllms, equiped with elaborately designed idp representation and fully activates the mllms ability on content parsing while generates the layout grounding results in a decoupled module. experimental results demonstrate that idp method effectively boosts the mllmbased methods, and illustrate advantage to the pipeline methods in terms of end-to-end and zeroshot performance, providing a significant advancement in mllm-based document parsing.