m PLUG - D oc O wl 1.5: Unified Structure Learning for OCR -free Document Understanding (2024)

Abstract: Structure information is critical for understanding the semantics of text-rich images, such as documents, tables, and charts. Existing Multimodal Large Language Models (MLLMs) for Visual Document Understanding are equipped with text recognition ability but lack general structure understanding abilities for text-rich document images. In this work, we emphasize the importance of structure information in Visual Document Understanding and propose Unified Structure Learning to boost the performance of MLLMs. Based on publicly available text-rich images, we build a comprehensive training set DocStruct4M to support structure-aware parsing tasks and multi-grained text localization tasks across 5 domains: document, webpage, table, chart, and natural image. To better encode structure information, we design a simple and effective vision-to-text module H-Reducer, which can not only maintain the layout information but also reduce the length of visual features by merging horizontal adjacent patches through convolution, enabling the LLM to understand high-resolution images more efficiently. Our model DocOwl 1.5 achieves state-of-the-art performance on 10 visual document understanding benchmarks. All codes, models, and datasets are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.

## introduction textual information in images manifests with a multitude of visual structures, spanning the simplicity of plain text to the systematic grid layouts of tables and incorporating a spectrum of graphical representations such as pie, line, and bar charts. these elements may appear in isolation or be intricately interwoven within the framework of documents and webpages, reflecting a rich diversity of informational architecture across posters, invoices, * corresponding authors infographics, scientific reports, academic and news websites, etc. as shown in fig. 1 , besides the basic textual content, structure information also plays a big role in visual document understanding @xcite @xcite . with basic abilities to understand general images and comprehend structured texts through the llm decoder, mllms @xcite @xcite have the potential to achieve unified structure learning on text-rich images. however, due to the visual encoder and vision-to-text module being trained on general image-text pairs and not specifically optimized to represent the textual and structural information in text-rich images, they still face great challenges with images with rich text information. for better visual document understanding with mllms, some works @xcite @xcite attempt to design text-reading tasks to strengthen the text recognition ability, but either ignore the structure comprehension or only cover limited domains, such as just webpages @xcite or documents @xcite . in this work, we first perform unified structure learning on text-rich images for mllms across 5 domains: document, webpage, table, chart, and natural image. for better structural understanding, we first design a simple and effective vision-to-text module, namely h-reducer. unlike the resampler @xcite or q-former @xcite which fuses visual features with learnable queries but affects spatial information, the h-reducer accumulates neighborhood visual features through convolution to keep the relative positional relationships. compared with v2t modules with only linear layers @xcite , it produces much fewer visual features, which is more efficient for llm to understand high-resolution document images. considering texts in document images are most organized from left to right, h-reducer merges visual features at the horizontal level. our unified structure learning comprises structure-aware parsing tasks and multi-grained text localization tasks. to learn the organization of text contents, the former mainly teaches the model to parse the texts in the image in a structure-aware style, such as using line feeds and spaces to represent the structure of documents or webpages, and using extended markdown syntax to represent the structure of tables and charts. multi-grained text localization tasks further enhance the ability to correlate visually situated texts and concrete positions in the image. to support unified structure learning, based on publicly available datasets, we carefully build a comprehensive training set docstruct4m by constructing structure-aware sequences and multigrained pairs of text and bounding boxes. the docowl 1.5 is first trained with the unified structure learning and then followed by the multi-task tuning among downstream tasks. our contributions in this work are three-fold: visual document understanding (vdu) @xcite , aims to comprehend images with rich text information. such images range from documents @xcite @xcite @xcite , tables @xcite @xcite , charts @xcite @xcite @xcite @xcite , natural images @xcite @xcite to webpage screenshots @xcite , where diverse composition of text and visual objects contains a wealth of information. according to whether relying on an off-the-shelf ocr system to recognize texts in the image, models for visual document understanding can be categorized into ocr-dependent models @xcite @xcite and ocr-free ones @xcite . to leverage recognized texts from an ocr system, ocr-dependent models are always trained to align textual and visual inputs, such as recovering masked text and layout information given image and retained text as inputs @xcite . as for ocr-free methods, training with tasks about text recognition is indispensable. dount @xcite designs the text reading task to output continuous text sequences that ignore structure information. to leverage structure information, pix2struct @xcite designs a screenshot parsing task to generate the html dom tree for webpage screenshots but is hard to apply to other types of images. in this work, we first propose unified structure learning for all image types and carefully build a comprehensive dataset to support layout learning. multimodal large language models (mllm) have shown strong vision understanding and openended conversation abilities @xcite @xcite @xcite for natural images. they follow the architecture paradigm of connecting a vision encoder,e.g. vit @xcite , with a large language model (llm) @xcite @xcite ) by a visionto-text (v2t) module, such as simple linear layers @xcite or a q-former @xcite /resampler @xcite /abstractor @xcite with learnable queries. to enable mllms to comprehend images with rich texts, there are major two challenges: how to encode high-resolution images and how to understand visually-situated texts. to tackle highresolution images, most works choose to further train @xcite or extraly add a high-resolution vision encoder @xcite . ureader @xcite ) first proposes to keep the low-resolution vision encoder and use a shape-adaptive cropping module to crop raw images into multiple sub-images with low resolution. to enhance the visually-situated text understanding, some work design tasks of reading texts from top-left to bottom-right without taking into account the importance of structure @xcite . cogagent @xcite and docpedia @xcite further try strengthening the layout understanding for documents, webpages, and natural images with text grounding tasks. however, the comprehension of the overall structure is ignored, and tables and charts are not covered. in this work, to strengthen structure understanding, we design structure-aware praising and multi-grained text localization tasks for all types of images, covering documents, tables, charts, webpages, and natural images. besides, we propose a vision-to-text architecture h-reducer to better maintain spatial information of visual features by convolution. compared with minigpt-v2 @xcite , h-reducer merges adjacent tokens with a convolution layer while minigpt-v2 con-catenates 4 tokens and merges them with a linear layer. the convolution layer is more convenient for adjusting the merging scope, enabling us to perform an ablation study of different window sizes to determine the most suitable one for document understanding. the c-abstractor @xcite comprises 18 convolution layers with 1x1 sliding window and 1 mean pooling layer, showing that it doesn't reduce tokens with the convolution, but average pooling. however, h-reducer just applies 1 convolution layer with a 1x4 window size to reduce vision tokens, with much fewer parameters and achieving comparable performance. docowl 1.5 consists of a visual encoder, a large language model, and a vision-to-text module, namely h-reducer to better keep the visual text and layout information by ensembling horizontal visual features. it's first trained with unified structure learning to enhance text recognition and structure understanding and then tuned on multiple downstream tasks of visual document understanding. ## conclusion to enhance the visual document understanding performance of mllms, we first propose unified structure learning across 5 domains of textrich images, including both structure-aware parsing tasks and multi-grained text localization tasks. to better maintain structure and spatial information during vision-and-language feature alignment, we design a simple and effective vision-to-text module, named h-reducer. it mainly utilizes a convolution layer to aggregate horizontally neighboring visual features. to support the unified structure learning, we build docstruct4m by collecting publicly available images and carefully constructing structure- human: what is the dept.no? ureader: 76/77-142 (×) docowl 1.5: 218-12 (√) human: which edition has unlimited remote desktop services and virtulization rights? ureader: enterprise edition (×) docowl 1.5: standard (×) ground truth: datacenter human: what is the forecast for the increase in customs duty revenue in 2030? ureader: 90.5 (×) docowl 1.5: 100 (√) human: who did clinical research? ureader: steve haas / art o'neal (×) docowl 1.5: steve peoples (√) (a) (b) (c) (d) aware text sequences and multi-grained pairs of texts and bounding boxes. with unified structure learning, our model docowl 1.5 achieves state-ofthe-art ocr-free performance on 10 visual document understanding benchmarks.