Visually Guided Generative Text-Layout Pre-training for Document Intelligence (2024)

Abstract: Prior study shows that pre-training techniques can boost the performance of visual document understanding (VDU), which typically requires models to gain abilities to perceive and reason both document texts and layouts (e.g., locations of texts and table-cells). To this end, we propose visually guided generative text-layout pre-training, named ViTLP. Given a document image, the model optimizes hierarchical language and layout modeling objectives to generate the interleaved text and layout sequence. In addition, to address the limitation of processing long documents by Transformers, we introduce a straightforward yet effective multi-segment generative pre-training scheme, facilitating ViTLP to process word-intensive documents of any length. ViTLP can function as a native OCR model to localize and recognize texts of document images. Besides, ViTLP can be effectively applied to various downstream VDU tasks. Extensive experiments show that ViTLP achieves competitive performance over existing baselines on benchmark VDU tasks, including information extraction, document classification, and document question answering.

## introduction processing and reasoning document images with dense texts (e.g., scanned pdf files, digital forms, and spreadsheets) is a persistent yet challenging task for the research community and industry @xcite @xcite . advances in multimodal pre-training substantially improve the performance of visual document understanding (vdu) @xcite @xcite @xcite . these pre-training methods typically take multimodal inputs of given document images including i) visual features, ii) pre-processed ocr texts, and iii) spatial layouts of document elements (e.g., 2d coordinates of texts and table-cells). among these inputs, spatial layout information plays an essential role in connecting visual and textual features, as well as developing thorough reasoning of document structures @xcite . though effective, the performance of most existing vdu approaches relies heavily on the ocr pipelines, because the pre-processed ocr texts and corresponding 2d coordinates are used as intermediate inputs to pre-trained vdu models. the external ocr pipelines may produce incorrect or incomplete recognition results, which cannot be jointly optimized by the gradient back from vdu models. another research line @xcite ) explores pre-training vdu models solely based on image inputs. despite no ocr errors introduced, these methods focus on understanding texts from raw document images but neglect layout information modeling. since the spatial information contained in layout locations is not exploited, it may hinder the models from understanding complex document structures, especially for documents containing nested paragraphs, forms, and tables. in this work, we propose visually guided generative text-layout pre-training (vitlp) to jointly model text and layout information from document images. as shown in figure 1 , vitlp can localize, recognize, and understand visual document texts given the input document image and task prefixes. to achieve this goal, vitlp is pre-trained to generate unified text-layout sequences from document images. since natively generating text and layout tokens in a flattened sequence is token-inefficient (see sec. 2.1), we introduce hierarchical generation modules to achieve both effective and efficient text-layout sequence generation. to the best of our knowledge, vitlp is the first attempt to learn ocr (i.e., text localization and recognition) and vdu (i.e., document understanding) abilities in a unified generative text-layout pre-training framework. besides, vitlp is designed to handle long documents with intensive texts. long document processing is ubiquitous in real-world scenarios. however, existing pre-trained models are constrained to certain token limits of input sequences. for instance, layoutlmv2 @xcite accepts the maximum inputs of 512 word tokens using a bertstructure encoder. in both pre-training and finetuning, the exceeded text tokens are truncated, leading to incomplete document information modeling. to tackle this issue, we introduce a multi-segment pre-training scheme which divides the target textlayout sequence into consecutive segments to perform generative pre-training. given that the full document information is already encoded in visual representations, vitlp takes the suffix tokens from previous segments as prefix prompts to generate the next-segment tokens. this multi-segment pre-training scheme further enables vitlp to process documents of arbitrary length in fine-tuning. notably, our multi-segment generation scheme retains the intact transformer architecture. thus, it is more feasible than other long-document modeling workarounds, e.g., sparse attention @xcite and memory modules @xcite , which need to modify the transformer architecture and may affect the capacity of pre-trained models. we evaluate vitlp on a variety of ocr and vdu tasks. experiment results demonstrate that vitlp can achieve superior overall performance on both ocr and vdu tasks. for instance, vitlp achieves the 95.59% f1 score on cord information extraction and 95.36% accuracy on rvl-cdip document classification, both of which outperform most previous approaches. notably, vitlp can intrinsically generate 2d layout locations for visual grounding, which helps in certain generative vdu tasks (e.g., visual document question answering) to be more interpretable and reliable to humans. ## related work visual document processing with multimodal pretraining is widely studied. from the perspectives of the document processing pipelines and model architectures, existing works can be generally divided into strands of research as listed below. ocr-based methods. most initial vdu efforts adopt ocr tools to localize and recognize document layouts and texts, and then feed them to the multimodal pre-trained models @xcite @xcite @xcite @xcite . these methods usually involve multiple pre-training objectives over the vision, text, and layout. for instance, document text location @xcite , paragraph and table regions @xcite are rich in structural information to align visual features with text embeddings. though promising, these pipeline models suffer from heavy ocr pre-processing overhead. moreover, incorrect ocr results may propagate errors to downstream tasks like document question answering @xcite . ## conclusion we propose visually guided generative text-layout pre-training (vitlp) to enhance visual document processing covering the ocr and vdu tasks. in the pre-training phase, vitlp optimizes hierarchical language and layout modeling objectives to generate interleaved text-layout target sequences. moreover, the proposed multi-segment pre-training scheme enables vitlp to process long documents with arbitrary lengths. vitlp can function as a native ocr model to locate and recognize texts of document images. experiments also show that vitlp achieves superior performance on various vdu tasks with document grounding capability.