Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding (2023)

Abstract: Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding (VDU). While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines. In this paper, we propose Wukong-Reader, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines. Experiments show that Wukong-Reader brings superior performance on various VDU tasks in both English and Chinese. The fine-grained alignment over textlines also empowers Wukong-Reader with promising localization ability.

## introduction visual document understanding (vdu) handles various types of digital-born or scanned documents like forms, tables, reports, or research papers, and is becoming increasingly important for real-world industrial practices @xcite . multi-modal pre-training on millions of documents is a popular solution for visual document understanding @xcite @xcite @xcite . unlike the conventional vision-language pre-training over natural images and their paired short and abstractive descriptions @xcite @xcite , the document texts are usually long and highly correlated with the images, since they can be easily figure 1 : samples of document textlines: a) a letter from funsd @xcite with keys (blue) and values (green); and b) a receipt from sroie @xcite with the restaurants name (dark blue) and the address (orange). towards that end, existing efforts seek to align the visual and textual knowledge of documents at different levels. a commonly used pre-training objective for documents is masked language modeling @xcite over document text tokens @xcite @xcite @xcite , often accompanied by the layout information encoded via the positional embedding. besides, various visual and vision-language multimodal pre-training objectives are also proposed, leveraging the patch-level features @xcite , objectlevel features from object detectors @xcite , or the whole image feature through a global text-image matching loss @xcite . however, as an intrinsic granularity for vdu, document textlines have been mostly neglected in past efforts. intuitively, a textline contains a set of words that are spatially and semantically related. for instance of information extraction, the desired text span (e.g., the names on letters and addresses on receipts in figure 1 ) often appears in a single textline. therefore, the document textline serves as an appealing fine-grained granularity for vdu tasks. while structuallm @xcite similarly considers textlines as cell layout information, they only use the textual features of these textlines in language modeling. instead, in this work, we seek to enhance the multi-modal representation of a document by aligning the visual region and text span corresponding to the same textline. in this work, we propose wukong-reader, a pre-trained document model with a hybrid dualand single-stream multimodal architecture. to learn fine-grained document representation, we propose the textline-region contrastive learning to align the visual and textual features of document textlines from the dual-stream encoders. the objective thus connects the spatial and semantic information among document textlines for various vdu tasks. additionally, we also introduce two other objectives to further improve the textline representation. we design the masked region modeling to recover the masked textline regions, so as to enhance the visual features of textline. we also propose the textline grid matching to strengthen the layout information of textlines, which localizes each word of textlines to the pre-defined image grids. similar to previous works @xcite @xcite , the classic masked language modeling objective is also applied over document texts. experimental results show that our wukong-reader notably improves various document understanding tasks across both english and chinese. for instance, wukong-reader large with 470m parameters achieves the weighted f1 score of 93.62 on funsd @xcite and 98.15 on sroie @xcite , leading the new state-of-the-art records on information extraction tasks. we also demonstrate that the textline-based pre-training objectives empower the model with meaningful textline features with promising localization ability. ## related work visual document understanding (vdu) has been widely studied in recent years @xcite @xcite @xcite . vdu tasks are abundant in textual and visual information, as intensive texts and their layout information can be extracted from documents via optical character recognition (ocr) or other document parsers. muti-modal pre-training has been a popular solution for vdu. usually, a pre-trained text encoder (e.g., bert @xcite ; roberta @xcite ) is applied to learn contextualized representations of the textual input. meanwhile, a pre-trained visual encoder such as cnn-based @xcite and transformer-based @xcite models are applied to process visual features. various self-supervised pre-training objectives over millions of documents have shown promising effects for vdu. reconstructive objectives such as masked language modelling (mlm) @xcite , and masked image modelling, (mim) @xcite , are often used to perform the self-supervised document pretraining @xcite . since the textual knowledge is parsed from the document image, existing efforts explore various document granularities to align the vision and language modalities. they can be generally divided into four categories: 1) word-level: lay-outlm @xcite jointly models the inner-relationship between texts and layout 2d positions from documents, via pre-trained language models @xcite . however, the visual features are not used in the pretraining architecture. tilt @xcite additionally adds a contextualized image embedding to the word embedding. 2) grid/patch-level: layoutlmv2 @xcite , docformer @xcite and ernie-layout @xcite extract image grid features with cnn backbones, and layoutlmv3 uses image patches to encode visual features inspired by vit @xcite . to achieve the cross-modal alignment, they adopt the textimage alignment (i.e., tia) and matching (i.e., tim) objectives during pre-training. 3) objectlevel: selfdoc @xcite and unidoc @xcite extract object features via document object detectors, and concatenate them with word features. selfdoc @xcite uses two cross-modality attention functions to identify the inner-relationships from one modality to another. unidoc @xcite designs the similaritypreserving knowledge distillation to encourage alignment between words and visual features. 4) cell-level: structuallm @xcite uses the textual features of cell layout information, which is similar to document textlines. however, it only considers the textual feature without the visual information. different from existing works, we target at the textline-level features of both textual and visual modalities. we propose a hybrid dual-and singlestream model architecture for multi-modal pretraining. similar architectures are previously explored in general multi-modal pretraining @xcite , however, they mostly focus on learning the global visual and textual features over natural images. instead, we aim to align the fine-grained knowledge nested in document textlines. we believe such an important granularity of documents can benefit both language and visual representation learning in vdu tasks. ## methodology we propose wukong-reader, a new pre-trained multi-modal model for visual document understanding. our model jointly encodes the visual image and textual tokens via two mono-modal encoders, followed by a multi-modal encoder to fuse the two modalities. to leverage the structural information nested in document textlines, we propose several novel pre-training objectives for finegrained representation learning of documents. ## experiments in this section, we empirically verify the proposed wukong-reader across different vdu tasks in both english and chinese. we first introduce the experimental setup and the main results on english vdu tasks in section 4.1 and section 4.2, respectively. section 4.3 provides further discussion, e.g., ablations and localization abilities. finally, we also investigate the ability of wukong-reader in chinese in section 4.4. the implementation of wukong-reader is based on mindspore @xcite . ## conclusion in this paper, we propose wukong-reader, a multi-modal pre-trained model for fine-grained visual document understanding. unlike existing solutions that ignore the intrinsic textual segment information, our wukong-reader aims to leverage the semantics in textline regions of documents, by aligning with the visual and textual contents over document textlines via textline-region contrastive learning. meanwhile, we also propose masked region modeling and textline grid matching to further enhance the visual and layout information of document textlines. we evaluate wukong-reader on various visual document understanding tasks such as information extraction and document classification, and the proposed model demonstrates superior performance against previous counterparts.