XD oc: Unified Pre-training for Cross-Format Document Understanding (2022)

Abstract: The surge of pre-training has witnessed the rapid development of document understanding recently. Pre-training and fine-tuning framework has been effectively used to tackle texts in various formats, including plain texts, document texts, and web texts. Despite achieving promising performance, existing pre-trained models usually target one specific document format at one time, making it difficult to combine knowledge from multiple document formats. To address this, we propose XDoc, a unified pre-trained model which deals with different document formats in a single model. For parameter efficiency, we share backbone parameters for different formats such as the word embedding layer and the Transformer layers. Meanwhile, we introduce adaptive layers with lightweight parameters to enhance the distinction across different formats. Experimental results have demonstrated that with only 36.7% parameters, XDoc achieves comparable or even better performance on a variety of downstream tasks compared with the individual pre-trained models, which is cost effective for real-world deployment. The code and pre-trained models are publicly available at https://aka.ms/xdoc .

## introduction document understanding has undoubtedly been an important research topic as documents play an essential role in message delivery in our daily lives @xcite . during the past several years, the flourishing blossom of deep learning has witnessed the rapid development of document understanding in various formats, ranging from plain texts @xcite @xcite , document texts @xcite @xcite , and web texts @xcite @xcite . recently, pretraining techniques have been the de facto standard for document understanding, where the model is first pre-trained in a self-supervised manner (e.g. using masked language modeling as the pretext task @xcite ) with large-scale corpus, then fine-tuned on a series of downstream tasks like question-answering @xcite , key information extraction @xcite and many others. albeit achieving impressive performance on specific tasks, existing pre-trained models are far from flexible as they can only tackle texts in a single format (e.g. layoutlm @xcite is designed for tackling document texts and is not suitable for web texts). this makes it difficult to combine knowledge from multiple document formats. meanwhile, the category of pre-trained models will keep increasing if more formats (e.g. word and powerpoint) are further studied in academia. among different pre-trained models for document understanding, it is observed that many pre-figure 2 : illustrations of three document formats. for each format, the corresponding meta-information is shown in the dash boxes. please note that the text content and 1d position are common attributes across three formats while 2d position and xpath strings (marked as red) are specific for document and web texts respectively. to this end, we propose xdoc, a unified architecture with multiple input heads designed for various categories of documents. for the sake of parameter efficiency, we share the backbone network architecture across different formats, including the word embedding layer, the 1d position embedding layer, and dense transformer layers. considering that the different parts only take up a small proportion in xdoc, we introduce adaptive layers to make the representation learning for different formats more robust. we collect the large-scale training samples for different document formats, and leverage masked language modeling to pre-train xdoc. specifically, we use three widely-used document formats for experiments, including plain, document, and web texts (see figure 2 for more details). to verify the model accuracy, we select the glue benchmark @xcite and squad @xcite to evaluate plain text understanding, funsd @xcite and docvqa @xcite to evaluate document understanding, and websrc @xcite for web text understanding. experimental results have demonstrated that xdoc achieves comparable or even better performance on these tasks while maintaining the parameter efficacy. the contributions of this paper are summarized as follows: ## xdoc in this section, we first introduce the architecture of xdoc and details of the embedding used for each document format, then introduce the objectives for pre-training the xdoc model. ## experiments in this section, we first introduce the model configuration and detail the hyperparameters in xdoc, then introduce the pre-training strategies of xdoc. next, we demonstrate the experimental results on a wide range of downstream tasks. at last, we verify the effectiveness of some designs in xdoc and have a discussion. ## related work in this section, we review the pre-trained methods for document understanding, ranging from plain, document, and web texts, respectively. pre-trained methods for plain texts the understanding of plain texts through pre-training has been extensively studied during the last decade @xcite @xcite @xcite @xcite @xcite @xcite . for example, gpt @xcite utilizes transformer @xcite to conduct single-director masked-word prediction in an unsupervised manner. besides, bert @xcite utilizes two self-supervised tasks, including mask language modeling and next sentence prediction to obtain the robust representations of methods word 1d position transformer 2d position xpath adaptive total pre-trained methods for document texts benefiting from the public large-scale document dataset @xcite , pre-training has become the de facto standard for analyzing document texts @xcite @xcite @xcite @xcite @xcite . layoutlm @xcite makes the first attempt to combine the layout knowledge during pre-training to obtain robust contextual features for document texts. based on layoutlm, layoutxlm @xcite utilizes multilingual document text datasets for pre-training. structurallm @xcite jointly utilizes cell and layout information from scanned documents to make the representations more robust. layoutlmv2 @xcite introduces a multi-modal architecture by combining additional image tokens in the transformer. bros @xcite utilizes the token-masking and area-masking strategies for tackling information extraction tasks. xylayoutlm @xcite proposes an augmented xy-cut algorithm to exploit proper reading orders during pre-training. recently, layoutlmv3 @xcite pre-trains the text branch and image branch simultaneously using mask language modeling and mask image modeling tasks, which makes it a robust model for tackling text-centric and image-centric tasks. pre-trained methods for web texts compared with plain and document text analysis, the understanding of web texts is less studied and is more challenging since the layout of each website is not fixed (i.e. depending on the resolution of devices). markuplm @xcite takes the first attempt to incorporate web-based knowledge during pretraining while utilitzing three pretext tasks, including masked language modeling, node relation prediction, and title-page matching. further, based on markuplm, dom-lm @xcite introduces a new pre-training task predicting masked html node. webformer @xcite simultaneously feeds text features and image features to the multi-modal transformer while constructing rich attention patterns between these tokens. generally, although the mentioned methods show impressive performance in one specific format, they can not be transferred to tackle other formats. to mitigate this problem, the proposed xdoc is a scalable and flexible framework that is friendly to a wide range of formats, thus bringing much convenience for people. ## conclusion and future work in this paper, we propose xdoc, a unified framework that can tackle multiple document formats (e.g. plain, document, and web texts) in one model. for parameter efficiency, xdoc shares most parameters, including the word embedding, 1d position embedding, and transformer layers, across different document formats. the experimental results show that with only 36.7% parameters, xdoc can achieve comparable or even better performance on downstream tasks spanning various document formats. for future work, we will consider exploit-ing the image features during pre-training to tackle image-centric tasks and designing more unified pretraining tasks for various document formats.