M arkup LM : Pre-training of Text and Markup Language for Visually Rich Document Understanding (2022)

Abstract: Multimodal pre-training with text, layout, and image has made significant progress for Visually Rich Document Understanding (VRDU), especially the fixed-layout documents such as scanned document images. While, there are still a large number of digital documents where the layout information is not fixed and needs to be interactively and dynamically rendered for visualization, making existing layout-based pre-training approaches not easy to apply. In this paper, we propose MarkupLM for document understanding tasks with markup languages as the backbone, such as HTML/XML-based documents, where text and markup information is jointly pre-trained. Experiment results show that the pre-trained MarkupLM significantly outperforms the existing strong baseline models on several document understanding tasks. The pre-trained model and code will be publicly available at https://aka.ms/markuplm .

## introduction multimodal pre-training with text, layout, and visual information has recently become the de facto approach @xcite @xcite @xcite @xcite @xcite in visually-rich document understanding (vrdu) tasks. these multimodal models are usually pre-trained with the transformer architecture @xcite using large-scale unlabeled scanned document images @xcite or digital-born pdf files, followed by task-specific fine-tuning with relatively small-scale labeled training samples to achieve the state-of-the-art performance on a variety of document understanding tasks, including form understanding @xcite , receipt understanding @xcite , complex document understanding @xcite , document type classification @xcite , and document visual question answering @xcite , etc. significant progress has been witnessed not only in research tasks within academia, but also in different real-world business applications such as finance, insurance, and many others. visually rich documents can be generally divided into two categories. the first one is the fixed-layout documents such as scanned document images and digital-born pdf files, where the layout and style information is pre-rendered and independent of software, hardware, or operating system. this property makes existing layout-based pre-training approaches easily applicable to document understanding tasks. while, the second category is the markup-language-based documents such as html/xml, where the layout and style information needs to be interactively and dynamically rendered for visualization depending on the software, hardware, or operating system, which is shown in figure 1 . for markup-language-based documents, the 2d layout information does not exist in an explicit format but usually needs to be dynamically rendered for different devices, e.g., mobile/tablet/desktop, which makes current layoutbased pre-trained models hard to apply. therefore, it is indispensable to leverage the markup structure into document-level pre-training for downstream vrdu tasks. to this end, we propose markuplm to jointly pre-train text and markup language in a single framework for markup-based vrdu tasks. distinct from fixed-layout documents, markup-based documents provide another viewpoint for the document representation learning through markup structures because the 2d position information and document image information cannot be used straightforwardly during the pre-training. instead, marku-plm takes advantage of the tree-based markup structures to model the relationship among different units within the document. similar to other multimodal pre-trained layout-based models, marku-plm has four input embedding layers: (1) a text embedding that represents the token sequence information; (2) an xpath embedding that represents the markup tag sequence information from the root node to the current node; the third is the title-page matching (tpm), where the content within "<@xmath0... </@xmath1. in this way, markuplm can better understand the contextual information through both the language and markup hierarchy perspectives. we evaluate the markuplm models on the web-based structural reading comprehension (websrc) dataset @xcite and the structured web data extraction (swde) dataset @xcite . experiment results show that the pre-trained markuplm significantly outperforms the several strong baseline models in these tasks. the contributions of this paper are summarized as follows: ## markuplm markuplm utilizes the dom tree in markup language and the xpath query language to obtain the markup streams along with natural texts in markuplanguage-based documents (section 2.1). we propose this transformer-based model with a new xpath embedding layer to accept the markup sequence inputs (section 2.2) and pre-train it with three different-level objectives, including masked markup language modeling (mmlm), node relation prediction (nrp), and title-page matching (tpm) (section 2.3). ## experiments in this work, we apply our markuplm framework to html-based webpages, which is one of the most common markup language scenarios. equipped with the existing webpage datasets common crawl (cc) foot_0 , we pre-train markuplm with large-scale unlabeled html data and evaluate the pre-trained models on web-based structural reading comprehension and information extraction tasks. ## conclusion and future work in this paper, we present markuplm, a simple yet effective pre-training approach for text and markup language. with the transformer architecture, markuplm integrates different input embeddings including text embeddings, positional embeddings, and xpath embeddings. furthermore, we also propose new pre-training objectives that are specially designed for understanding the markup language. we evaluate the pre-trained markuplm model on the websrc and swde datasets. experiments show that markuplm significantly outperforms several sota baselines in these tasks. for future research, we will investigate the markuplm pre-training with more data and more computation resources, as well as the language expansion. furthermore, we will also pre-train markuplm models for digital-born pdfs and office documents that use xml dom as the backbones. in addition, we will also explore the relationship between markuplm and layout-based models (like layoutlm) to deeply understand whether these two kinds of models can be pre-trained under a unified multi-view and multi-task setting and whether the knowledge from these two kinds of models can be transferred to each other to better understand the structural information.