L ayout LM v2: Multi-modal Pre-training for Visually-rich Document Understanding (2021)

Abstract: Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specifically, with a two-stream multi-modal Transformer encoder, LayoutLMv2 uses not only the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks, which make it better capture the cross-modality interaction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks, including FUNSD (0.7895 to 0.8420), CORD (0.9493 to 0.9601), SROIE (0.9524 to 0.9781), Kleister-NDA (0.8340 to 0.8520), RVL-CDIP (0.9443 to 0.9564), and DocVQA (0.7295 to 0.8672).

## introduction visually-rich document understanding (vrdu) aims to analyze scanned/digital-born business documents (images of invoices, forms in pdf format, etc.) where structured information can be automatically extracted and organized for many business * equal contributions during internship at msra applications. distinct from conventional information extraction tasks, the vrdu task relies on not only textual information but also visual and layout information that is vital for visually-rich documents. different types of documents indicate that the text fields of interest located at different positions within the document, which is often determined by the style and format of each type as well as the document content. therefore, to accurately recognize the text fields of interest, it is inevitable to take advantage of the cross-modality nature of visually-rich documents, where the textual, visual, and layout information should be jointly modeled and learned end-to-end in a single framework. the recent progress of vrdu lies primarily in two directions. the first direction is usually built on the shallow fusion between textual and visual/layout/style information @xcite @xcite @xcite @xcite . these approaches leverage the pre-trained nlp and cv models individually and combine the information from multiple modalities for supervised learning. although good performance has been achieved, the domain knowledge of one document type cannot be easily transferred into another, so that these models often need to be re-trained once the document type is changed. thereby the local invariance in general document layout (key-value pairs in a left-right layout, tables in a grid layout, etc.) cannot be fully exploited. to this end, the second direction relies on the deep fusion among textual, visual, and layout information from a great number of unlabeled documents in different domains, where pre-training techniques play an important role in learning the cross-modality interaction in an end-to-end fashion @xcite . in this way, the pre-trained models absorb cross-modal knowledge from different document types, where the local invariance among these layouts and styles is preserved. furthermore, when the model needs to be transferred into another domain with different document formats, only a few labeled samples would be sufficient to fine-tune the generic model in order to achieve state-of-the-art accuracy. therefore, the proposed model in this paper follows the second direction, and we explore how to further improve the pre-training strategies for the vrdu tasks. in this paper, we present an improved version of layoutlm @xcite , aka layoutlmv2. different from the vanilla layoutlm model where visual embeddings are combined in the fine-tuning stage, we integrate the visual information in the pre-training stage in layoutlmv2 by taking advantage of the transformer architecture to learn the cross-modality interaction between visual and textual information. in addition, inspired by the 1-d relative position representations @xcite @xcite , we propose the spatial-aware self-attention mechanism for layoutlmv2, which involves a 2-d relative position representation for token pairs. different from the absolute 2-d position embeddings that lay-outlm uses to model the page layout, the relative position embeddings explicitly provide a broader view for the contextual spatial modeling. for the pre-training strategies, we use two new training objectives for layoutlmv2 in addition to the masked visual-language modeling. the first is the proposed text-image alignment strategy, which aligns the text lines and the corresponding image regions. the second is the text-image matching strategy popular in previous vision-language pre-training models @xcite @xcite @xcite , where the model learns whether the document image and textual content are correlated. we select six publicly available benchmark datasets as the downstream tasks to evaluate the performance of the pre-trained layoutlmv2 model, which are the funsd dataset @xcite for form understanding, the cord dataset @xcite and the sroie dataset @xcite for receipt understanding, the kleister-nda dataset @xcite for long document understanding with a complex layout, the rvl-cdip dataset @xcite for document image classification, and the docvqa dataset @xcite for visual question answering on document images. experiment results show that the layoutlmv2 model significantly outperforms strong baselines, including the vanilla layoutlm, and achieves new state-of-the-art results in all of these tasks. the contributions of this paper are summarized as follows: ## approach in this section, we will introduce the model architecture and the multi-modal pre-training tasks of layoutlmv2, which is illustrated in figure 1 . ## related work in recent years, pre-training techniques have become popular in both nlp and cv areas, and have also been leveraged in the vrdu tasks. @xcite introduced a new language representation model called bert, which is designed to pre-train deep bidirectional representations from the unlabeled text by jointly conditioning on both left and right context in all layers. @xcite propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model. our multi-modal transformer architecture and the mvlm pre-training strategy extend transformer and mlm used in these work to leverage visual information. @xcite proposed vilbert for learning task-agnostic joint representations of image content and natural language by extending the popular bert architecture to a multi-modal two-stream model. @xcite proposed vl-bert that adopts the transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. different from these vision-language pre-training approaches, the visual part of layoutlmv2 directly uses the feature map instead of pooled roi features, and benefits from the new tia pre-training task. @xcite proposed layoutlm to jointly model interactions between text and layout information across scanned document images, benefiting a great number of real-world document image understanding tasks such as information extraction from scanned documents. this work is a natural extension of the vanilla layoutlm, which takes advantage of textual, layout, and visual information in a single multi-modal pre-training framework. ## conclusion in this paper, we present a multi-modal pre-training approach for visually-rich document understanding tasks, aka layoutlmv2. distinct from existing methods for vrdu, the layoutlmv2 model not only considers the text and layout information but also integrates the image information in the pretraining stage with a single multi-modal framework. meanwhile, the spatial-aware self-attention mechanism is integrated into the transformer architecture to capture the relative relationship among different bounding boxes. furthermore, new pre-training objectives are also leveraged to enforce the learning of cross-modal interaction among different modalities. experiment results on 6 different vrdu tasks have illustrated that the pre-trained layoutlmv2 model has substantially outperformed the sota baselines in the document intelligence area, which greatly benefits a number of real-world document understanding tasks. for future research, we will further explore the network architecture as well as the pre-training strategies for the layoutlm family. meanwhile, we will also investigate the language expansion to make the multi-lingual layoutlmv2 model available for different languages, especially the non-english areas around the world.