L ayout M ask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding (2023)

Abstract: Visually-rich Document Understanding (VrDU) has attracted much research attention over the past years. Pre-trained models on a large number of document images with transformer-based backbones have led to significant performance gains in this field. The major challenge is how to fusion the different modalities (text, layout, and image) of the documents in a unified model with different pre-training tasks. This paper focuses on improving text-layout interactions and proposes a novel multi-modal pre-training model, LayoutMask. LayoutMask uses local 1D position, instead of global 1D position, as layout input and has two pre-training objectives: (1) Masked Language Modeling: predicting masked tokens with two novel masking strategies; (2) Masked Position Modeling: predicting masked 2D positions to improve layout representation learning. LayoutMask can enhance the interactions between text and layout modalities in a unified model and produce adaptive and robust multi-modal representations for downstream tasks. Experimental results show that our proposed method can achieve state-of-the-art results on a wide variety of VrDU problems, including form understanding, receipt understanding, and document image classification.

## introduction visually-rich document understanding (vrdu) is an important research area that aims to understand various types of documents (e.g., forms, receipts, and posters), and it has attracted much attention from both academia and industry. in recent years, pre-training techniques @xcite have been introduced into this area and self-supervised pre-training multi-modal models have demonstrated great successes in various vrdu tasks @xcite @xcite . blue arrow: when using global 1d position, the reading order is explicitly implied by the ascending numbers, so the word after "qty" is "price". red arrows: when using local 1d position, the successor of "qty" is not directly given and can have more possible choices, so their semantic relations and 2d positions will be considered during pre-training. however, existing document pre-training models suffer from reading order issues. following the idea of bert @xcite , these methods @xcite @xcite usually adopt ascending numbers (e.g., 0, 1, 2,.., 511) to represent the global reading order of tokens in the document. then, these numbers are encoded into 1d position embeddings to provide explicit reading order supervision during pre-training, which are called "global 1d position". while such global 1d positions are widely used in nlp models for textual data, it is not a good choice for document data. firstly, plain texts always have definite and linear reading orders, but the reading order of a document may not be unique or even linear, which cannot be simply encoded with monotonically increasing numbers. secondly, the global reading order of a document is usually obtained by ordering detected text segments from ocr tools with empirical rules, so it heavily relies on stable and consistent ocr results, affecting the generalization ability in real-world applications. moreover, the empirical rules to obtain reading orders (e.g., "topdown and left-right") may not be able to handle documents with complex layouts, thus providing inaccurate supervision. some previous studies have attempted to solve the above reading order issues. layoutreader @xcite proposes a sequence-tosequence framework for reading order detection with supervised reading order annotations. xylay-outlm @xcite utilizes an augmented xy cut algorithm to generate different proper reading orders during pre-training to increase generalization ability. ernie-layout @xcite rearranges the order of input tokens in serialization modules and adopts a reading order prediction task in pre-training. while these studies propose databased or rule-based solutions to provide explicit reading order supervision, we believe that the selfsupervised pre-training process on a large number of documents without using extra supervision is sufficient to help the model to learn reading order knowledge, and such knowledge can be implicitly encoded into the pre-trained model with better adaptiveness and robustness to various document layouts. we proposed a novel multi-modal pre-training model, layoutmask, to achieve this goal. lay-outmask only uses text and layout information as model input and aims to enhance text-layout interactions and layout representation learning during pre-training. it differs from previous studies in three aspects: choice of 1d position, masking strategy, and pre-training objective. instead of global 1d position, layoutmask proposes to use the in-segment token orders as 1d position, which is referred to as "local 1d position" (see illustration in figure 1 ). as local 1d position does not provide cross-segment orders, layout-mask is supposed to infer global reading order by jointly using 1d position, 2d position, and semantic information, thus bringing in-depth text-layout interactions. to further promote such interactions, we equip the commonly used pre-training objective, masked language modeling (mlm), with two novel masking strategies, whole word masking and layout-aware masking, and design an auxiliary pre-training objective, masked position modeling, to predict masked 2d positions during pre-training. with the above designs, we increase the difficulty of pre-training objectives and force the model to focus more on layout information to obtain reading order clues in various document layouts in self-supervised learning, thus producing more adaptive and robust text-layout representations for document understanding tasks. experimental results show that our proposed method can bring significant improvements to vrdu tasks and achieve sota performance with only text and layout modalities, indicating that previous studies have not fully explored the potential power of layout information and text-layout interactions. the contributions of this paper are summarized as follows: ## related work the early studies in vrdu area usually use unimodal models or multi-modal models with shallow fusion @xcite @xcite . in recent years, pretraining techniques in nlp @xcite @xcite and cv @xcite have become more and more popular, and they have been introduced into this area. inspired by bert @xcite , layoutlm @xcite first improved the masked language modeling task by using the 2d coordinates of each token as layout embeddings, which can jointly model interactions be-tween text and layout information and benefits document understanding tasks. following this idea, layoutlmv2 @xcite propose to concatenate image patches with textual tokens to enhance text-image interactions, and layoutlmv3 @xcite proposed to learn cross-modal alignment with unified text and image masking. while the above methods focus on text-image interactions, some other studies have realized the importance of layout information. structurallm @xcite utilizes segment-level layout features to provide word-segment relations. doc-former @xcite ) combines text, vision, and spatial features with a novel multimodal self-attention layer and shares learned spatial embeddings across modalities. lilt @xcite proposes a language-independent layout transformer where the text and layout information are separately embedded. ernie-layout @xcite adopts a reading order prediction task in pre-training and rearranges the token sequence with the layout knowledge. ## methodology layoutmask is a multi-modal transformer that can encode text and layout information of documents and produce multi-modal representations. the pipeline of layoutmask can be seen in figure 2 . layoutmask uses the transformer model with a spatial-aware self-attention mechanism proposed in layoutlmv2 @xcite as the backbone and follows its preprocessing settings for text and layout embeddings. in section 3.1, we will discuss the different choices of layout information in layoutmask. in section 3.2, we will introduce the pre-training tasks and masking strategies used in layoutmask. ## conclusion in this paper, we propose layoutmask, a novel multi-modal pre-training model, to solve the reading order issues in vrdu tasks. layoutmask adopts local 1d position as layout input and can generate adaptive and robust multi-modal representations. in layoutmask, we equip the mlm task with two masking strategies and design a novel pretraining objective, masked position modeling, to enhance the text-layout interactions and layout representation learning. with only using text and layout modalities, our method can achieve excellent results and significantly outperforms many sota methods in vrdu tasks.