ERNIE -Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding (2022)

Abstract: Recent years have witnessed the rise and success of pre-training techniques in visually-rich document understanding. However, most existing methods lack the systematic mining and utilization of layout-centered knowledge, leading to sub-optimal performances. In this paper, we propose ERNIE-Layout, a novel document pre-training solution with layout knowledge enhancement in the whole workflow, to learn better representations that combine the features from text, layout, and image. Specifically, we first rearrange input sequences in the serialization stage, and then present a correlative pre-training task, reading order prediction, to learn the proper reading order of documents. To improve the layout awareness of the model, we integrate a spatial-aware disentangled attention into the multi-modal transformer and a replaced regions prediction task into the pre-training phase. Experimental results show that ERNIE-Layout achieves superior performance on various downstream tasks, setting new state-of-the-art on key information extraction, document image classification, and document question answering datasets. The code and models are publicly available at PaddleNLP.

## introduction visually-rich document understanding (vrdu) is an important research field aiming to handle various types of scanned or digital-born business documents (e.g., forms, invoices), which has attracted great attention from the industry and academia due to its various applications. distinct from conventional natural language understanding (nlu) tasks that use only plain text, vrdu models have the opportunity to access the most primitive data features. herein, the diversity and complexity of document formats pose new challenges to the task, an ideal model needs to make full use of the textual, layout, and even visual information to fully understand visually-rich document like humans. the preliminary works for vrdu @xcite @xcite @xcite usually adopt uni-modal or shallow multi-modal fusion approaches, which are task-specific and require massive data annotations. recently, pre-training language models have swept the field, layoutlm @xcite , lay-outlmv2 @xcite , and some advanced document pre-training approaches @xcite @xcite have been proposed successively and achieved great successes in various vrdu tasks. unlike popular uni-modal or vision-language frameworks @xcite @xcite , the uniqueness of document understanding models lies in how to exploit the layout knowledge. however, existing document pre-training solutions typically fall into the trap of simply taking 2d coordinates as an extension of 1d positions to endow the model layout awareness. considering the characteristics of vrdu, we believe that the layout-centered knowledge should be systematically mined and utilized from two aspects: (1) on the one hand, layout implicitly reflects the proper reading order of documents, while previous methods are used to perform the serialization by multiplexing the results of optical character recognition (ocr), which roughly arrange tokens in the top-to-bottom and left-to-right manner @xcite . inevitably, it is inconsistent with human reading habits for documents with complex layouts (e.g., tables, forms, multi-column templates) and leads to sub-optimal performances for downstream tasks. (2) on the other hand, layout is actually the third modality besides language and vision, while current models are used to take lay-out as a special position feature, such as the layout embedding in input layer @xcite or the bias item in attention layer @xcite . the lack of cross-modal interaction between layout and text/image might restrict the model from learning the role of layout in semantic expression. to achieve these goals, we propose a systematic layout knowledge enhanced pre-training approach, ernie-layout foot_0 , to improve the performances of document understanding tasks. first of all, we employ an off-the-shelf layout-based document parser in the serialization stage to generate an appropriate reading order for each input document, so that the input sequences received by the model are more in line with human reading habits than using the rough raster-scanning order. then, each textual/visual token is equipped with its position embedding and layout embedding, and sent to the stacked multimodal transformer layers. to enhance cross-modal interaction, we present a spatial-aware disentangled attention mechanism, inspired by the disentangled attention of deberta @xcite , in which the attention weights between tokens are computed using disentangled matrices based on their hidden states and relative positions. in the end, layout not only acts as the 2d position attribute of input tokens, but also contributes a spatial perspective to the calculation of semantic similarity. with satisfactory serialization results, we propose the pre-training task, reading order prediction, to predict the next token for each position, which facilitates the consistency within the same arranged text segment and the discrimination between different segments. furthermore, when pre-training, we also adopt the classic masked visual-language modeling and text-image alignment tasks @xcite , and present a fine-grained multi-modal task, replaced regions prediction, to learn the correlation among language, vision and layout. we construct broad experiments on three representative vrdu downstream tasks with six publicly available datasets to evaluate the performance of the pre-trained model, i.e., the key information extraction task with the funsd @xcite , cord @xcite , sroie @xcite , kleister-nda @xcite datasets, the document question answering task with the docvqa @xcite dataset, and the document image classification task with the rvl-cdip @xcite dataset. the results show that ernie-layout significantly outperforms strong baselines on almost all tasks, proving the effectiveness of our two-part layout knowledge enhancement philosophy. the contributions are summarized as follows: to the best of our knowledge, ernie-layout is the first attempt to consider the proper reading order in document pre-training. based on the powerful modeling ability of transformer @xcite , layoutlm @xcite initially embeds the 2d coordinates as layout embeddings for each token and extends the famous masked language modeling pre-training task @xcite to masked visual-language modeling, which opens the prologue of layout-aware pre-trained models. afterwards, layoutlmv2 @xcite concatenates document image patches with textual tokens, and two pre-training tasks, textimage matching and text-image alignment, are proposed to realize the cross-modal interaction. struc-trallm @xcite ) leverages segment-level, instead of word-level, layout features to make the model aware of which words come from the same cell. docformer @xcite shares the learned spatial embeddings across modalities, making it easy for the model to correlate text to visual tokens and vice versa. tilt @xcite proposes an encoder-decoder model to generate results that are not explicitly included in the input sequence to solve the limitations of sequence t1 t2 [cls] [sep] … v1 v2 v3 v7 … v8 v9 0.02 … 0.04 0.87 0.01 transformer layers with spatial-aware disentangled attention mechanism visual features textual features t1 t2 [cls] t4 t3 t5 t6 t7 … t8 [sep] … 0 … 0 1 0 1 0 0.08 0.76 bce replaced region prediction labeling. however, these methods ignore the potential value of layout in-depth and directly rely on a raster-scanning serialization, which is contrary to human reading habits. to solve this problem, lay-outreader @xcite designs a sequenceto-sequence framework to generate an appropriate reading order for each document. unfortunately, it is carefully designed for reading order detection and cannot directly empower various document understanding tasks. besides, the above methods are used to regard layout as a subsidiary feature of text along with the idea of layoutlm, but the same text with different layouts may also express different semantics. therefore, we believe that layout should be regarded as the third modality independent of language and vision. knowledge-enhanced representation. following the bert @xcite architecture, many efforts are devoted to pre-trained language models for learning informative representations. there are some studies show that extra knowledge, such as facts in wikidata and wordnet, can further benefit the pre-trained models @xcite @xcite , but the embeddings of words in the text and entities in the knowledge graphs are not in the same vector space, so a cumbersome adaptation module is required @xcite . another research line is to excavate the potential human cognitive laws of the text itself: ernie @xcite creativity proposes entity-level mask in pre-training to incorporate the human knowledge into language models. similarly, spanbert @xcite modifies the making schema and training objectives to better represent and predict text spans. bert-wwm @xcite introduces a whole word masking strategy for chinese language models. outside the field of plain text, ernie-vil @xcite incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. inspired by the above work, we leverage the implicit knowledge related to layout, e.g., reading order, for the understanding of visually rich documents. ## methodology figure 1 shows an overview of the ernie-layout. given a document, ernie-layout rearranges the token sequence with the layout knowledge and extracts visual features from the visual encoder. the textual and layout embeddings are combined into textual features through a linear projection, and similar operations are executed for visual embeddings. next, these features are concatenated and fed into the stacked multi-modal transformer layers, which are equipped with the proposed spatialaware disentangled attention mechanism. for pretraining, ernie-layout adopts four pre-training tasks, including the new proposed reading order prediction, replaced region prediction tasks, and the traditional masked visual-language modeling, text-image alignment tasks. by using document-parser, the perplexity of the document with a complex layout is significantly reduced. ## conclusion in this paper, we propose ernie-layout, to integrate layout knowledge into document pre-training models from two aspects: serialization and attention. ernie-layout attempts to rearrange the recognized words of documents, which achieves considerable improvement on downstream tasks over the original raster-scanning order. besides, we also design a novel attention mechanism to help ernie-layout build better interaction between text/image and layout features. extensive experiments demonstrate the effectiveness of ernie-layout, and various analyses show the impact of different utilization of layout knowledge on vrdu tasks. an important preprocessing step for document understanding is serializing the extracted document tokens. the popular method for this serialization is performed directly on the output results of ocr in raster-scanning order and is sub-optimal though simple to implement. with the layout-parser and table-parser in the document-parser toolkit, the order of the tokens will be further rearranged according to the layout knowledge. during the parsing processing, the tables and figures are detected as spatial layouts, and the free texts are processed by paragraph analysis, combining heuristics and detection models to get the paragraph layout information and the upper-lower boundary relationship.