S tructural LM : Structural Pre-training for Form Understanding (2021)

Abstract: Large pre-trained language models achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, they almost exclusively focus on text-only representation, while neglecting cell-level layout information that is important for form image understanding. In this paper, we propose a new pre-training approach, StructuralLM, to jointly leverage cell and layout information from scanned documents. Specifically, we pre-train StructuralLM with two new designs to make the most of the interactions of cell and layout information: 1) each cell as a semantic unit; 2) classification of cell positions. The pre-trained StructuralLM achieves new state-of-the-art results in different types of downstream tasks, including form understanding (from 78.95 to 85.14), document visual question answering (from 72.59 to 83.94) and document image classification (from 94.43 to 96.08).

## introduction document understanding is an essential problem in nlp, which aims to read and analyze textual documents. in addition to plain text, many realworld applications require to understand scanned documents with rich text. as shown in figure 1 , such scanned documents contain various structured information, like tables, digital forms, receipts, and invoices. the information of a document image is usually presented in natural language, but the format can be organized in many ways from multicolumn layout to various tables/forms. inspired by the recent development of pretrained language models @xcite @xcite in various nlp tasks, recent studies on document image pretraining @xcite have pushed the limits of a variety of document image understanding tasks, which learn the interaction be-tween text and layout information across scanned document images. @xcite propose layoutlm, which is a pre-training method of text and layout for document image understanding tasks. it uses 2dposition embeddings to model the word-level layout information. however, it is not enough to model the word-level layout information, and the model should consider the cell as a semantic unit. it is important to know which words are from the same cell and to model the cell-level layout information. for example, as shown in figure 1 (a), which is from form understanding task @xcite , determining that the "lorillard" and the "entities" are from the same cell is critical for semantic entity labeling. the "loril-lard entities" should be predicted as answer entity, but layoutlm predicts "lorillard" and "entities" as two separate entities. the input to traditional natural language tasks is usually presented as plain text, and text-only models need to obtain the semantic representation of the input sentences and the semantic relationship between sentences. in contrast, document images like forms and tables are composed of cells that are recognized as bounding boxes by ocr. as shown in figure 1 , the words from the same cell generally express a meaning together and should be modeled as a semantic unit. this requires a text-layout model to capture not only the semantic representation of individual cells but also the spatial relationship between cells. in this paper, we propose structurallm to jointly exploit cell and layout information from scanned documents. different from previous text-based pretrained models @xcite and layoutlm @xcite from the same cell, and thus enables the model to derive representation for the cells. in addition, we keep classic 1d-position embeddings to preserve the positional relationship of the tokens within every cell. we propose a new pre-training objective called cell position classification, in addition to the masked visual-language model. specifically, we first divide an image into n areas of the same size, and then mask the 2d-positions of some cells. structurallm is asked to predict which area the masked cells are located in. in this way, struc-turallm is capable of learning the interactions between cells and layout. we conduct experiments on three benchmark datasets publicly available, all of which contain table or form images. empirical results show that our structurallm outperforms strong baselines and achieves new state-of-the-art results in the downstream tasks. in addition, struc-turallm does not rely on image features, and thus is readily applicable to real-world document understanding tasks. we summarize the major contributions in this paper as follows: ## structurallm we present structurallm, a self-supervised pretraining method designed to better model the interactions of cells and layout information in scanned document images. the overall framework of struc-turallm is shown in figure 2 . our approach is inspired by layoutlm @xcite , but different from it in three ways. first, we use cell-level 2d-position embeddings to model the layout information of cells rather than word-level 2d-position embeddings. we also introduce a novel training objective, the cell position classification, which tries to predict the position of the cells only depending on the position of surrounding cells and the semantic relationship between them. finally, structurallm retains the 1d-position embeddings to model the positional relationship between tokens from the same cell, and removes the image embeddings in layoutlm that is only used in the downstream tasks. ## conclusion in this paper, we propose structurallm, a novel structural pre-training approach on large unlabeled documents. it is built upon an extension of the transformer encoder, and jointly exploit cell and layout information from scanned documents. different from previous pre-trained models, structurallm uses cell-level 2d-position embeddings with tokens in the cell sharing the same 2dposition. this makes structurallm aware of which words are from the same cell, and thus enables the model to derive representation for the cells. we propose a new pre-training objective called cell position classification. in this way, structurallm is capable of learning the interactions between cells and layout. we conduct experiments on three benchmark datasets publicly available, and structurallm outperforms strong baselines and achieves new state-of-the-art results in the downstream tasks.