L i LT : A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding (2022)

Abstract: Structured document understanding has attracted considerable attention and made significant progress recently, owing to its crucial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective Language-independent Layout Transformer (LiLT) for structured document understanding. LiLT can be pre-trained on the structured documents of a single language and then directly fine-tuned on other languages with the corresponding off-the-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout structure. Code and model are publicly available at https://github.com/jpWang/LiLT .

## introduction structured document understanding (sdu) aims at reading and analyzing the textual and structured information contained in scanned/digital-born documents. with the acceleration of the digitization process, it has been regarded as a crucial part of intelligent document processing and required by many real-world applications in various industries such as finance, medical treatment and insurance. recently, inspired by the rapid development of pre-trained language models of plain texts @xcite @xcite , many researches on structured document pre-training @xcite @xcite have also pushed the limit of a variety of sdu tasks. however, almost all of them only focus on pre-training and fine-tuning on the documents in a single language, typically english. this is extremely limited for other languages, especially in the case of lacking pre-training structured document data. in this regard, we consider how to make the sdu tasks enjoy language-independent benefit from the pre-training of document layout structure. here, we give an observation as shown in figure 1 . when the layout structure remains unchanged, the substitution of language does not make obvious unnaturalness. it fully motivates us to decouple and reuse the layout invariance among different languages. based on this inspiration, in this paper, we propose a simple yet effective language-independent layout transformer (lilt) for structured document understanding. in our framework, the text and layout information are first decoupled and jointoptimized during pre-training, and then re-coupled for fine-tuning. to ensure that the two modalities have sufficient language-independent interaction, we further propose a novel bi-directional attention complementation mechanism (biacm) to enhance the cross-modality cooperation. moreover, we present the key point location (kpl) and crossmodal alignment identification (cai) tasks, which are combined with the widely-used masked visual-language modeling (mvlm) to serve as our pretraining objectives. during fine-tuning, the layout flow (lilt) can be separated and combined with the off-the-shelf pre-trained textual models (such as roberta @xcite , xlm-r @xcite , infoxlm @xcite , etc) to deal with the downstream tasks. in this way, our method decouples and learns the layout knowledge from the monolingual structured documents before generalizing it to the multilingual ones. to the best of our knowledge, the only preexisting multilingual sdu model is layoutxlm @xcite . it scraps multilingual pdf documents of 53 languages from a web crawler and introduces extra pre-processing steps to clean the collected data, filter the low-quality documents, and classify them into different languages. after this, it utilizes a heuristic distribution to sample 22 million multilingual documents, which are further combined with the 8 million sampled english ones from the iit-cdip @xcite dataset (11 million english documents), resulting 30 million for pre-training with the layoutlmv2 @xcite framework. however, this process is timeconsuming and laborious. on the contrary, lilt can be pre-trained with only iit-cdip and then adapted to other languages. in this respect, lilt is the first language-independent method for structured document understanding. experimental results on eight languages have shown that lilt can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which substantially benefits numerous real-world sdu applications. our main contributions can be summarized as follows: ## lilt figure 2 shows the overall illustration of our method. given an input document image, we first use off-the-shelf ocr engines to get text bounding boxes and contents. then, the text and layout information are separately embedded and fed into the corresponding transformer-based architecture to obtain enhanced features. bi-directional attention complementation mechanism (biacm) is introduced to accomplish the cross-modality interaction of text and layout clues. finally, the encoded text and layout features are concatenated and additional heads are added upon them, for the self-supervised pre-training or the downstream fine-tuning. ## related work during the past decade, deep learning methods became the mainstream for document understanding tasks @xcite @xcite . grid-based methods @xcite @xcite were proposed for 2d document representation where text pixels were encoded using character or word embeddings and classified into specific field types, using a convolutional neural network. gnn-based approaches @xcite @xcite adopted multi-modal features of text segments as nodes to model the document graph, and used graph neural networks to propagate information between neighboring nodes to attain a richer representation. in recent years, self-supervised pre-training has achieved great success. inspired by the development of the pre-trained language models in various nlp tasks, recent studies on structured document pre-training @xcite @xcite have pushed the limits. layoutlm @xcite ) modified the bert @xcite architecture by adding 2d spatial coordinate embeddings. in comparison, our lilt can be regarded as a more powerful and flexible solution for structured document understanding. layoutlmv2 @xcite improved over layoutlm by treating the visual fea- ## conclusion in this paper, we present lilt, a languageindependent layout transformer that can learn the layout knowledge from monolingual structured documents and then generalize it to deal with multilingual ones. our framework successfully first decouples the text and layout information in pre-training and then re-couples them for finetuning. experimental results on eight languages under three settings (language-specific, cross-lingual zero-shot transfer, and multi-task fine-tuning) have fully illustrated its effectiveness, which substantially bridges the language gap in real-world structured document understanding applications. the public availability of lilt is also expected to promote the development of document intelligence. for future research, we will continue to follow the pattern of transferring from "monolingual" to "multilingual" and further unlock the power of lilt. in addition, we will also explore the generalized rather than language-specific visual information contained in multilingual structured documents.