GMN : Generative Multi-modal Network for Practical Document Information Extraction (2022)

Abstract: Document Information Extraction (DIE) has attracted increasing attention due to its various advanced applications in the real world. Although recent literature has already achieved competitive results, these approaches usually fail when dealing with complex documents with noisy OCR results or mutative layouts. This paper proposes Generative Multi-modal Network (GMN) for real-world scenarios to address these problems, which is a robust multi-modal generation method without predefined label categories. With the carefully designed spatial encoder and modal-aware mask module, GMN can deal with complex documents that are hard to serialized into sequential order. Moreover, GMN tolerates errors in OCR results and requires no character-level annotation, which is vital because fine-grained annotation of numerous documents is laborious and even requires annotators with specialized domain knowledge. Extensive experiments show that GMN achieves new state-of-the-art performance on several public DIE datasets and surpasses other methods by a large margin, especially in realistic scenes.

## introduction document information extraction (die) aims to map each document to a structured form consistent with the target ontology (e.g., database schema), which has recently become an increasingly important task. recent research @xcite @xcite @xcite has achieved competitive results for information extraction in the idealized scenario with accurate ocr results, word-level annotations, and serialized document words in reading order. these methods regard die as a sequence labeling (sl) task. given ocr results of a document image, the traditional sequence labeling method first serializes * work is done during an internship at tencent youtu lab. total: 24,000 | cash: 24,000 duplicated value total 24,000 cash 24,000 o total -1 b-total o b-total total -2 total : 24,000 | total:24,000 sl gmn fig4 confused key subtotal: 15.90 | total excl. of gst: 15.90 subtotal : 15.90 | total:15.90 subtotal 15.90 total excl. of gst 15.00 o b-subtotal o o o o b-total sl gmn fig3 total -2 total -2 company : gucci g uc c l bicaster b-com company -1 i-com i-com i-com o company : guuc1 sl gmn fig2 nosiy ocr gucci gucc1 figure 1: examples in public die benchmarks with practical problems. the three rows from top to bottom in each sub-figure are 1) input images, 2) raw intermediate tags and final results generated by the sl method, and 3) results of our gmn method, respectively. the error parts are marked in red, while the correct parts are in green. best viewed in color. as shown in figure 1 , multiple challenging problems for practical document understanding still exist in realistic scenes. 1) document serialization requires pre-composition processing, which is difficult in real scenarios with ambiguous word orders. one entity may be incorrectly divided into multiple entities when the input sequences are sorted by coordinates. 2) ocr results are usually noisy because of inevitable recognition errors. 3) the volume of keys in practical scenarios is generally substantial and expanded frequently. existing sequence labeling methods could not identify undefined keys. 4) while facing duplicated values, collecting word-level annotations is necessary for sequence labeling methods. however, this is difficult in practical scenarios since they are costly and labor-intensive. to address the limitations mentioned above, we propose a robust information extraction method named generative multi-modal network (gmn) for practical document understanding. unlike sequence labeling methods that label each input word with a predefined category, we regard die as a translation task that translates source ocr results to a structured format (like key-value pairs in this paper). we use unilm @xcite as the basic model structure, which is a transformerbased pre-trained network that can handle both natural language understanding (nlu) and natural language generation (nlg) tasks simultaneously. conditioned on a sequence of source words, gmn generates one word at each time step to compose a series of key-value pairs in them. regarding the sequence serialization problem, a novel two-dimensional position embedding method is proposed while the original one-dimensional positional embedding in the transformer is removed because all information in document understanding can be acquired from 2d layouts. in this manner, gmn bypasses the serialization problem. furthermore, benefiting from the large-scale selfsupervised pre-training processed on a vast document collection, gmn can correct ocr errors commonly encountered in practical scenarios. moreover, using a weakly supervised training strategy that utilizes only key information sequences as supervision, gmn needs no word-level annotations that are indispensable in traditional sequence labeling methods like layoutlm @xcite and structurallm @xcite . experiments illustrate that the proposed gmn model outperforms several sota pre-trained models on benchmark datasets, including sroie and cord. the contributions of this paper are summarized as follows: ## related works traditional methods @xcite @xcite on die tasks rely heavily on predefined rules, templates, and handcrafted features, giving rise to difficulty in generalizing to unseen documents. with the development of deep learning technology, document information extraction methods have recently improved substantially in both performance and robustness. these deep learning-based methods can be classified into three categories: textual contentbased methods, multi-modal-based methods, and pre-trained transformer-based methods. textual content-based methods. palm et al. 2017); sage et al. (2019 adopt the idea from natural language processing and use recurrent neural networks (rnn) to extract entities of interest from documents. however, they discard the layout information during the text serialization, which is crucial for document understanding. multi-modal-based methods. some works @xcite take the layout information into consideration and try to reconstruct character or word segmentation of the document. katti et al. (2018) encode each document page as a two-dimensional grid of characters that represents text representation with a two-dimensional layout. yu et al. (2020); majumder et al. (2020); zhang et al. (2020); wang et al. (2021b) further integrate image embeddings for better feature extraction. yu et al. (2020); tang et al. ( 2021 ) represent documents by graphs, with nodes representing word segments and edges either connecting all the nodes or only spatially near neighbors. convolutional or recurrent mechanisms are then applied to the graph for predicting the field type of each node. however, due to the lack of large-scale pre-training, the robustness and accuracy of the model are relatively limited. pre-trained transformer-based methods. recently, pre-trained models @xcite show effective knowledge transferability with large-scale training data and various selfsupervised tasks. layoutlm @xcite ) first proposes a document-level pre-training framework that semantic and layout information are jointly learned. layoutlm v2 @xcite further improves the layoutlm model by integrating the image information in the pre-training stage. li et al. (2021a) propose the structurallm pre-training approach to exploit text block information. methods semantic image feature extraction date date:11/12/96 [dsep] attention: john j.mulderig c/o mike baker [dsep] company: philip morris management corp. [sep] source input md-bert (0,w,w) (x0,x1,w)1 ••• (0,w,w) (0,h,h) (y0,y1,h)1 ••• (0,h,h) img (img)1 ••• null [beg] facsimi ••• [sep] embeddings x y visual md-bert md-bert : md-bert the overall architecture of our generative multi-modal network, md-bert in encoder and decoder are with parameter sharing. ## methodology in this section, we first introduce the overall architecture of gmn, followed by illustrating multimodal feature extraction, generative pre-training model with multi-modal decoupling in detail, respectively.