G en KIE : Robust Generative Multimodal Document Key Information Extraction (2023)

Abstract: Key information extraction (KIE) from scanned documents has gained increasing attention because of its applications in various domains. Although promising results have been achieved by some recent KIE approaches, they are usually built based on discriminative models, which lack the ability to handle optical character recognition (OCR) errors and require laborious token-level labeling. In this paper, we propose a novel generative end-to-end model, named GenKIE, to address the KIE task. GenKIE is a sequence-to-sequence multimodal generative model that utilizes multimodal encoders to embed visual, layout and textual features and a decoder to generate the desired output. Well-designed prompts are leveraged to incorporate the label semantics as the weakly supervised signals and entice the generation of the key information. One notable advantage of the generative model is that it enables automatic correction of OCR errors. Besides, token-level granular annotation is not required. Extensive experiments on multiple public real-world datasets show that GenKIE effectively generalizes over different types of documents and achieves state-of-the-art results. Our experiments also validate the model’s robustness against OCR errors, making GenKIE highly applicable in real-world scenarios.

## introduction the key information extraction (kie) task aims to extract structured entity information (e.g. keyvalue pairs) from scanned documents such as receipts @xcite , forms @xcite , financial reports @xcite , etc. this task is critical to many document understanding applications, such as information retrieval and text mining @xcite , where kie frees the 53,55,57 & 59, jalan sagu … company book ta .k … 8hd address no.53,55,57 & 59, jaian sacu … prompt company is? address is? business from manually processing a great number of documents and saves a significant amount of time and labour resource @xcite . the kie task is often tackled by a pipeline of approaches @xcite , including optical character recognition (ocr) and named entity recognition (ner). the ocr technique, exemplified by tesseract foot_0 , is employed to discern textual and layout attributes from the input images, namely the scanned documents. the ner model @xcite is used to discriminatively extract salient details from the derived texts, such as pinpointing specific entities from the text and layout features based on the annotated beginning-insideoutside (bio) tags via a sequence classification strategy. a plethora of methodologies adhering to this framework have emerged, leveraging a synergy of multimodal features-textual, visual, and layout data-from the document image. notably, contemporary kie models like structext @xcite , bros @xcite , and layoutlmv2 @xcite exhibit commendable results through the use of multimodal pretrained frameworks. however, one limitation of these models is that they highly rely on the ocr model to extract texts from scanned documents and inevitably suffer from ocr errors. these ocr errors in texts will eventually render wrong entity information. as shown in figure 1 , although the classification-based model tags the entities correctly, the result is still wrong due to the ocr errors. another limitation is that the semantic ambiguity in the document is hard to be captured by the existing models. for example, as shown in figure 1 , two image patches of the same text (taman dayan) in the receipt have different entity types. it is difficult for some existing approaches to improve performance by using only textual information. other signals such as layout and visual information play a critical role in identifying the correct entity type @xcite . therefore effective incorporation of multimodal features is indispensable to improve the model performance for the kie task. to cope with the mentioned problems, we propose genkie, a robust multimodal generative model for document kie. genkie utilizes the encoder-decoder transformer @xcite as the backbone and formulates document kie as a sequence-to-sequence generation task. the encoder effectively incorporates multimodal features to handle semantic ambiguity and the decoder generates the desired output autoregressively following the carefully designed template (i.e. prompt) and auto-correcting ocr errors. an example is shown in figure 1 to demonstrate the prompting technique, the label company in the prompt guides the model to generate words that correspond to company names. other labels such as address serve as additional label semantic signals and provide shared context about the task. thanks to the generation capability, genkie does not need the laborious granular token-level labelling that is usually required by discriminative models. extensive experiments demonstrate that our proposed genkie model has not only achieved performance levels comparable to state-of-the-art (sota) models across three public kie datasets but also exhibits enhanced robustness against ocr errors. our contributions are summarized as follows: ## task formulation in this work, we address the kie task which supports many downstream applications such as entity labeling @xcite and entity extraction @xcite . in particular, given a collection of scanned document images i d , the goal of kie is to extract a set of key entity fields (i.e. key-value pairs for each image @xmath0, where t i is the predefined entity type, v i is the entity value, and n i is the number of entity fields in image i. the entity types of each image example can be given or not, depending on the downstream applications. for example, the output of the entity extraction task is the entity values of predefined entity types, while the output of the entity labelling task is the extracted entities and their entity types. ## methodology the overall architecture of genkie is shown in figure 2. unlike the encoder-based models (e.g. lay-outlmv2 @xcite ), genkie is a generative model that uses a multimodal encoder-decoder model, i.e. ofa @xcite , as the backbone. the encoder of genkie embeds multimodal features (e.g. text, layout and images) from the input and the decoder generates the textual output by following the prompts. entity information is parsed from the decoder output. in the following sections, we illustrate the process of multimodal feature embedding, and then go over the techniques of prompting and inferring. ## conclusion we propose genkie, a novel prompt-based generative model to address the kie task. genkie is effective in generating the key information from the scanned document images by following carefully designed prompts. the validated strong robustness against the ocr errors makes the model applicable in real-world scenarios. extensive experiments over two kie tasks (i.e. entity labelling and entity extraction) on three public cross-domain datasets demonstrate the model's competitive performance compared with sota baselines. our genkie incorporates multimodal features, which enables the integration with other vision language models, offering possibilities for future exploration and experimentation.