Document Information Extraction via Global Tagging (2023)

Abstract: “Document Information Extraction (DIE) is a crucial task for extracting key information fromvisually-rich documents. The typical pipeline approach for this task involves Optical Charac-ter Recognition (OCR), serializer, Semantic Entity Recognition (SER), and Relation Extraction(RE) modules. However, this pipeline presents significant challenges in real-world scenariosdue to issues such as unnatural text order and error propagation between different modules. Toaddress these challenges, we propose a novel tagging-based method – Global TaggeR (GTR),which converts the original sequence labeling task into a token relation classification task. Thisapproach globally links discontinuous semantic entities in complex layouts, and jointly extractsentities and relations from documents. In addition, we design a joint training loss and a jointdecoding strategy for SER and RE tasks based on GTR. Our experiments on multiple datasetsdemonstrate that GTR not only mitigates the issue of text in the wrong order but also improvesRE performance. Introduction”

## introduction document information extraction (die), which is to extract key information from document with complex layouts, has become increasingly important in recent years @xcite . it not only enables us to efficiently compress document data, but also facilitates the retrieval of important information from documents. a typical pipeline approach for the die task is depicted in figure 1 (a) @xcite . first, the document with complex layout is converted into text blocks using optical character recognition (ocr) tools. next, the serializer module organizes these text blocks into a more appropriate order. finally, the well-ordered text blocks are input sequentially into the semantic entity recognition (ser) and relation extraction (re) modules to extract key-value pairs. however, the pipeline approach in figure 1 (a) presents significant challenges in real-world scenarios. (1) mainstream models for the die task, such as layoutlm @xcite , layoutlmv2 @xcite and layoutxlm @xcite , usually use sequence labeling in the beginning-inside-outside (bio) tagging schema, which assume that tokens belonging to the same semantic entity are grouped together after serialization. if the serializer module fails to order the text blocks correctly, the final performance can be severely impacted. a potential solution is to train a strong and robust serializer module, but this is difficult due to the labor-intensive labeling process under rich and diverse styles of documents; (2) in addition to the issue of text order, this pipeline also suffers from error propagation when using a ser module and a re module. in research settings, the results of the ser and re tasks are generally tested separately, with the ground truth of the ser results being used as default auxiliary information for the re task. however, in real-world scenarios, the ser module in the pipeline cannot provide 100% accurate results, which ineluctably leads to error propagation on re performance. researchers have explored alternative methods for modeling ocr results directly without serializer module to tackle the issue of text in the wrong order. some have utilized graph convolution networks to model the relationships between tokens @xcite @xcite . others have converted the die task into a parsing problem, modeling tree structure for the document @xcite . besides, generative encoder-decoder frameworks are applied to avoid the weakness of the bio tagging schema essentially @xcite . while these methods can mitigate the problem of text in the wrong order, they still face challenges. for example, graph-based methods require a more delicate model design, and generative models are usually difficult to train and require a great amount of document data for pre-training. to address abovementioned two problems, we propose a simple yet effective method named global tagger (gtr). our approach is inspired by @xcite , which converts the original sequence labeling task into a token relation classification task. for the ser task, we tag all token pairs and design a decoding strategy based on disjoint sets to decode the semantic entities. and we find gtr naturally resistant to wrong text order to a certain extent. for example, there is a document fragment "登记表 姓名李雷性别男" and we tag the token pair {姓, 名} so that we know "姓名" is a semantic entity. even if we shuffle this fragment to "登记表姓李性男名雷别", we still can know "姓名" is a semantic entity using the same tag {姓, 名}. in other words, gtr enables us to recognize discontinuous semantic entities, regardless of text in the wrong order. additionally, for the re task, we combine re and ser tags for joint training and extend the decoding strategy for joint decoding. the pipeline of this study is depicted in figure 1(b) . we remove the serializer module from the original pipeline to make it easier and propose an end-to-end extraction framework for jointly training the ser and re tasks to prevent error propagation problem. the contributions of this work are summarized as follows: ## approach in this section, we introduce our gtr approach in four parts. first, we propose the global tagging schema of the die task. next, a token pair scoring layer added to baseline model is proposed. then, we design a corresponding decoding strategy to decode entities and relations from the predicted tagging matrix. finally, we introduce our training loss for jointly training ser and re tasks. ## conclusion in this paper, we propose an end-to-end approach named global tagger to solve the document information extraction task. experiments on the funsd and xfun datasets demonstrate its efficacy in effectively mitigating the gap between token order in ocr input and golden input. furthermore, our experimental results indicate that joint training and decoding of semantic entity recognition and relation extraction tasks in this end-to-end extraction framework can alleviate the negative impact of error propagation and improve the performance of the relation extraction results.