Cost-effective End-to-end Information Extraction for Semi-structured Document Images (2021)

Abstract: A real-world information extraction (IE) system for semi-structured document images often involves a long pipeline of multiple modules, whose complexity dramatically increases its development and maintenance cost. One can instead consider an end-to-end model that directly maps the input to the target output and simplify the entire process. However, such generation approach is known to lead to unstable performance if not designed carefully. Here we present our recent effort on transitioning from our existing pipeline-based IE system to an end-to-end system focusing on practical challenges that are associated with replacing and deploying the system in real, large-scale production. By carefully formulating document IE as a sequence generation task, we show that a single end-to-end IE system can be built and still achieve competent performance.

## introduction information extraction (ie) for semi-structured documents is an important first step towards automated document processing. one way of building such ie system is to develop multiple separate modules specialized in each sub-task. for example, our currently-deployed ie system for name card and receipt images, pot @xcite ), foot_0 consists of three manually engineered modules and one data-driven module (fig. 1a ). this system first accepts text segments and their 2d coordinates (we dub it "2d text") from an ocr system and generates pseudo-1d-text using a serializer. the text is then iob-tagged and mapped to a raw, structured parse. finally, the raw parse is normalized by trimming and reformatting with regular expressions. a b ie system doc. images parses { store name: "wyvern store", store tel: "13-806-4852" } ..., { menu name: "polyjuice portion", count: "2", unit price: "1,000", price: "2,000" }, { menu name: "cheap wand", count: "1", ... ocr tag2parse serializer tagging model parse normalizer doc. images e2e serializer tagging model parses ocr gt tags inference routine training routine gt: ground truth loss func. parse annotator tag annotator loss func. ## model wyvern consists of the following three major modules: (1) the transformer encoder that accepts 2d text, (2) the decoder for sequence generation, and (3) the transition module that converts the generated sequence into parse tree. ## datasets and setups datasets due to confidential issues related to industrial documents, we use our four large-scale internal datasets: three strong-label (nj, rk, rj) and one weak-label (nj-w) datasets. the properties are summarized in tbl. 2. baseline we compare wyvern to pot that consists of four separated modules (fig. 1a , sec. 1). the technical details are described in sec. a.1. evaluation the parses consist of hierarchically grouped key-value pairs, where a key indicates a field type such as menu, and a value is either a corresponding text sequence or a sub parse forming a recursive structure (e.g., menu under item in fig. 2a ). we conduct extensive evaluation with three different metrics; f 1 , nted, and a/btest. f 1 is calculated by counting the number of exactly matched key-value pairs. since f 1 ignores partially correct prediction (even a single character difference is counted as wrong), we use another metric nted, a tree edit distance normalized by the number of nodes. nted considers both lexical and structural differences between parse trees (fig. 2b ). finally, we conduct a/b-test for full evaluation of the models by human. see sec. a.2 for the technical details of evaluation methods. ## conclusion here, we present wyvern that is trained via weak supervision. wyvern achieves competent performance compared to the strongly supervised model while its development and maintenance cost is significantly lower. we also provide cost analysis of developing ie systems for semistructured documents. currently, wyvern shows slower training and inference time compared to the tagging based approach like other autoregressive models. future work will be focused on optimizing the training protocol and the inference speed.