Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction (2023)

Abstract: Recent advances in multimodal pre-trained models have significantly improved information extraction from visually-rich documents (VrDs), in which named entity recognition (NER) is treated as a sequence-labeling task of predicting the BIO entity tags for tokens, following the typical setting of NLP. However, BIO-tagging scheme relies on the correct order of model inputs, which is not guaranteed in real-world NER on scanned VrDs where text are recognized and arranged by OCR systems. Such reading order issue hinders the accurate marking of entities by BIO-tagging scheme, making it impossible for sequence-labeling methods to predict correct named entities. To address the reading order issue, we introduce Token Path Prediction (TPP), a simple prediction head to predict entity mentions as token sequences within documents. Alternative to token classification, TPP models the document layout as a complete directed graph of tokens, and predicts token paths within the graph as entities. For better evaluation of VrD-NER systems, we also propose two revised benchmark datasets of NER on scanned documents which can reflect real-world scenarios. Experiment results demonstrate the effectiveness of our method, and suggest its potential to be a universal solution to various information extraction tasks on documents.

## introduction visually-rich documents (vrds), including forms, receipts and contracts, are essential tools for gathering, carrying, and displaying information in the digital era. the ability to understand and extract information from vrds is critical for realworld applications. in particular, the recognition and comprehension of scanned vrds are necessary in various scenarios, including legal, business, and financial fields @xcite @xcite . recently, several transformer-based multimodal pretrained models @xcite @xcite @xcite have been proposed. known as document transformers, these models can encode text, layout, and image inputs into a unified feature representation, typically by marking each input text token with its xy-coordinate on the document layout using an additional 2d positional embedding. thus, document transformers can adapt to a wide range of vrd tasks, such as named entity recognition (ner) and entity linking (el) @xcite . however, in the practical application of information extraction (ie) from scanned vrds, the reading order issue is known as a pervasive problem that lead to suboptimal performance of current methods. this problem is particularly typical in vrd-ner, a task that aims to identify word sequences in a document as entities of predefined semantic types, such as headers, names and addresses. following the classic settings of nlp, current document transformers typically treat this task as a sequence-labeling problem, tagging each text token using the bio-tagging scheme @xcite and predicting the entity tag for tokens through a token classification head. these sequence-labeling-based methods assumes that each entity mention is a continuous and front-to-back word sequence within inputs, which is always valid in plain texts. however, for scanned vrds in real-world, where text and layout annotations are recognized and arranged by ocr systems, typically in a top-to-down and left-to-right order, the assumption may fail and the reading order issue arises, rendering the incorrect order of model inputs for document transformers, and making these sequence-labeling methods inapplicable. for instance, as depicted in figure 1 , the document contains two entity mentions, namely "name of account" and "# of stores supplied". however, due to their layout positions, the ocr system would recognize the contents as three segments and arrange them in the following order:(1)"# of stores"; (2)"name of account"; (3)"supplied". such disordered input leads to significant confusion in the biotagging scheme, making the models unable to assign a proper label for each word to mark the entity mentions clearly. unfortunately, the reading order issue is particularly severe in documents with complex layouts, such as tables, multi-column contents, and unaligned contents within the same row or column, which are quite common in scanned vrds. therefore, we believe that the sequencelabeling paradigm is not a practical approach to address ner on scanned vrds in real-world scenarios. to address the reading order issue, we introduce token path prediction (tpp), a simple yet strong prediction head for vrd-ie tasks. tpp is compatible with commonly used document transformers, figure 2 : the reading order issue of scanned documents in real-world scenarios. left: the entity "total 230,000" is disordered when reading from top to down. the entity "cash change" lies in multiple rows and is separated to two segments. right: the entity "total 30,000" is interrupted by "(qty 1.00)" when reading from left to right. all these situations result in disordered model inputs and affect the performance of sequencelabeling based vrd-ner methods. more real-world examples of disordered layouts are displayed in figure 7 in appendix. and can be adapted to various vrd-ie and vrd understanding (vrdu) tasks, such as ner, el, and reading order prediction (rop). specifically, when using tpp for vrd-ner, we model the token inputs as a complete directed graph of tokens, and each entity as a token path, which is a group of directed edges within the graph. we adopt a grid label for each entity type to represent the token paths as n * n binary values of whether two tokens are linked or not, where n is the number of text tokens. model learns to predict the grid labels by binary classification in training, and search for token paths from positively-predicted token pairs in inference. overall, tpp provides a viable solution for vrd-ner by presenting a suitable label form, modeling vrd-ner as predicting token paths within a graph, and proposes a straightforward method for prediction. this method does not require any prior reading order and is therefore unaffected by the reading order issue. for adaptation to other vrd tasks, tpp is applied to vrd-rop by predicting a global path of all tokens that denotes the predicted reading order. tpp is also capable of modeling the entity linking relations by marking linked token pairs within the grid label, making it suitable for direct adaptation to the vrd-el task. our work is related to discontinuous ner in nlp as the tasks share a similar form, yet current discontinuous ner methods cannot be directly applied to address the reading order issue since they also require a proper reading order of contents. for better evaluation of our proposed method, we also propose two revised datasets for vrd-ner. in current benchmarks of ner on scanned vrds, such as funsd @xcite and cord @xcite , the reading order is manuallycorrected, thus failing to reflect the reading order issue in real-world scenarios. to address this limitation, we reannotate the layouts and entity mentions of the above datasets to obtain two revised datasets, funsd-r and cord-r, which accurately reflect the real-world situations and make it possible to evaluate the vrd-ner methods in disordered scenarios. we conduct extensive experiments by integrating the tpp head with different document transformer backbones and report quantitative results on multiple vrd tasks. for vrd-ner, experiments on the funsd-r and cord-r datasets demonstrate the effectiveness of tpp, both as an independent vrd-ner model, and as a pre-processing mechanism to reorder inputs for sequence-labeling models. also, tpp achieves sota performance on benchmarks for vrd-el and vrd-rop, highlighting its potential as a universal solution for information extraction tasks on vrds. the main contribution of our work are listed as follows: 1. we identify that sequence-labeling-based vrd-ner methods is unsuitable for real-world scenarios due to the reading order issue, which is not adequately reflected by current benchmarks. 2. we introduce token path prediction, a simple yet strong approach to address the reading order issue in information extraction on vrds. 3. our proposed method outperforms sota methods in various vrd tasks, including vrd-ner, vrd-el, and vrd-rop. we also propose two revised vrd-ner benchmarks reflecting realworld scenarios of ner on scanned vrds. reading-order-aware methods several studies have addressed the reading order issue on vrds in two directions: (1) task-specific models that directly predict the reading order, such as lay-outreader, which uses a sequence-to-sequence approach @xcite . however, this method is limited to the task at hand and cannot be directly applied to vrd-ie tasks. (2) pre-trained models that learn from supervised signals during pre-training to improve their awareness of reading order. for instance, ernie-layout includes a pre-training objective of reading order prediction @xcite ; xylayoutlm enhances the generalization of reading order pre-training by generating various proper reading orders using an augmented xy cut algorithm @xcite . however, these methods require massive amount of labeled data and computational resources during pre-training. comparing with the above methods, our work is applicable to multiple vrd-ie tasks of vrds, and can be integrated with various document transformers, without the need for additional supervised data or computational costs. discontinuous ner the form of discontinuous ner is similar to vrd-ner, as it involves identifying discontinuous token sequences from text as named entities. current methods of discontinuous ner can be broadly categorized into four groups: ## conclusion and future work in this paper, we point out the reading order issue in vrd-ner, which can lead to suboptimal performance of current methods in practical applications. to address this issue, we propose token path prediction, a simple and easy-to-implement method which models the task as predicting token paths from a complete directed graph of tokens. tpp can be applied to various vrd-ie tasks with a unified architecture. we conduct extensive experiments to verify the effectiveness of tpp on real-world vrd-ner, where it serves both as an independent vrd-ner model and as a pre-processing mechanism to reorder model inputs. also, tpp achieves sota performance on vrd-el and vrd-rop tasks. we also propose two revised vrd-ner benchmarks to reflect the real situations of ner on scanned vrds. in future, we plan to further improve our method and verify its effectiveness on more vrd-ie tasks. we hope that our work will inspire future research to identify and tackle the challenges posed by vrd-ie in practical scenarios.