L ayout R eader: Pre-training of Text and Layout for Reading Order Detection (2021)

Abstract: Reading order detection is the cornerstone to understanding visually-rich documents (e.g., receipts and forms). Unfortunately, no existing work took advantage of advanced deep learning models because it is too laborious to annotate a large enough dataset. We observe that the reading order of WORD documents is embedded in their XML metadata; meanwhile, it is easy to convert WORD documents to PDFs or images. Therefore, in an automated manner, we construct ReadingBank, a benchmark dataset that contains reading order, text, and layout information for 500,000 document images covering a wide spectrum of document types. This first-ever large-scale dataset unleashes the power of deep neural networks for reading order detection. Specifically, our proposed LayoutReader captures the text and layout information for reading order prediction using the seq2seq model. It performs almost perfectly in reading order detection and significantly improves both open-source and commercial OCR engines in ordering text lines in their results in our experiments. The dataset and models are publicly available at https://aka.ms/layoutreader .

## introduction reading order detection, aiming to capture the word sequence which can be naturally comprehended by human readers, is a fundamental task for visually-rich document understanding. current off-the-shelf methods usually directly borrow the results from the optical character recognition (ocr) engines @xcite while most ocr engines arrange the recognized tokens or text lines in a top-to-bottom and left-to-right way @xcite . apparently, as shown in figure 1 , this heuristic method is not optimal for certain document types, such as multi-column templates, forms, invoices, and many others. an incorrect reading order will lead to unacceptable results for document ⇤ contributions during internship at msra. in the past decades, some conventional machine learning based or rule based methods @xcite @xcite @xcite have been proposed. however, these approaches are usually trained with only a small number of samples within a restricted domain or resort to unsupervised methods with empirical rules, because it is too laborious to annotate a large enough dataset. these models can barely show case studies of certain reading order scenarios and cannot be easily adapted for real-world reading order problems. recently, deep learning models @xcite have been applied to address the reading order issues for images from e-commerce platforms. although good performance has been achieved, it is time-consuming and labor-intensive to produce an in-house dataset, while they are still not publicly available to compare with other deep learning approaches. therefore, to facilitate the long-term research of reading order detection, it is inevitable to leverage automated approaches to create a real-world dataset in general domains, not only with high quality but also of larger magnitude than the existing datasets. to this end, we propose readingbank, a benchmark dataset with 500,000 real-world document images for reading order detection. distinct from the conventional human-labeled data, the proposed method obtains high-quality reading order annotations in a simple but effective way with automated metadata extraction. inspired by existing document layout annotations @xcite @xcite , there are a large number of microsoft word documents with a wide variety of templates that are available on the internet. typically, the word documents have two formats: the binary format (doc files) and the xml format (docx files). in this work, we exclusively use word documents with the xml format as the reading order information is embedded in the xml metadata. furthermore, we convert the word documents into the pdf format so that the 2d bounding box of each word can be easily extracted using any off-the-shelf pdf parser. finally, we apply a carefully designed coloring scheme to align the text in the xml metadata with the bounding boxes in pdfs. with the large-scale dataset, it is possible to take advantage of deep neural networks to solve reading order detection task. we further propose lay-outreader, a novel reading order detection model in which the seq2seq model is used by encoding the text and layout information and generating the index sequence in the reading order. ablation studies on the input modalities show that both text and layout information are essential to the final performance. the layoutreader with both modalities surpasses other comparative methods and performs almost perfectly in reading order detection. in addition, we also adapt the results of layoutreader to open-source and commercial ocr engines in ordering text lines. experiments show that the line ordering of both open-source and commercial ocr engines can be greatly improved. we believe that readingbank and layoutreader will empower more deep learning models in the reading order detection task and foster more customized neural architectures to push the new sota on this task. the contributions are summarized as follows: • we present readingbank, a benchmark dataset with 500,000 document images for reading order detection. to the best of our knowledge, this is the first large-scale benchmark for the research of reading order detection. ## problem formulation reading order refers to a well-organized readable word sequence. although it seems a fundamental requirement of nlp datasets, it is non-trivial to obtain proper reading orders from document images due to various formats, e.g., tables, multiple columns, and most ocr engines fail to provide the proper reading order. to solve this problem, we address the reading order detection task, aiming to extract the natural reading sequence from document images. specifically, given a visually-rich document image d, we acquire discrete token set {t 1 , t 2 , t 3 , ...} where each token t i consists of a word w i and the its bounding box coordinates (x i 0 , y i 0 , x i 1 , y i 1 ) (the lefttop corner and right-bottom corner). equipped with the textual and layout information of the tokens in the document image, we intend to sort the tokens into the reading order. ## readingbank readingbank includes two parts, the word sequence and its corresponding bounding box coordinates. we denote the word sequence as reading sequence that is extracted from docx files. the corresponding bounding boxes are extracted from the pdf files which are generated from docx files. we propose a coloring scheme to solve the word duplication when we match each word and its bounding box. in this section, we introduce the data pipeline in detail, including document collection, reading sequence extraction, and layout alignment with the coloring scheme. the current readingbank totally includes 500,000 document pages, where the training set includes 400,000 document pages and both the validation set and the test set include 50,000 document pages, respectively. ## layoutreader with the readingbank, we further propose lay-outreader to solve the reading order detection task. layoutreader is a sequence-to-sequence model using both textual and layout information, where we leverage the layout-aware language model lay-outlm @xcite as encoder and modify the generation step in the encoder-decoder structure to layoutlm is a layout-aware pre-trained language model for tasks in document pages with both text and bounding boxes from ocr. it first normalizes and rounds the bounding box coordinates into integers from 0 to 1000. then coordinates are embedded as trainable vectors like word embeddings. this new embedding layer is then added to bert @xcite . layoutlm is first initialized with bert and then further pre-trained with masked language model task and document classification. encoder: in the encoding stage, layoutreader packs the pair of source and target segments into a contiguous input sequence of layoutlm and carefully designs the self-attention mask to control the visibility between tokens. as shown in figure 3 , layoutreader allows the tokens in the source segment to attend to each other while preventing the tokens in the target segment from attending to the rightward context. if 1 means allowing and 0 means preventing, the detail of the mask m is as follows: decoder: in the decoding stage, since the source and target are reordered sequences, the prediction candidates can be constrained to the source segment. therefore, we ask the model to predict the indices in the source sequence. the probability is calculated as follows: ## experiments we introduce the comparative methods, implementation details, and evaluation metrics for the experiments. we design three experiments for lay-outreader on readingbank, including reading order detection, input order study, and adaption on ocr engines. in addition, we also show the realworld examples in the case study. ## related work reading order detection was first proposed in @xcite , where they used a propositional language of qualitative rectangle relations to detect reading order from document images. this is also considered as the first rule-based reading order detection system. with the development of machine learning methods, @xcite proposed a probabilistic classifier using the bayesian framework and reconstructing either single or multiple chains of layout components. meanwhile, @xcite applied an ilp learning algorithm to introduce the definitions of the two predicates and establish an ordering relationship. after that, @xcite investigated the problem of detecting the reading order relationship between components of a logical structure with domain specific knowledge. @xcite presented an unsupervised strategy for identifying the correct reading order of a document page's components based on abstract argumentation. the method is based on an empirical assumption about how humans behave when reading documents. more recently, deep learning models have become the mainstream solution for many machine learning problems. @xcite proposed an end-toend ocr text reorganizing model, where they use a graph neural network with an attention map to encode the text blocks with visual layout features, with an attention-based sequence decoder to reorder the ocr text into a proper sequence. ## conclusion in this paper, we present readingbank, a benchmark dataset for reading order detection that contains 500,000 document images. in addition, we also propose layoutreader, a novel reading order detection approach built upon the pre-trained layoutlm model. experiments show that the lay-outreader has significantly outperformed the leftto-right and top-to-bottom heuristics as well as several strong baselines. furthermore, the lay-outreader can be easily adapted to any ocr engines so that the reading order can be improved for downstream tasks. the readingbank dataset and layoutreader model will be publicly available to support more research on reading order detection. for future research, we will investigate how to generate a larger synthesized dataset from the read-ingbank, where noisy information and rotation can be applied to the clean images to make the model more robust. moreover, we will label the reading order information on a real-world dataset from scanned documents. considering the lay-outreader model as a pre-trained reading order detection model, we will also explore whether a few human labeled samples would be sufficient for the reading order detection in a specific domain.