D oc B ank: A Benchmark Dataset for Document Layout Analysis (2020)

Abstract: Document layout analysis usually relies on computer vision models to understand documents while ignoring textual information that is vital to capture. Meanwhile, high quality labeled datasets with both visual and textual information are still insufficient. In this paper, we present DocBank, a benchmark dataset that contains 500K document pages with fine-grained token-level annotations for document layout analysis. DocBank is constructed using a simple yet effective way with weak supervision from the LaTeX documents available on the arXiv.com. With DocBank, models from different modalities can be compared fairly and multi-modal approaches will be further investigated and boost the performance of document layout analysis. We build several strong baselines and manually split train/dev/test sets for evaluation. Experiment results show that models trained on DocBank accurately recognize the layout information for a variety of documents. The DocBank dataset is publicly available at https://github.com/doc-analysis/DocBank .

## introduction document layout analysis is an important task in many document understanding applications as it can transform semi-structured information into a structured representation, meanwhile extracting key information from the documents. it is a challenging problem due to the varying layouts and formats of the documents. existing techniques have been proposed based on conventional rule-based or machine learning methods, where most of them fail to generalize well because they rely on hand crafted features that may be not robust to layout variations. recently, the rapid development of deep learning in computer vision has significantly boosted the data-driven image-based approaches for document layout analysis. although these approaches have been widely adopted and made significant progress, they usually leverage visual features while neglecting textual features from the documents. therefore, it is inevitable to explore how to leverage the visual and textual information in a unified way for document layout analysis. nowadays, the state-of-the-art computer vision and nlp models are often built upon the pre-trained models @xcite @xcite @xcite @xcite followed by fine-tuning on specific downstream tasks, which achieves very promising results. however, pre-trained models not only require large-scale unlabeled data for self-supervised learning, but also need high quality labeled data for task-specific fine-tuning to achieve good performance. for document layout analysis tasks, there have been some image-based document layout datasets, while most of them are built for computer vision approaches and they are difficult to apply to nlp methods. in addition, image-based datasets mainly include the page images and the bounding boxes of large semantic structures, which are not fine-grained token-level annotations. moreover, it is also time-consuming and labor-intensive to produce humanlabeled and fine-grained token-level text block arrangement. therefore, it is vital to leverage weak supervision to obtain fine-grained labeled documents with minimum efforts, meanwhile making the data be easily applied to any nlp and computer vision approaches. to this end, we build the docbank dataset, a document-level benchmark that contains 500k document pages with fine-grained token-level annotations for layout analysis. distinct from the conventional human-labeled datasets, our approach obtains high quality annotations in a simple yet effective way with weak supervision. inspired by existing document layout annotations @xcite @xcite , there are a great number of digital-born documents such as the pdfs of research papers that are compiled by l a t e x using their source code. the l a t e x system contains the explicit semantic structure information using mark-up tags as the building blocks, such as abstract, author, caption, equation, figure, footer, list, paragraph, reference, section, table and title. to distinguish individual semantic structures, we manipulate the source code to specify different colors to the text of different semantic units. in this way, different text zones can be clearly segmented and identified as separate logical roles, which is shown in figure 1 . the advantage of docbank is that, it can be used in any sequence labeling models from the nlp perspective. meanwhile, docbank can also be easily converted into image-based annotations to support object detection models in computer vision. in this way, models from different modalities can be compared fairly using docbank, and multi-modal approaches will be further investigated and boost the performance of document layout analysis. to verify the effectiveness of docbank, we conduct experiments using four baseline models: 1) bert @xcite , a pre-trained model using only textual information based on the transformer architecture. 2) roberta @xcite , a robustly optimized method for pre-training the transformer architecture. 3) layoutlm @xcite , a multi-modal architecture that integrates both the text information and layout information. 4) faster r-cnn @xcite , a high performance object detection networks depending on region proposal algorithms to hypothesize object locations. the experiment results show that the layoutlm model significantly outperforms the bert and roberta models and the object detection model on docbank for document layout analysis. we hope docbank will empower more document layout analysis models, meanwhile promoting more customized network structures to make substantial advances in this area. the contributions of this paper are summarized as follows: the document layout analysis task is to extract the pre-defined semantic units in visually rich documents. specifically, given a document d composed of discrete token set @xmath0, t 1 , ..., t n }, each token t @xmath1, (x 0 , y 0 , x 1 , y 1 )) consists of word w and its bounding box (x 0 , y 0 , x 1 , y 1 ). and @xmath2, c 1 , .., c m } defines the semantic categories that the tokens are classified into. we intend to find a function f : (c, d) → s, where s is the prediction set: we build docbank with token-level annotations that supports both nlp and computer vision models. as shown in figure 2 , the construction of docbank has three steps: document acquisition, semantic structures detection, token annotation. meanwhile, docbank can be converted to the format that is used by computer vision models in a few steps. the current docbank dataset totally includes 500k document pages, where the training set includes 400k document pages and both the validation set and the test set include 50k document pages. ## method as the dataset was fully annotated at token-level, we consider the document layout analysis task as a text-based sequence labeling task. under this setting, we evaluate three representative pre-trained language models on our dataset including bert, roberta and layoutlm to validate the effectiveness of docbank. to verify the performance of the models from different modalities on docbank, we train the faster r-cnn model on the object detection format of docbank and unify its output with the sequence labeling models to evaluate. ## case study we visualize the outputs of pre-trained bert and pre-trained layoutlm on some samples of the test set in figure 3 and figure 4 . generally, it is observed that the sequence labeling method performs well on the docbank dataset, where different semantic units can be identified. for the pre-trained bert model, we can see some tokens are detected incorrectly, which illustrates that only using text information is still not sufficient for document layout analysis tasks, and visual information should be considered as well. compared with the pre-trained bert model, the pre-trained layoutlm model integrates both the text and layout information. therefore, it produces much better performance on the benchmark dataset. this is because the 2d position embeddings can model spatial distance and boundary of semantic structures in a unified framework, which leads to the better detection accuracy. (a) original document page (b) groundtruth (c) pre-trained bert (d) pre-trained layoutlm figure 4: example output of pre-trained layoutlm and pre-trained bert on the test set 7 related work the research of document layout analysis can be divided into three categories: rule-based approaches, conventional machine learning approaches, and deep learning approaches. ## conclusion to empower the document layout analysis research, we present docbank with 500k high-quality document pages that are built in an automatic way with weak supervision, which enables document layout analysis models using both textual and visual information. to verify the effectiveness of docbank, we conduct an empirical study with four baseline models, which are bert, roberta, layoutlm and faster r-cnn. experiment results show that the methods integrating text and layout information is a promising research direction with the help of docbank. we expect that docbank will further release the power of other deep learning models in document layout analysis tasks.