D oc H ie N et: A Large and Diverse Dataset for Document Hierarchy Parsing (2024)

Abstract: Parsing documents from pixels, such as pictures and scanned PDFs, into hierarchical structures is extensively demanded in the daily routines of data storage, retrieval and understanding. However, previously the research on this topic has been largely hindered since most existing datasets are small-scale, or contain documents of only a single type, which are characterized by a lack of document diversity. Moreover, there is a significant discrepancy in the annotation standards across datasets. In this paper, we introduce a large and diverse document hierarchy parsing (DHP) dataset to compensate for the data scarcity and inconsistency problem. We aim to set a new standard as a more practical, long-standing benchmark. Meanwhile, we present a new DHP framework designed to grasp both fine-grained text content and coarse-grained pattern at layout element level, enhancing the capacity of pre-trained text-layout models in handling the multi-page and multi-level challenges in DHP. Through exhaustive experiments, we validate the effectiveness of our proposed dataset and method.

## introduction nowadays, an overwhelming amount of information is generated daily and stored in documents as pixels, such as pictures and scanned pdfs, rather than in hierarchically structured formats. it introduces a significant challenge in practice, as structured formats are essential for efficient database storage and standardized data handling @xcite , as well as downstream tasks, such as information retrieval and natural language processing @xcite . particularly, it has been studied that documents with structural metadata further enhance the capabilities of large language models (llms), which has been outstanding across various domains, in processing lengthy documents and knowledge-intensive tasks @xcite . document hierarchy parsing (dhp) aims at reconstructing the hierarchical relationships among document layout elements (e.g., titles, paragraphs, figures), as shown in fig. 1 and thus organizing the document in a machine-understandable, hierarchically structured format. for documents as pixels, the layout elements can be extracted by off-the-shelf document layout analysis systems @xcite , and the dhp model focuses on predicting the hierarchical relationship among them. issues on previous datasets have hindered the progress of research and application. first, the datasets struggle to reflect the complexity of realworld documents. the arxivdocs @xcite and e-periodica @xcite are considered small-scale, containing only hundreds of single pages. regarding hrdoc and comp-hrdoc @xcite , although they are large-scale and exhibit various lengths, they contain only monotonous scientific articles, which share similar layout designs and hierarchical structures, such as examples in the 3rd row of fig. 1 . second, the annotation standards are inconsistent. for instance, the granularity of layout element annotations varies among datasets, including those based on text line level and layout block level. moreover, their definitions of hierarchical relations also differ with the varying definitions of layout elements. regarding the models, dhp presents two primary challenges: the handling of extended, multipage inputs and the comprehension of both textual content and the high-level layout relationships. previous works employ heuristic rules @xcite and lstm networks @xcite for their efficiency with lengthy inputs. @xcite utilize a pre-trained language model (plm) as the encoder to enhance the model performance. but this model extracts the text features of each layout element independently, thus overlooking the fine-grained contexts of layout elements. as a result of the issues with the dataset and model design, existing dhp methods struggle to be applicable in the real-world scenarios. in order to promote the development of dhp in more complex and realistic scenarios, we proposed dochienet, a large-scale, multi-page, multi-domain, multi-layout and bi-lingual dataset for dhp. dochienet contains 1673 multi-page documents from different scenarios including public sector, research, industry, etc. the multi-page documents, up to 50 pages, are characterized by large heterogeneity in their presentation and thus complex document structures (fig. 1 ), which are close to real-world conditions. the data collection of dochienet inherently encourages the development of models capable of addressing dhp on highly diverse documents. statistics of the datasets are summarized in tab. 1. with dochienet available, we propose a transformer-based framework, dhformer, which effectively overcomes the multi-page and multilevel challenges in dhp. it adopts a sparse textlayout encoder, derived from the powerful layoutaware language models (lms) @xcite to represent the layout elements with enriched fine-grained contexts. subsequently, a layout element-level reasoning decoder is exploited to capture collective information from multiple pages at the global range. besides, dhformer leverages the page embeddings and inner-layout position embeddings in order to better depict the cross-page and multi-level patterns. experiments show that the proposed method is highly competitive and outperforms previous methods by a large margin. our main contributions can be summarized as follows: ## problem definition in this paper, we consider the dhp as recognizing the hierarchical structure among layout elements. specifically, the input is given as a multi-page document along with m extracted layout elements @xmath0, e 2 , ..., e m } in traversal order, which can be obtained by the off-the-shelf optical character recognition (ocr) and document layout analysis system @xcite . the output is the hierarchical structure of the elements (e, r), where r is the relation set which captures relationships between layout elements. relation r j is defined as a tuple (e parent , e child ) which represents a hierarchical relation between elements. the definitions of the layout elements and their relationships vary among datasets. fig. 2 depicts a document image, with annotations visualized according to labeling systems of different datasets. e-periodica (see fig. 2 (b) ), defines layout elements as multi-granular content blocks with hierarchical relations which exist between elements of different granularities, and sequential relations which indicate reading order. this setup imposes stringent requisites on the layout analysis module for multi-granular elements, and it also results in semantically incomplete elements by annotating single pages separately. in hrdoc, annotations are based on text lines, simplifying issues of multigranularity by requiring the model to additionally identify text lines belonging to the same layout block (see green lines of 'connect' relationship in fig. 2 (c )). this approach neglects the advanced document layout analysis models. besides, the prevalence of the 'connect' relationship far exceeds other relations, making line-level evaluation a poor reflection of prediction quality due to the simplicity of the 'connect' pattern compared to the more complex hierarchical relationship. integrating the merits of different definitions and referencing prevailing works in the document layout analysis, we design the labeling system of dochienet to annotate only fine-grained layout blocks and capture both hierarchical and sequential relationships, as illustrated in fig. 2 (d) . ## dataset the dochienet contains a total of 1673 documents, of which 1110 are in english and 563 are in chinese. it covers a wide range of domains including legal, financial, educational, technical, and scientific documents. furthermore, as illustrated in fig. 1 , the documents are of diversified layout. ## method the proposed dhformer framework, as illustrated in fig. 4 , leveraging both fine-grained and holistic information, and making full use of pre-trained layout-aware lms, effectively tackles the multipage and multi-level challenges in dhp. firstly, the entire document, including tokens and their 2d positions, is fed into a sparse text-layout encoder e sp to create a fine-grained contextualized representation for each token. then, through pooling, the information is input into a layout element-level decoder d. the decoder captures collective information from higher-level and global contexts to obtain representations of layout elements. we specially equip the text-layout model with additional page embeddings and inner-layout position embeddings to enhance the capacity of modeling cross-page and multi-level relations. finally, the contextualized layout features are fed into the relation prediction head to get the final output. ## experiment 6.1 implementation details we employ pre-trained geolayoutlm @xcite as the basic text-layout encoder and a 2-layer ssa with a window size of 48 as the decoder. the adamw optimizer @xcite is employed for training with a base learning rate of 4e-5. the training epoch is set to 100 as the default, where the learning rates progressively decrease to 1e-6. during training, we set the max tokens of the text-layout encoder as 512 with the max number of chunks, as 32 (128 for testing). all the experiments of dhformer are performed on the platform with 2 nvidia tesla v100 gpus. ## conclusion in this paper, we present dochienet, a dhp dataset featuring large-scale, multi-page, multi-domain, multi-layout and bi-lingual documents. we carry out detailed analyses of data statistics, annotation paradigms and evaluation using various baselines. our findings demonstrate the challenging nature of the dochienet and the advantage of its annotations format. furthermore, we introduce an effective framework, dhformer, which consistently improves the model performance, particularly on the complex dochienet dataset. we hope this work could not only advance the understanding of dhp task but also set a foundation for future exploration.