Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding (2024)

Abstract: Modeling and leveraging layout reading order in visually-rich documents (VrDs) is critical in document intelligence as it captures the rich structure semantics within documents.Previous works typically formulated layout reading order as a permutation of layout elements, i.e. a sequence containing all the layout elements.However, we argue that this formulation does not adequately convey the complete reading order information in the layout, which may potentially lead to performance decline in downstream tasks.To address this issue, we propose to model the layout reading order as ordering relations over the set of layout elements, which have sufficient expressive capability for the complete reading order information. To enable empirical evaluation on methods towards the improved form of reading order prediction (ROP), we establish a comprehensive benchmark dataset including the reading order annotation as relations over layout elements, together with a relation-extraction-based method that outperforms previous models. Moreover, we propose a reading-order-relation-enhancing pipeline to improve model performance on any arbitrary VrD task by introducing additional reading order relation inputs.We conduct comprehensive experiments to demonstrate that the pipeline generally benefits downstream VrD tasks:(1) with utilizing the reading order relation information, the enhanced downstream models achieve SOTA results on both two task settings of the targeted dataset; (2) with utilizing the pseudo reading order information generated by the proposed ROP model, the performance of the enhanced models has improved across all three models and eight cross-domain VrD-IE/QA task settings without targeted optimization.

## introduction the field of visually-rich documents (vrds) has attracted widespread attention recently. towards the growing industrial demands of automated processing of vrds, a wide range of challenging tasks have arisen including understanding, classifying, extracting, and post-processing the information of vrds @xcite @xcite . among those tasks, reading order prediction (rop) is a peculiar task on vrd layouts aiming to determine the human-acceptable reading order of layout elements, i.e. 2d-situated words or text regions (segments) given their bounding boxes. rop is a fundamental task for layout understanding as it reveals the influence of layouts for human reading strategies @xcite . the task is challenging due to the diversity and complexity of layout designs among documents, and is also important in applications as previous works have demonstrated that the performance of downstream vrd tasks can be substantially improved by leveraging proper reading order information @xcite . however, we argue that the prevailing task formulation of rop is inappropriate as it does not convey the complete layout reading order information. in recent works, the layout reading order is typically represented as a permutation of layout elements, i.e. a sequence containing all the words or segments within the layout @xcite . however, as shown in fig. 1 , there exists multiple permutations of the displayed layout that represents a humanacceptable reading order, and some of which conflict with other ones. 1 therefore, representing the layout reading order information with one single permutation is unable to convey the complete reading order information in the layout, and could possibly inject noise in the annotations. both issues negatively affects its application in downstream tasks. given the problem, we propose two questions: (1) how to improve the formulation of reading order to include the complete information? (2) would the improved formulation be truly helpful in practical applications that downstream vrd tasks could be further benefited by leveraging the improved form of reading order information? to address the first question, we propose to model the layout reading order as ordering relations over the set of layout elements. since there lacks a clear definition of layout reading order, we investigate its properties to conceptualize it with two terms, each of which is represented as an ordering relation: ( 1 experiments demonstrate that the proposed method outperforms previous sequence-based methods by a large margin on rop. to address the second question, we propose the reading-order-relation-enhancing (rore) pipeline which is confirmed to be effective by comprehensive experiments on downstream vrd-ie/qa tasks. for injecting reading order relation information, we propose to represent it in a n * n binary matrix over n input textual tokens of ptlms, and introduce a relation-aware attention mechanism to incorporate the reading order information. experiment results show that with additional reading order inputs, the performance of three baseline ptlms have shown universal improvement. moreover, considering that ground truth annotation of reading order information is unavailable in real-world scenarios, it is expected to leverage pseudo reading order labels generated by an off-the-shelf rop model in any arbitrary downstream tasks. the expectation is proven feasible by experiment results, since the performance of three reading-order-relationenhanced models are comprehensively improved across eight prevailing benchmarks of vrd-ie/qa from different domains, achieving sota on two of them without targeted optimization. overall, the main contributions of this work are: 1. we identify the existing problem that the current task form of rop fails to convey the complete layout reading order information, and suggest modeling the reading order as ordering relations over layout elements. we introduce a new comprehensive benchmark for quantitative evaluation of the improved formulation of rop. table within the layout can be read either vertically or horizontally; (d) indirect reading order relationship is also important as relevant elements may be separated by other contents. generate high-quality pseudo reading-orderrelation labels for arbitrary inputs from various domains. 3. we introduce a pipeline to enhance arbitrary vrd tasks based on the proposed rop model. comprehensive results demonstrate that the performance of several prevailing methods on various downstream vrd-ie/qa tasks can be universally improved by leveraging the (probably pseudo) reading order relation information. the enhanced methods have achieved sota results on the targeted benchmark and also two cross-domain task settings. the proposed dataset and code implementation of proposed models are released at https://github. com/chongzhangfdu/roor . ## related work modeling of layout reading order the assumption that layout reading order could be represented as a permutation on layout elements can be traced back to @xcite . since then, rule-based methods @xcite , statistical-ml-based methods @xcite and deeplearning-based methods @xcite have been proposed. however, several previous works have noted that layout reading order information cannot be fully conveyed by one single permutation of layout elements. these works improved their utilization of reading order information according to specific application requirements @xcite . however, these works do not delve further into the underlying problem that the formulation of layout reading order should be improved. in this work, we model the reading order as ordering relations over layout elements, inspired by previous works which modeled the non-linear structures of vrd layouts as binary relations @xcite or ordered structures @xcite , and also order-theoretic modeling of linguistic objects @xcite @xcite . a more comprehensive introduction of the evolution of modeling and utilizing layout reading order is available at appendix b. introduced a special module for better modeling the global relationships. more works attempted to better leverage the relative positional relationship between layout elements by improving attention mechanism @xcite @xcite @xcite @xcite , utilizing graph structure @xcite @xcite @xcite @xcite , or relation-aware pre-training @xcite @xcite @xcite . ## conclusion in this paper, we investigate reading order prediction for visually-rich document layouts, a classic and important task of document ai. we question the traditional formulation of this task as it is unable to convey the complete reading order information within the layouts, indicating that the improper formulation would possibly affect model performance on downstream vrd tasks. to address the problem, we propose an improved formulation of this task by conceptualizing layout reading order as ordering relations. we further prove the effectiveness of utilizing the improved form of reading order information in downstream tasks by extensive experiments. the proposed reading-order-relation-enhanced methods achieve universal performance improvement on multiple task settings of vrd-ie and vrd-qa, including sota performance on two targeted settings and two cross-domain settings. to facilitate the development of automated methods on rop in the improved form, we also establish a benchmark dataset with high-quality reading order relation annotations together with a baseline model, which builds a pipeline to generate usable pseudo reading order labels for any arbitrary downstream tasks.