D oc T rack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Reading (2023)

Abstract: The use of visually-rich documents in various fields has created a demand for Document AI models that can read and comprehend documents like humans, which requires the overcoming of technical, linguistic, and cognitive barriers. Unfortunately, the lack of appropriate datasets has significantly hindered advancements in the field. To address this issue, we introduce DocTrack, a visually-rich document dataset really aligned with human eye-movement information using eye-tracking technology. This dataset can be used to investigate the challenges mentioned above. Additionally, we explore the impact of human reading order on document understanding tasks and examine what would happen if a machine reads in the same order as a human. Our results suggest that although Document AI models have made significant progresses, they still have a long way to go before they can read visually richer documents as accurately, continuously, and flexibly as humans do. These findings have potential implications for future research and development of document intelligence.

## introduction with the continuous development of information technology, our access to information is becoming increasingly diverse. among the various formats, the proportion of visual information in daily documents such as tables, graphs, diagrams, etc., is on the rise @xcite . therefore, effectively utilizing such visual information has become a hot research topic in the nlp research community and introduces a new challenge of understanding visually-rich documents (vrds) @xcite , which are documents that contain substantial visual components. ## human reading order the human reading order plays a significant role in the comprehension of vrds. human reading order refers to the direction and sequential order in which people scan documents as they read the texts. generally, people read left-to-right or top-tobottom to comprehend the text and obtain information. this order is also related to the way the text is written in most languages and scripts, including english and modern chinese. however, other reading order patterns exist. for example, traditional chinese and japanese are usually written vertically from right to left, while arabic is written horizontally from right to left @xcite . as a result, people naturally scan their eyes in this order while reading @xcite . although the reading order of text generally follows a relatively fixed pattern, vrds usually present information in a mixture of modalities (e.g., text, images, graphics, etc.) with a complex two-dimensional layout and semantic structure. readers typically choose the appropriate reading order by considering the spatial structure of text content, image information, and the typographic position of the document in combination. this natural temporal processing can help readers better understand the content and intent of the document and establish connections between different modalities. for example, when presented with a diagram, readers typically first visually analyze it as a whole to get a general idea of what it describes. from there, they usually focus on the most obvious data or labels first, and then gradually scan the rest of the data and labels to gain a more complete understanding of the information presented. ## experiments and analysis to gain a deeper understanding of the differences between human reading order and machine processing order, we conduct both intrinsic and extrinsic evaluations using the doctrack dataset. by comparing different modal fusions, the intrinsic evaluation allows us to assess the quality of the reading order generation model, which in turn helps us to understand the extent to which information from each modality influences the reading order built by humans. in order to determine ex-actly what reading order is needed for a machine document intelligence model, i.e., what input order enhances machine document comprehension, we make use of extrinsic evaluation to assess the quality of temporal order on human-like reading order generation. ## conclusion we investigate the impact of human reading order on document ai models for vrd understanding tasks. we propose different methods to generate human-like reading orders, along with a practical preordering pipeline that can leverage the generated reading orders. our observations suggest that true human reading order may not always be suitable for reading vrds. the dataset we construct can help in designing better document ai models and human reading robots in the future.