READ oc: A Unified Benchmark for Realistic Document Structured Extraction (2025)

Abstract: Document Structured Extraction (DSE) aims to extract structured content from raw documents. Despite the emergence of numerous DSE systems, their unified evaluation remains inadequate, significantly hindering the field’s advancement. This problem is largely attributed to existing benchmark paradigms, which exhibit fragmented and localized characteristics. To offer a thorough evaluation of DSE systems, we introduce a novel benchmark named READoc, which defines DSE as a realistic task of converting unstructured PDFs into semantically rich Markdown. The READoc dataset is derived from 3,576 diverse and real-world documents from arXiv, GitHub, and Zenodo. In addition, we develop a DSE Evaluation S ^3 uite comprising Standardization, Segmentation and Scoring modules, to conduct a unified evaluation of state-of-the-art DSE approaches. By evaluating a range of pipeline tools, expert visual models, and general Vision-Language Models, we identify the gap between current work and the unified, realistic DSE objective for the first time. We aspire that READoc will catalyze future research in DSE, fostering more comprehensive and practical solutions.

## introduction the wealth of knowledge preserved in documents is immeasurable. document structured extraction (dse), which involves converting raw documents into machine-readable structured text @xcite @xcite , is increasingly crucial in realworld scenarios. it facilitates building extensive knowledge bases @xcite , constructing high-quality corpora @xcite , and plays a pivotal role in retrieval-augmented generation (rag) @xcite applications for large language models (llms) @xcite . recent progress in document ai @xcite @xcite has led to the creation of numerous dse systems @xcite @xcite . however, the absence of unified evaluation in real-world scenarios has left uncertainty about their performance levels and hindered their further development. this issue is largely due to the limitations of prevailing benchmark paradigms, which exhibit fragmented and unrealistic characteristics. firstly, as depicted in figure 1 , existing benchmarks typically fragment dse into distinct subtasks, including document layout analysis @xcite , optical character recognition @xcite , tableof-contents extraction @xcite , reading order detection @xcite , table recognition @xcite and formula conversion @xcite . due to their narrow focus, diverse data sources, and inconsistent formats, existing benchmarks lack a unified framework to com- @xcite , docbank @xcite , robust reading @xcite , pubtabnet @xcite , tablex @xcite , im2latex-100k @xcite , readingbank @xcite , and hrdoc @xcite . to address these issues, we introduce readoc, a unified benchmark designed to quantify the gap between existing work and the goal of realistic document structured extraction. readoc formulates dse as an end-to-end task, converting multi-page pdfs into structured markdown. we automatically construct 3,576 pdf-markdown pairs from arxiv, github, and zenodo, covering diverse types, years, and topics to reflect real-world complexity. additionally, we develop a dse evaluation s 3 uite with three modules: standardization, segmentation, and scoring, enabling unified evaluation of diverse dse systems, including pipeline tools @xcite , expert models @xcite , and vision-language models @xcite . our contributions are three-fold: 1) readoc is the first benchmark to frame dse as a pdf-to-markdown paradigm, which is realistic, end-toend, and incorporates diverse data. 2) an evaluation s 3 uite is proposed to support the unified assessment of various dse systems and to quantify multiple capabilities required for dse. 3) we present the gap between current research and realistic dse, emphasizing the importance of explor-ing new modeling paradigms. the code and data are publicly available at https://github.com/ icip-cas/readoc . recent research conceptualizes dse as a singlepage image-to-markup task @xcite , and targeted benchmarks such as omnidocbench @xcite have emerged. although omnidocbench demonstrates commendable richness in evaluation and diversity of data, its single-page approach falls short in handling multi-page or lengthy documents often encountered in real-world scenarios. ## task definition we establish a realistic task paradigm for end-toend dse, using raw pdf documents as input due to their prevalence and unstructured nature, which poses challenges with dispersed content and multimodal information. on the other hand, we employ markdown as the output format, leveraging its lightweight markup to represent structural elements like headings and lists. we adopt a variant of markdown @xcite that supports l a t e x syntax for tables and formulas. markdown, as the target format, can be chunked, indexed as flat text, or directly ingested by llms. in summary, readoc uniformly defines dse as a task that takes a complete pdf file as input and generates structured text in markdown format, which is well-defined, practical, and challenging for dse systems. examples of task inputs and outputs are provided in appendix a. ## benchmark construction readoc is a unified dse benchmark derived from real-world documents. we select heterogeneous documents from arxiv preprints foot_0 , github readmes foot_1 and zenodo's open research repository foot_2 , which are then automatically processed to construct pdf-markdown pairs. readoc consists of 3,576 documents: 1,009 in the readoc-arxiv subset, 1,224 in the readoc-github subset, and 1343 in the readoc-zenodo subset. each subset offers unique challenges: readoc-arxiv features complex academic structures such as formulas and tables, with diverse multi-column layout templates. however, its heading styles are simple, often following easily recognizable patterns like "1.1 introduction." in contrast, readoc-github includes only basic elements like paragraphs and headings, and presents a uniform singlecolumn layout style. however, building its toc structure is challenging due to varied and often unmarked heading styles. readoc-zenodo contains longer documents (many exceeding 30 pages) and diverse types, such as posters, reports, theses, and books, challenging layout analysis. additionally, its 27 languages further increase text processing and semantic understanding complexity. each subset exhibits significant diversity in types, topics, eras, and so on, establishing readoc as a robust benchmark. we describe more construction details in appendix b. ## evaluation s 3 uite considering the potential confusion that various models may have with our markdown syntax and the multifaceted nature of the capabilities required for dse tasks, we propose an evaluation s 3 uite, consisting of three sequential modules: standardization, segmentation, and scoring, as shown in figure 3 . the s 3 uite ensures that readoc can automatically perform a unified evaluation of the end-to-end dse task, yielding reliable and effective assessment results. more implementation details are in appendix c. ## conclusion this paper proposes readoc, a novel benchmark that frames document structured extraction as a realistic, end-to-end task, i.e., transforming unstructured pdfs into semantically rich markdown text. based on an evaluation s 3 uite, we conduct a unified evaluation of state-of-the-art approaches, including pipeline tools, expert models and general vlms. our experimental results reveal critical gaps in current researches when applied to realistic scenarios and underscore the importance of exploring new research paradigms.