D oc B ench: A Benchmark for Evaluating LLM -based Document Reading Systems (2025)

Abstract: Recent advancements in proprietary large language models (LLMs), such as those from OpenAI and Anthropic, have led to the development of document reading systems capable of handling raw files with complex layouts, intricate formatting, lengthy content, and multi-modal information. However, the absence of a standardized benchmark hinders objective evaluation of these systems. To address this gap, we introduce DocBench, a benchmark designed to simulate real-world scenarios, where each raw file consists of a document paired with one or more questions. DocBench uniquely evaluates entire document reading systems and adopts a user-centric approach, allowing users to identify the system best suited to their needs.

## introduction the emergence of large language models (llms) has marked a significant milestone in the field of natural language processing, revolutionizing the way we approach a variety of tasks @xcite @xcite @xcite @xcite . existing llms such as gpt-4 @xcite , @xcite , and claude-3 @xcite have shown exceptional abilities in following human instructions to perform tasks such as answering questions, translating languages and summarizing texts. these tasks are typically characterized by straightforward input-output interactions, where the models generate responses solely based on the provided text. however, many real-world applications require more complex interactions involving user-provided documents. for instance, financial analysts might need to query comprehensive financial reports to inform their investment decisions @xcite . legal professionals often search through extensive legal documents to find relevant case law @xcite . similarly, scientific researchers frequently sift through academic papers to identify related works and extract key findings @xcite . when users pose queries based on their provided documents, the situation becomes more intricate and challenging @xcite . unlike standalone llms that are primarily trained to process and respond to textual inputs (or images in the case of vision llms), handling user-provided documents necessitates a more sophisticated approach that stretches beyond the capabilities of a single llm. in order to provide accurate responses, an llm-based document reading system should not only comprehend natural language queries, but also excel in a range of processing skills, including parsing and interpreting user documents and layouts, navigating complex formatting structures, extracting relevant metadata, and managing long textual contexts along with any embedded images. mastery of these diverse skills is essential for generating precise and contextually relevant responses. at the same time, recent advancements in proprietary llm developers such as openai and anthropic have provoked the release of several llmbased document reading systems. figure 1 public blogs 12 , the absence of a standardized benchmark makes it difficult to objectively evaluate and compare the document reading performance across these systems, thereby leaving a critical gap in fairly assessing these capabilities in a fine-grained manner. to fill this gap, our paper introduces docbench, a novel benchmark specifically designed to evaluate llm-based document reading systems. docbench is developed to mirror real-world scenarios where each input consists of a document paired with one or multiple associated questions, and each question is annotated with a golden answer. our benchmark undergoes a meticulous development process, incorporating human annotation and synthetic question generation. to the end, docbench features 229 real-world documents and 1,102 questions spanning 5 diverse domains: academia, finance, government, laws, and news. besides, the benchmark involves 4 question categories, including text-only, multi-modal (i.e., tables and figures), meta-data, and unanswerable, ensuring comprehensive coverage of various document reading capabilities. based upon docbench, we evaluate several proprietary llm-based systems that are accessible via web interfaces or apis. however, these proprietary systems are close-sourced, thus leading to the limited disclosure of their detailed op-1 blog: claude can now use tools https://www. anthropic.com/news/tool-use-ga 2 blog: talk with documents using lla-maindex https://codemaker2016.medium.com/ talk-with-documents-using-llamaindex-3952c76bd511 erational strategies. as a result, we additionally assess a straightforward parse-then-read pipeline employing a series of open-source llms. our evaluations reveal noticeable gaps between existing llm-based document reading systems and human performance, underscoring the challenges of developing proficient systems. in summary, docbench serves as the first standardized benchmark to evaluate llm-based document reading systems within real-world scenarios, where the systems take a document file paired with one or multiple related questions as input and generate textual responses as output. moreover, our benchmark is carefully designed to encompass 5 diverse domains and 4 distinct question types, ensuring a nuanced and thorough assessment. by facilitating fair comparisons across different systems, docbench highlights current limitations and paves the way for future advancements. ## conclusion in this paper, we introduce docbench, a novel benchmark created to assess llm-based document reading systems in a comprehensive and finegrained manner. docbench consists of 229 documents and 1,102 questions, spanning 5 domains and 4 question types, developed with the help of human annotators and synthetic questions. we eval-uate both proprietary llm systems, accessible via web interfaces or apis, and a parse-then-read approach using open-source llms. our findings reveal significant disparities in document reading capabilities among these systems, highlighting current limitations, presenting potential challenges, and thus driving forward progress in this field.