DLUE : Benchmarking Document Language Understanding (2024)

Abstract: “Understanding documents is central to many real-world tasks but remains a challenging topic.Unfortunately, there is no well-established consensus on how to comprehensively evaluate docu-ment understanding abilities, which significantly hinders the fair comparison and measuring theprogress of the field. To benchmark document understanding researches, this paper summarizesfour representative abilities, i.e., document classification, document structural analysis, docu-ment information extraction, and document transcription. Under the new evaluation framework,we propose Document Language Understanding Evaluation – DLUE, a new task suite whichcovers a wide-range of tasks in various forms, domains and document genres. We also systemat-ically evaluate six well-established transformer models and representative LLMs on DLUE, andfind that due to the lengthy content, complicated underlying structure and dispersed knowledge,document understanding is still far from being solved in complex real-world scenarios.”

## introduction documents are basic units of the organization of natural language @xcite . understanding the structures and the semantics of documents is the foundation for understanding news articles @xcite , scientific papers @xcite , government reports @xcite , stories @xcite , etc. evaluating how a machine intelligence system can read, analyze and generate documents is an important part of evaluating its natural language abilities, which has long been a critical direction in nlp field. while standard benchmarks like glue @xcite and superglue @xcite ) have become a critical part of nlp community, they primarily focused on short utterances like sentences or paragraphs. however, documents are much more than bag-of-sentences, which usually focus on the central theme @xcite , with underlying structure bound by complex linguistic elements @xcite and dependent knowledge dispersed across the whole text @xcite . therefore, these benchmarks cannot be used to evaluate document understanding due to its unique challenges: first, documents usually have lengthy content, i.e., they usually are much longer than sentences/paragraphs thus it is difficult to process them due to the computational memory/runtime limitation of current nn models. second, documents have underlying structures, which play a critical role during understanding. for example, mining arguments from a document needs to model both the local coherence between sentences and the global interactions between claims and arguments, which cannot be accomplished by only exploiting sentence-or paragraph-level information. third, the knowledge in a document is usually dispersed beyond sentences/paragraphs, which makes it necessary to model and explore document-level context. for example, long-distance within-document coreference resolution needs to integrate global information to find all expressions that refer to the same entity distributed in full text. recently, an increasing number of researches have paid attention to the evaluation of document understanding. yi @xcite proposes the long range arena (lra), which contains two syn-figure 1 : overview of dlue, which covers a wide-range of tasks and datasets in diverse domains to evaluate four representative document understanding abilities. to systematically evaluate document language understanding abilities, this paper proposes document language understanding evaluation -dlue, a new task suite which covers a wide-range of tasks in various forms, different domains and document genres. figure 1 shows the overview of dlue. specifically, we summarize 4 kinds of document understanding abilities, including 1) document classification, which evaluates whether a model can understand the overall semantics of a document, e.g., its topic @xcite and standpoint @xcite ; 2) document structural analysis, which evaluates whether a model can analyze and leverage the underlying structure of a document, e.g., its discourse structure @xcite and argument structure @xcite ; 3) document information extraction, which evaluates whether a model can recognize and aggregate associated information spanning cross whole document, e.g., long-distance within-document coreference @xcite ; 4) document transcription, which evaluates whether a model can capture and transcript important information of a document, e.g., summarization @xcite and abstractive qa @xcite . then we collect 9 datasets and align them with the above abilities. datasets of the same group are converted to a unified format. in this way, dlue provides a comprehensive benchmark for document language understanding, which enables the research community to fairly compare and measure the progress of this field. to better understand the challenges of document language understanding and analyze the performance of current approaches, we experiment on dlue using several advanced document understanding models, including 1) memory-based approaches, which includes xlnet @xcite ; 2) pattern-based approaches, which includes longformer @xcite , bigbird @xcite , sparse transformer @xcite ; 3) low-rank/kernel-based approaches, which includes linformer @xcite and performer @xcite ; 4) large language models, which includes gpt-4, gpt-3.5-turbo 0 , vicuna-7b @xcite , llama-2-7b-chat @xcite , mistral-7binstruct @xcite and chatglm3 @xcite . results show that document understanding is still far from being solved due to lengthy content, complex structure and dispersed knowledge, and currently there is no neural architecture that dominates all tasks, raising needs for a universal document 0 https://openai.com/blog/openai-api c c l 2 0 2 4 understanding architecture. generally, the contributions 1 of this paper are: ## dlue: document language understanding evaluation benchmark the section describes the dlue benchmark, used to evaluate 4 representative document understanding abilities. specifically, dlue is centered on 9 english document understanding datasets, which cover a wide-range of tasks in various forms, different domains and document genres. ## conclusions we propose a new benchmark dlue that places the spot on documents and their lengthy content, complex underlying structure and dispersed knowledge challenges. dlue covers diverse document-level tasks to evaluate four basic abilities required by document understanding, including document classification, document structure analysis, document information extraction and document transcription. based on dlue, we conduct an extensive side-by-side comparison of three document understanding architec-c c l 2 0 2 4 tures. experiments demonstrate document understanding is far from being solved, and there exists a need for a universal architecture that can dominate all tasks. limitations dlue now focuses on plain text documents, while the documents one encounter, e.g., scientific articles, company announcements, or even personal notes, may also contain multi-modal information and with non-sequential structure. in future work, we intend to integrate these multi-modal, complex structure information into our document understanding benchmark. besides, due to the huge cost of computing resources, we didn't pretrain models specialized for document understanding, but directly use the public pretrained versions or train from scratch. we believe an unified pretraining incorporating document-related tasks will enhance understanding performance.