Leave No Document Behind: Benchmarking Long-Context LLM s with Extended Multi-Doc QA (2024)

Abstract: Long-context modeling capabilities of Large Language Models (LLMs) have garnered widespread attention, leading to the emergence of LLMs with ultra-context windows. Meanwhile, benchmarks for evaluating long-context language models are gradually catching up. However, existing benchmarks employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-context applications. To bridge this gap, we propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Unlike typical document QA, in Loong’s test cases, each document is relevant to the final answer, ignoring any document will lead to the failure of the answer. Furthermore, Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Extensive experiments indicate that existing long-context language models still exhibit considerable potential for enhancement. Retrieval augmented generation (RAG) achieves poor performance, demonstrating that Loong can reliably assess the model’s long-context modeling capabilities.

## introduction large language models (llms) have exhibited remarkable proficiency in diverse downstream applications @xcite . recent works focus on scaling up the context window of llms @xcite @xcite , which is crucial for llms in handling complex * equal contribution. we commence with "leave no document behind" and scatter the evidence across multidocument long contexts. in this context, bypassing any document will lead to an erroneous answer, which better tests the long-context modeling ability. to this end, this paper develops loong, an innovative benchmark crafted to evaluate the long-context ability of llms across multiple documents in realworld scenarios. loong typically consists of 11 documents per test instance on average, nearly all of which are selected from the year 2024, spanning three real-world scenarios in english and chinese: (1) financial reports, (2) legal cases, and (3) academic papers. meanwhile, loong introduces new evaluation tasks from the perspectives of spotlight locating, comparison, clustering, and chain of reasoning, every test case is newly annotated. furthermore, loong features inputs of varying lengths (e.g., 10k-50k, 50k-100k, 100k-200k, >200k) and evaluation tasks of diverse difficulty, enabling fine-grained assessment of llms across different context lengths and task complexities. we conduct extensive experiments on loong to test the long-context modeling capabilities of several advanced llms. the empirical results show that even the current most powerful llms still struggle with the tasks in loong, suggesting significant room for improvement in current llms. furthermore, this paper conducts in-depth analyses regarding the behavior of long-context llms involving rag and the scaling law of context size. our main contributions are summarized as follows: all test instances are newly annotated and checked to guarantee quality. ## conclusion in this study, we propose loong, a questionanswering format benchmark designed to evaluate long-context comprehension in real-world multidocument scenarios. we analyze six advanced language models (llms), considering variations in their parameter sizes and context windows, including gpt-4o and gemini-pro1.5. notably, even the most powerful long-context llms fail to achieve satisfactory performance. furthermore, we conduct in-depth analyses to enhance long-context modeling capabilities by comparing the rag approach and the scaling laws related to context size.