100- L ong B ench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability? (2025)

Abstract: Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM shall enable its users to effortlessly process many originally exhausting tasks — e.g., digesting a long-form document to find answers v.s., directly asking an LLM about it. However, existing real-task-based long-context evaluation benchmarks have a few major shortcomings. For instance, some Needle-in-a-Haystack-like benchmarks are too synthetic, and therefore do not represent the real world usage of LLMs. While some real-task-based benchmarks like LongBench avoid this problem, such benchmarks are often formed in a way where each data sample has a fixed sequence length, which not only makes them solely suitable for models with a certain range of context windows, but also lacks a proxy to know at what length the model/method-of-interest would fail. Last, most benchmarks tend to not provide proper metrics to separate long-context performance from the model’s baseline ability, so when conducting a cross-model/recipe comparison, such conflation makes the user unable to understand how exactly one model or recipe excels at the long-context task in relation to its baseline ability. To address these issues, we introduce a length-controllable, real-life reflective benchmark with a novel metric that disentangles baseline knowledge from long-context capabilities. Experiments demonstrate the superiority of our datasets in effectively evaluating LLMs. All assets are available at https://github.com/uservan/100-LongBench.git.

## introduction the long-context capability has become one of the fundamental competencies @xcite @xcite of contemporary large language models (llms) because it takes the average human critical ## how to truly evaluate language models' long-context capability? to address the two identified problems, we 1) construct a length-controllable long-context benchmark to reduce performance variance across task lengths, and 2) introduce longscore, a new metric designed to accurately evaluate long-context capabilities by disentangling the model's baseline abilities. in detail, we restructure the long-context datasets, based on longbench, l-eval, and other benchmarks. we then design a new pipeline to generate controllable-length long contexts by combining different articles. additionally, we introduce a filtering mechanism in qa-related tasks to mitigate prior knowledge. subsequently, we propose a new metric to isolate a model's long-text capability from base ability (performance on short texts). ## experimental analysis in this section, we conduct comprehensive experiments to first validate the reliability of -longbench and the effectiveness of the proposed ## related works in this section, we review relevant prior research connected to our study. we summarize cuttingedge models known for their strong long-text processing capabilities, explore methods designed to enhance these abilities, and examine the benchmarks commonly used to assess long-text proficiency. additionally, wwe discuss the limitations of existing benchmarks, not disentangling base ability from true long-context capabilities. long-context language models. both opensource and closed-source state-of-the-art models now support extended context lengths of up to 128k tokens or more, including gpt-4 @xcite , gemini @xcite , claude @xcite , llama-3 @xcite . these models typically achieve long-context capabilities through a combination of improved pretraining and post-training techniques. for instance, many models adopt two-stage or continued pretraining pipelines, where an initial short context window (e.g., 4k or 8k) is later extended to longer lengths (e.g., 128k) using scalable attention mechanisms such as flashattention @xcite and optimized positional encoding schemes @xcite @xcite . this trend is well-documented in recent technical reports @xcite @xcite , which highlight how careful adjustments to training schedules, data distribution, and architecture design contribute to stable performance in extreme long-context settings. nonetheless, despite these advancements, effectively evaluating and comparing the true reasoning ability of such models in long-context scenarios remains a significant challenge in the real situations and scenarios. ## conclusion our benchmark and metric address key shortcomings in current evaluation methodologies, such as the inability to isolate long-context reasoning from baseline performance and reliance on insufficiently representative tasks. by incorporating realworld data, diverse task types and difficulties, and a novel metric (longscore), -longbench provides a robust platform to evaluate and compare llms across varying context lengths. this allows for a deeper understanding of how models handle extended contexts while minimizing the influence of prior knowledge or base abilities. as llms continue to evolve, the ability to rigorously assess their long-context reasoning will play a critical role in identifying bottlenecks and guiding the design of next-generation models. our approach sets a new standard for assessing llms, paving the way for more robust innovations in long-context evaluation. furthermore, it will provide an actionable insight for optimizing model architectures and training strategies to enhance long-context capabilities.