\infty B ench: Extending Long Context Evaluation Beyond 100 K Tokens (2024)

Abstract: Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction. Despite recent strides in making LLMs process contexts with more than 100K tokens, there is currently a lack of a standardized benchmark to evaluate this long-context capability. Existing public benchmarks typically focus on contexts around 10K tokens, limiting the assessment and comparison of LLMs in processing longer contexts. In this paper, we propose , the first LLM benchmark featuring an average data length surpassing 100K tokens. comprises synthetic and realistic tasks spanning diverse domains in English and Chinese. The tasks in are designed to require an understanding of long dependencies in contexts and make simply retrieving a limited number of passages from contexts not sufficient for these tasks. Based on , we evaluate several state-of-the-art LLMs tailored for processing long contexts. The experimental results indicate that existing long-context LLMs still require significant advancements to process 100K+ contexts effectively. Furthermore, we present three intriguing analyses regarding the behavior of LLMs processing long context. Our code and data is released.

## introduction in recent years, large language models (llms) @xcite have exhibited exceptional performance across a range of natural language processing (nlp) tasks @xcite . llms are showing a promising direction toward generalist task assistance, being capable of aiding users in practical tasks through conversational interactions. these tasks include navigation of web pages @xcite , analysis of code repositories @xcite , and extraction of useful information from documents @xcite , indicating a step towards artificial general intelligence. for these llm-based scenarios, the ability to process long contexts is increasingly critical, in addition to understanding fine-grained semantics and possessing extensive knowledge @xcite . textual documents, historical dialogues, complex instructions, and cumbersome workflows, which constitute the data most directly processed in daily tasks, must be input to llms as long contexts for effective processing. despite this growing importance, llms consistently face challenges in processing long contexts, primarily due to the substantial computational resources required for long sequence training @xcite as well as the apparent inability to generalize to sequences longer than those encountered during training @xcite . llms are typically trained on sequences containing no more than 8k tokens @xcite @xcite , and thus cannot well handle contexts exceeding 8k tokens. these limitations largely restrict most llms from being applied to those benchmark avg len en zh code math novel dialogue synthetic lra @xcite ∼10k ✓ ✗ ✗ ✓ ✗ ✗ ✓ longbench @xcite ∼10k ✓ ✓ ✓ ✗ ✓ ✓ ✓ l-eval @xcite 4k -60k ✓ ✗ ✓ ✓ ✗ ✗ ✓ loogle @xcite recent advancements in training infrastructure @xcite @xcite , and efforts to improve length generalization @xcite @xcite foot_0 have led to rapid developments in building long-context llms. based on these improved training infrastructures and length generalization methods, several llms have purportedly managed to process sequences exceeding 100k tokens @xcite @xcite , with claude 2 @xcite and kimi-chat (ai, 2023) even claiming to be able to process up to 200k tokens. however, the rapid emergence of long-context llms has outpaced the development of adequate evaluation benchmarks. present longcontext benchmarks predominantly feature contexts averaging around 10k tokens @xcite , invariably falling below 100k tokens. this lag in the advancement of longcontext evaluation methodologies impedes both the comparative analysis of diverse long-context llms and the pinpointing of potential enhancements in long-context processing. in this work, we present ∞bench , the first comprehensive benchmark featuring an average data length surpassing 100k tokens. ∞bench includes tasks in different domains (novels, code, math, etc.) and languages (english and chinese). to fully evaluate the performance of long-context llms, ∞bench integrates synthetic tasks that can be auto-generated for even longer contexts (e.g., finding the top-k number in an array) and a set of realistic tasks. to construct tasks annotated by humans, we develop 5 annotation pipelines for detailed example annotation. these pipelines undergo iterative refinement until the examples meet quality standards. auto-generated tasks, conversely, can be easily scaled to various lengths. upon completing ∞bench, we assess the performance of several state-of-the-art (sota) long-context llms on this benchmark to gauge its difficulty and evaluate the effectiveness of these models. the results show that current sota llms are not fully equipped to handle all tasks within ∞bench, highlighting the ongoing challenge of making llms process long contexts effectively. we conduct intriguing analyses on the behavior of llms in such long contexts, including the task length ablation, the absence of "lost in the middle phenomenon @xcite )", and the context recalling prompting techniques. our contributions can be summarized as follows: llms for 100k+ tokens many llms have shown the ability to handle over 100k tokens. some popular proprietary 100k+ llms include gpt-4, claude 2 @xcite , and kimi-chat (ai, 2023). on the other hand, there are much fewer open-source 100k+ models. some notable models include yarn @xcite and yi-200k @xcite . in this paper, we benchmark gpt-4, claude 2, kimi-chat, yarn-mistral-7b-128k foot_1 , yi-6b-200k foot_2 , yi-34b-200k foot_3 , chatglm3-6b-128k foot_4 , and minicpm-2.4b-128k foot_5 on ∞bench, which are some of the latest and strongest llms that claim to be able to handle over 100k tokens. inference infrastructure numerous studies aim to accelerate self-attention computation. research primarily concentrates on refining attention mechanisms through improved io management @xcite , memory optimization @xcite @xcite , and enhanced parallelization in decoding @xcite . approaches like sliding window attention @xcite , lm-infinite @xcite , and streamingllm @xcite introduce attention variants for handling infinitely long sequences without overwhelming computation or memory overhead. however, these techniques face challenges in maintaining historical information. long context benchmarks several benchmarks exist for evaluating long-context ai models, notably featuring context lengths of around 10k tokens @xcite . l-eval @xcite and longbench @xcite are prominent examples, aggregating preexisting tasks @xcite @xcite ; ?) into comprehensive benchmarks. longbench encompasses four categories-qa, summarization, synthetic retrieval, and code-spanning 21 tasks, 100k 500k 2000k en.sum en.qa en.mc en.dia zh.qa code.debug code.run math.calc math.find retrieve.passkey retrieve.number retrieve.kv with four being novel. conversely, l-eval incorporates 18 tasks across qa, summarization, math, retrieval, and multiple-choice (mc) domains, introducing three new tasks. another notable benchmark, loogle @xcite , differentiates between short and long dependency examples, focusing on summary and qa tasks; its summary corpus contrasts with ours, utilizing academic papers over novels. the long-range arena (lra) @xcite ) further diversifies with six tasks in text, image, and math, designed for scalability. in comparison, ∞bench stands out for its substantially longer contexts and a broader range of task domains. in total, ∞bench includes 3946 examples, featuring a length beyond 100k tokens (average approximately 200k). figure 2 illustrates the distribution of these tasks. examples per task. next, we illustrate each task in detail. the tasks can be grouped into two broad categories. the first involves realistic context collected from real-world scenarios which has potential practical usage of long context llms. the second depends on synthetic contexts which are created or collected for testing certain capabilities of long-context llms. ## experiments we conduct a thorough set of experiments on ∞bench. we will introduce the baselines, experimental setup, and main results in this section. ## analysis we subsequently perform a detailed analysis of the results, identifying and emphasizing several notable and interesting phenomena. ## conclusions we introduce ∞bench, the first benchmark tailored for long contexts exceeding 100k in average length. empirical evidence indicates that despite claims of proficiency with such extensive contexts, current llms demonstrate significant performance degradation when dealing with them. this finding highlights the need for advanced methodologies to improve llms' efficiency in processing long context. additionally, our analysis offers insights into llm behavior in long-context tasks, guiding future research.