L ong L eader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios (2025)

Abstract: Large Language Models (LLMs), exemplified by Claude and LLama, have exhibited impressive proficiency in tackling a myriad of Natural Language Processing (NLP) tasks. Yet, in pursuit of the ambitious goal of attaining Artificial General Intelligence (AGI), there remains ample room for enhancing LLM capabilities. Chief among these is the pressing need to bolster long-context comprehension. Numerous real-world scenarios demand LLMs to adeptly reason across extended contexts, such as multi-turn dialogues or agent workflow. Hence, recent advancements have been dedicated to stretching the upper bounds of long-context comprehension, with models like Claude 3 accommodating up to 200k tokens, employing various techniques to achieve this feat. Aligned with this progression, we propose a leaderboard LongLeader that seeks to comprehensively assess different long-context comprehension abilities of diverse LLMs and context length extension strategies across meticulously selected benchmarks. Specifically, we aim to address the following questions: 1) Do LLMs genuinely deliver the long-context proficiency they purport? 2) Which benchmarks offer reliable metrics for evaluating long-context comprehension? 3) What technical strategies prove effective in extending the understanding of longer contexts? We streamline the evaluation process for LLMs on the benchmarks, offering open-source access to the benchmarks and maintaining a dedicated website for leaderboards. We will continuously curate new datasets and update models to the leaderboards.

## introduction long-context large language models (llms) refer to those llms that can handle very long input or output lengths, by understanding and generating, retrieving and reasoning over long text sequences. as llms have stronger capabilities and wider applications @xcite , its long-context capability becomes a vital bottleneck for various use cases (e.g. in-context learning, retrieval augmented generation, agent workflow etc.) @xcite @xcite . prior work regarding long-context llms has proposed various training-or inference-stage methods that can extend llm's contexts @xcite , which shows reasonable and improved performance on long-context benchmarks @xcite . however, different benchmarks focus on different perspectives of long context capabilities of llms, and different methods report fluctuated benchmark performances, because there is no gold standard for reporting unified benchmark scores regarding various long context capabilities, leading to unclear understanding the effectiveness of various long context extending methods. to this end, we propose longleader, a standard and unified long-context llm benchmark which covers various dimensions, where various llms and long-context extending methods can be fairly and comprehensively compared. we also continually pretrained llms with various context extending methods, and reported performances of these methods on our unified benchmark, shedding lights on the effectiveness of those methods. specifically, although there are many longcontext benchmarks proposed and frequently used for evaluating long-context llms, the reliability of each datasets is unclear. also, one benchmark typically focus on sepecific dimensions of long context capability. meanwhile, metrics and evaluation details are typically not unified for different models, even on the same benchmark, leading to unfair comparison of models, and even over-claimed effectiveness of some methods. to address this, we carefully select some representative datasets to compose a new reliable and comprehensive benchmark, covering 6 dimensions of long-context capabilities (i.e. multi-choice and freeform qa, in context learning, summarization, logical reasoning and synthetic retrieval). to ensure a fair comparison of a model's long-context capability across various context lengths, we define the long context capability for both 0-32k and 0-128k token ranges. we then evaluate llms under each setting, with the 128k setting encompassing the 32k subset. with our proposed benchmark, we re-evaluated several off-the-shelf leading open and closed llms, in order to have a fair comparison of those models, assessing whether each of them achieve the long context capability they claimed. for open models, we selected llama 2 @xcite and mistral&mixtral @xcite etc. as the representative models. for closed models, we chose the claude family models, because they are well-recognized models with leading performance. with a pretrained base llm, it is still unclear with which continual pretraining methods, we can extend the limited context length to our desired context length. thus, we also continually pretrain open llms with various context-length extension methods, and evaluate them on our proposed benchmarks, which enables us to demystifying which methods can most effectively extend long contexts of llms during continual pretraining. according to the results of off-the-self llms and our continually pretrained llms on our benchmark, our major findings are summarized as fol-lows: ## benchmark selection to fortify the robustness and ensure the reliability of our analytical processes, we instituted a detailed and systematic selection protocol for assessing existing long-context evaluation benchmarks. a benchmark of high calibre should: ## leaderboards in this section, we present the leaderboards for our selected benchmarks. we feature two primary leaderboards, comparing 10 widely-used and popular public models across the two sets of benchmarks, each with context lengths of 32k/128k. additionally, we include an auxiliary leaderboard to assess various context extension techniques using the same continuing pretraining recipe. ## analysis and insights to extend the context window by continue pretraining, yarn is better. from a strong base model means strong long context abilities. long context capability is nothing different from other perspectives of llms. it's mainly from the base model. we can clearly see that, with the same fine-tuning data and same extension methods, larger models perform much better than small models. most long context tasks have a bunch of noisy or useless information. generally, larger models are more capable of focusing on and extracting useful information from the input texts. we argue that this is one of the major reasons of larger models' superiority on long context tasks. also, with similar size, stronger base models implicitly have stronger long context abilities. to further illustrate this, we conduct more experiments for llama-2-7b, llama-2-70b and llama-3-8b on the needle-in-a-haystack task. we use selfextend @xcite to extend the three models' context window. we suppose selfextend, as a finetuning free method, will not change models' abilities. from appendix e consistent long context generation is challenging. the original passkey retrieval task (mohtashami and jaggi, 2024) uses a 5-digit passkey. it can be perfectly (i.e. 100% accuracy) solved by many existing long context handling methods @xcite @xcite @xcite . but it becomes different with the introduction of longer passkeys. our findings indicate a decline in performance as the length of the passkey increases, as in appendix c super long passkey challenges llms' abilities to precisely memorize useful information and generate consistent prediction according to it. compared to standard short context tasks, usually, long context tasks have lower information density. the model just needs a small part of the context to make predictions. if the required information is short, it's easy for a model to memorize the information. as the required information lengthens, the task of accurately memorizing and regenerating every detail becomes increasingly complex. moreover, the prediction of longer passkeys demands a higher tolerance for error accumulation during inference. supporting this observation, nearly all failure cases still successfully identify the location of the passkey, but they cannot precisely repeat the whole passkey sequence, such as missing parts of the passkey or wrongly repeating several digits. this inconsistency underscores the challenges faced by llms in generation of long sequences. with high noise level, retrieval tasks can be difficult. retrieval is one of the most fundamental tasks in machine learning. typically, these tasks do not require complex reasoning abilities; however, they can become significantly challenging with increased noise levels. all models generally perform well within their pretraining context windows on passkey retrieval tasks. conversely, their perfor-mance tends to diverge during key-value (kv) retrieval tasks. both tasks require the large language model (llm) to retrieve a simple message from the context according to a specified 'key'. the primary distinction between the two is that kv retrieval involves a noisier context compared to passkey retrieval. specifically, the context in kv retrieval comprises other 'key-message' pairs, whereas the context in passkey retrieval consistently repeats a single, irrelevant sentence multiple times, which bears no relation to the target 'key-message' pair. this results in differents failure patterns of the two tasks: kv retrieval failures often occur when the model is distracted by competing pairs and extracts the incorrect pair, whereas passkey retrieval failures typically involve the model's inability to generate the complete sequence as described earlier. similar findings are reported by ruler @xcite , which designed several 'needle-ina-haystack' variants by adjusting noise levels. in these studies, llms consistently exhibited poorer performance on variants with higher noise levels. nevertheless, simple retrieval tasks like passkey retrieval remain valuable as they serve as touchstone for assessing whether a model can access all information in an input sequence. existing research @xcite @xcite indicates that some linear-time models or attention mechanism, despite abilities comparable to vanilla transformers, have a limited receptive field to the input sequence, leading to their failure in these simple retrieval tasks. we plan to test such models in the near future. abf-based models can somewhat work beyond their pre-training windows. despite the context window being only 8k for mixtral and 32k for mistral, these models effectively handle information retrieval tasks involving up to 128k contexts. what happens here is similar to what is observed in longlora @xcite : after interpolation, although continuously trained on short sequences, llms are able to generalize to longer sequences. more details about the connection between this phenomenon and existing works are in appendix f. ## related work long context benchmarks. our work is closely related to other works on benchmarking longcontext language models. longeval (dacheng li*, 2023) is one of the pioneer work which includes various long context retrieval tasks. long-bench @xcite ) contains various longcontext tasks in a bilingual setting. it includes variable task types such as question answering, coding, summarization, and others. infinitebench @xcite ) is similar to longbench, but its tasks have a greater than 100k token length, while most data in longbench is less than 20k tokens. some tasks in infinitebench are so difficult that nearly no models can handle them. ada-leval @xcite introduces novel segment sorting and many-choice selection tasks. it has a more fine-grained length distribution ranging from 1k to 128k tokens. longiclbench @xcite targets challenging the in-context learning abilities of large language models (llms). for each task, it constructs the data in rounds. a round means a complete set of all candidate labels. each task includes settings of 1 to 5 rounds, varying the context to investigate the influence of the number of examples in in-context learning. ruler @xcite is designed to challenge the true context window length of long-context llms. to control the task complexity and data length, it is composed entirely of synthetic data such as variants of needlein-a-haystack by adjusting the number of needles and the contents of the contexts. loogle @xcite tries to test llms' abilities to extract and understand long-range dependencies over the entire sequence. the data in loogle is mainly around 32k tokens. context window extension methods. several methods extend the context window for llms: retrieval-based approaches: these use an external memory module to store past context and fetch related documents during inference, necessitating modifications to llm architectures. examples include activation beacon @xcite and landmark attention @xcite . fine-tuning based approaches: one line of these methods interpolates long context positions into the original llm context window, optimizing pre-trained models with large base values for position embeddings. notable implementations are codellama @xcite and lla-malong @xcite . longlora @xcite and clex @xcite reduce gpu resource demands by fine-tuning on sequences shorter than the target length. another line uses short sequences to mimic long sequences during fine-tuning such as pose @xcite . attention-based methods: beyond position em-bedding interpolation, these methods adapt attention mechanisms to manage input context without requiring fine-tuning. sliding window attention limits attention to nearby tokens but fails to retain distant token information @xcite . streamlm @xcite and llm-infinite @xcite apply similar strategies, focusing on head and neighboring tokens while masking intermediates. selfextend @xcite maintains all input tokens but only retains accurate positional information for tokens within a close range, simplifying distant positions to those seen during pre-training. kv cache utilization: techniques such as heavy-hitter oracle @xcite , snap-kv @xcite and in-fllm @xcite develop kv cache eviction policies that optimize token retention based on attention scores or token window relevance during generation, thereby efficiently managing memory constraints. ## conclusion in this work, we propose a comprehensive benchmark for evaluating long-context large language models, addressing the lack of a unified standard for consistent performance comparison. our benchmark covers various dimensions and context lengths, enabling a fair comparison of different llms and context-extending methods. this approach provides clearer insights into the effectiveness of these models and methods.