Context Length Alone Hurts LLM Performance Despite Perfect Retrieval (2025)

Abstract: Large language models (LLMs) often fail to scale their performance on long-context tasks performance in line with the context lengths they support. This gap is commonly attributed to retrieval failures—the models’ inability to identify information in the long inputs that is relevant to the task they are solving. Accordingly, recent efforts often focus on evaluating and improving LLMs’ retrieval performance: if retrieval is perfect, a model should, in principle, perform just as well on a long input as it does on a short one—or should it? This paper presents findings that the answer to this question may be negative. Our systematic experiments across 5 open- and closed-source LLMs on math, question answering, and coding tasks reveal that, even when models can perfectly retrieve all relevant information, their performance still degrades substantially (13.9%–85%) as input length increases but remains well within their claimed context lengths. This failure occurs even when the irrelevant tokens are replaced with minimally distracting whitespace, and, more surprisingly, when they are all masked and the models are forced to attend only to the relevant tokens. A similar performance drop is observed when all relevant evidence is placed immediately before the question. Our findings reveal a previously-unrealized limitation: the sheer length of the input alone can hurt LLM performance, independent of retrieval quality and without any distraction. They motivate our simple, model-agnostic mitigation strategy that transforms a long-context task into a short-context one by prompting the model to recite the retrieved evidence before attempting to solve the problem. On RULER, we observe a consistent improvement of GPT-4o up to 4% on an already strong baseline.

## introduction recent large language models (llms) have substantially expanded their context windows. for * equal contribution. recent studies suggest that llms approach longcontext tasks through two interleaved processes: this work provides evidence that calls this premise into question. our systematic, controlled experiments across 5 open-and closed-source models on math, question answering, and code generation tasks show that even when a model can perfectly retrieve all the evidence-in the strictest possible sense, reciting all tokens with 100% exact match-its performance still degrades substantially as input length increases ( §3). for example, llama-3.1-8b instruct, with a claimed 128k context length, is able to retrieve all evidence with exact matches for 970 of 1000 mmlu problems @xcite extended to 30k tokens with irrelevant tokens -matching its retrieval performance on the same problems presented in the original form with shorter contexts. however, despite this robustness in retrieval, its accuracy drops by 24.2% compared to the short-context case. more concerningly, this failure occurs even when the irrelevant tokens in the long context consist of minimally distracting whitespace (fig. 1 ; §4.1), and even when the evidence is placed immediately before the question. surprisingly, in a separate experiment, we observe a similar performance drop even when all irrelevant tokens are masked and the model attends only to the evidence and the question-identical to those in the short-context setting except for the longer distance between the evidence and the question ( §4.1). these findings reveal a previously-unrealized limitation: the sheer length of the input alone can hurt llm performance, independent of retrieval quality and without any distraction. they motivate the following hypothesis: even when retrieval is perfect, the model's performance can still be improved by limiting the number of tokens in the input. our controlled experiments provide evidence supporting this hypothesis, and yield a simple and effective retrieve-then-reason mitigation strategy ( §5). specifically, we prompt the model to recite the evidence retrieved from the long context and prepend it directly before the question to form a new, shorter prompt to get the final output; this effectively converts the long-context task into a short-context one. our experiments on gpt-4o on ruler show that this simple approach consistently improves performance by up to 4% on top of an already high baseline performance. our findings reveal previously underappreciated limitations in how current models approach longcontext tasks. they offer a potential explanation for a recurring observation in retrieval-augmented generation (rag) that performance often saturates or even degrades as more documents are added to the context @xcite @xcite , and for recent findings that long cots can sometimes hurt the performance @xcite . these results call for a rethinking of how long-context capabilities are evaluated. in particular, benchmarks that isolate retrieval as a standalone capability might overestimate progress, as improvements in retrieval alone do not necessarily translate into better longcontext performance; instead long-context capabilities should be evaluated holistically. practically, our proposed mitigation strategy is model-agnostic, simple, and effective. ## background and related work recent work on long-context llms has largely followed a dichotomy of long context capabilities: to better reflect real-world use cases, later benchmarks extend this setup to include reasoning tasks that require aggregating multiple pieces of evidence, such as multi-step inference, variable binding, and multi-document question answering @xcite @xcite @xcite @xcite . findings from these evaluations show that strong performance on synthetic retrieval tests does not always translate to more complex long-context tasks @xcite @xcite , suggesting a different conclusion that language models struggle to use information in long-context inputs as effectively as when the information is contained in a short-context @xcite @xcite . these failures are typically attributed to suboptimal retrieval. for example, retrieval performance often drops when more distractors are present @xcite , when retrieval and aggregation of multiple pieces of evidence is required @xcite @xcite , when relevant passages have low lexical overlap @xcite , or when the evidence appears near the middle of the context @xcite . some studies have also noted that irrelevant tokens can distract the model and impair its reasoning @xcite . a deeper understanding of these failuresand llms long-context capabilities in generalrequires carefully controlled experiments that disentangle factors such as context length, token-level distraction, and task complexity. to the best of our knowledge, such analysis remains scarce, yet it is essential for uncovering the true limits of current models and guiding future efforts. this work takes steps toward that goal through a series of controlled experiments designed to address a central question: what prevents a model from solving a problem when it already has perfect access to all the information it needs? while the retrieval performance and the distractions from irrelevant content are both important, our findings reveal a previously overlooked factor: the sheer length of the context itself. our findings complement the prevailing conclusions that long-context performance is often bottlenecked by retrieval failures or distraction. ## measuring long-context performance under perfect retrieval to better understand the factors limiting llms' long-context performance, §3.1 presents a series of systematic, controlled experiments designed to answer a simple but fundamental question: when a model can perfectly retrieve all the information it needs, can it solve long-context tasks as effectively as short-context ones? we observe a consistent and substantial performance drop across 5 open and closed models even when all evidence can be retrieved with a 100% exact match ( §3.2). ## models struggle even without distraction this section aims to answer what prevents a model from effectively using information it has successfully retrieved? in addition to the distraction from the irrelevant tokens, which aligns with existing observations @xcite , we reveal a surprising finding: the sheer length of the context alone can negatively impact the model's performance, even when there is little ( §4.1) to no ( §4.2) distraction. ## shortening input through retrieval: a simple fix having learned the negative impact of the sheer length of the context ( §4), we naturally arrive at the following hypothesis: even when retrieval is perfect, a model's performance can still be improved by limiting the number of tokens in the input. this section introduces a simple, modelagnostic, and effective mitigation strategy for cases where accurate retrieval-though not necessarily perfect-can be achieved. we then present experimental results that support the above hypothesis. retrieve then solve given a long-context input problem, our strategy first prompts the model to retrieve and recite all relevant information from the input context. this recited evidence is then concatenated with the original problem statement to form a new, shorter prompt. the model solves the problem based solely on the recited evidence, without having the long context as part of its input, similarly to starting a new chat session in chatgpt (see fig. 5 for an illustrative example).this effectively turns a long-context task into a short-context one using an additional prompt. this approach is related to @xcite , which improves the model by training it to align to both retrieval and reasoning objectives. our method, in contrast, does not address the retrieval problem itself; rather, with an explicit retrieval step, it assumes accurate retrieval as a prerequisite. experiments we evaluate our strategy on two benchmarks: (1) our synthetic benchmark under the initial setting, where we insert essay tokens between evidence and question of gsm8k problems. (2) two qa tasks, qa1 and qa2, of ruler @xcite , an established long-context benchmark, in which models are provided with a problem and a number of potentially related documents, and are required to retrieve the answer from one or more of the documents. we compare our strategy against the baseline method where the model is directly asked to answer the question based on the input. on our synthetic benchmark, as shown in on qa1 and qa2 tasks of ruler, we experiment with gpt-4o, taking advantage of its retrieval capabilities. as shown in section 5, under varying context lengths ranging from 128k to 4k tokens, while the baseline already achieves strong performance on qa1 (88.2-90.4%), our method yields consistent improvements, reaching 92.2% at 4k. our method achieves a larger improvement on qa2 by maximum 4% at 32k. our results suggests that this simple and modelagnostic approach can enhance a model's ability to make use of the information they can accurately retrieve from long contexts. in doing so, it helps close the gap between improvements in retrieval performance and actual gains on long-context tasks. findings from @xcite which attribute the drop to a distribution bias with position introduced during training. while @xcite targets the drop in retrieval and @xcite addresses the long-context performance as a whole, our work further shows that this cause also applies to the degradation caused by the length itself, regardless of retrieval or distraction strength. our results further imply that the previous twopart decomposition of long-context problem solving into retrieval and problem solving @xcite @xcite is inconclusive, urging researchers to further explore the underlying mechanisms. the current effort to independently focus on improving models' longcontext retrieval and short-context capacities at training @xcite @xcite may not fully translate to an overall improvement in long-context ability: failure is possible despite the model excelling in both. our conclusion encourages a more comprehensive evaluation beyond focusing on retrieval, with a more finegrained analysis to precisely separate each failure mode. our conclusion supplements existing observations for practical applications. it supports previous findings (e.g., @xcite ) that rag suffers from retrieving too many documents. it also aligns with @xcite , which argue that generating excessively long cots can hurt reasoning models, despite being a common strategy. ## conclusion in this work, we expose a previously noticed but unexplored limitation: the performance degradation of language models may be attributed to the length of the input itself, even when the model is able to retrieve all relevant information, and all distractions are removed. our findings challenge the popular view of decomposing long-context task solving into retrieval and problem solving, and encourage more consideration on future model designs and evaluations. our simple yet effective strategy shows that the degradation can be mitigated through reducing the context length, serving as an initial attempt in bridging the gap between retrieval and long-context performance.