Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps (2025)

Abstract: Long-context language models (LCLMs), characterized by their extensive context window, are becoming popular. However, despite the fact that they are nearly perfect at standard long-context retrieval tasks, our evaluations demonstrate they fail in some basic cases. Later, we find they can be well addressed with a sufficient number of reasoning steps, guided by specific CoT prompts. This result emphasizes the potential necessity of solving specific long-context tasks using long-CoT methods, while previous long-context benchmarks always ignore the necessity of long reasoning for long-context tasks and treat them as direct QA tasks. Our code and datasets are available at https://github.com/yuyijiong/hard_retrieval_for_llm

## introduction in the past years, long-context language models (lclms) such as gpt-4o-128k () and gemini-1.5-1000k @xcite have surged in popularity, raising questions about their efficacy in handling extended context tasks. while various lclms have demonstrated perfect long-context retrieval ability by passing the "needle in a haystack" test @xcite in over 100k context length, benchmarks like loogle @xcite , ruler @xcite , and loong @xcite have highlighted their shortcomings in more complex tasks. however, previous long-context benchmarks, despite the variety of task types, typically categorized tasks based on their intuitive forms rather than their inherent difficulty or nature, making it hard to rank their true difficulties or determine which sub-task is the bottleneck. what is more, they usually ignore the necessity of reasoning, and simply regard the long-context task as a direct qa task, as they always set the prompt as "directly give a concise * corresponding author answer". this leads to an unclear ability boundary of lclms and difficulties of long-context tasks. some recent works @xcite @xcite have tested models on challenging long-context reasoning tasks which necessitate cot, such as multi-query, multi-hop questions, or questions requiring logic. however, these tasks only increase the complexity of the question, in other words, the number of reasoning steps are just determined by the question itself (so we call these tasks multi-step-question), but not the context (a long context task usually consists of a context and a question). as a result, they still do not find long-context tasks with a straightforward and short question may also necessitate long-cot. therefore, in order to more scientifically define the difficulty of long-context tasks, we conduct a detailed analysis (detailed in appendix e.2 and e.3) of various tasks (especially retrieval-based tasks) from previous long-context benchmarks. we summarize 2 basic cases, which exhibit much higher difficulty and exhibit distinct properties for lclms: multi-matching retrieval and logic-based retrieval. multi-matching retrieval involves recalling multiple items simultaneously, and logic-based retrieval involves logical judgment within retrieval criteria. although they are both "basic" retrieval problems in a straightforward form, our experiments, as shown in figure 1b , demonstrate they can exceed the ability boundary of lclms as the context length grows in non-cot settings, manifested as a rapid drop in accuracy when the context length exceeds a certain value. in contrast, the performance of traditional ones (direct retrieval and multi-step-question retrieval, which are previously researched) is stable. additionally, we conduct some analytic experiments on the micro-level behaviors of lclms to explain why they always fail in these cases. later, we test whether cot prompting @xcite could help in these cases. at first, we try to simply asking the model to think step-by-step @xcite , but find it only makes the model generate unnecessary steps and hardly improves retrieval accuracy. but when trying more well-designed prompts, we identify a sufficient condition of resolving these problems: adopting sufficient reasoning steps. specifically, "sufficient" means the number is at least n + n 2 +n 2 , where n signify total items in the context and n signify the number of items to retrieve. based on the above findings, we categorize longcontext retrieval problems into three basic types: (1) simple: they do not need cot (mainly simple retrieval) (2) question-difficult: they need cot, usually with a few steps, the number of steps are determined by the question itself (mainly multistep-question retrieval) (3) context-difficult: they also need cot, but usually with numerous steps, and the number of steps are determined by the context, specifically, the length of the context or the number of items in the context (such as multimatching and logic-based retrieval) in order to help readers distinguish the difference between question-difficult and context-difficult tasks more clearly, in figure 1a and appendix e.1, we use examples to detail the formal distinctions between multi-matching retrieval, logic-based retrieval and their question-difficult variants (multiquery and logic-required retrieval). in conclusion, certain types of tasks are actually beyond the traditional "long-context capability" of lclms, but rather require a long reasoning process with sufficient steps. so, we summarize two helpful insights for the long-context research: (1) to solve a wider range of long context problems, relying solely on a larger context window is not enough, and long reasoning may also be needed; ## lclms perform poorly in multi-matching and logic-based retrieval in this section, we conduct evaluations on 5 popular long-context models with context windows over 128k tokens, including phi-3.5-mini-instruct (phi-3.5) @xcite , meta-llama-3.1-70b-instruct (llama3.1-70b) @xcite , deepseek-v2.5 (deepseek) @xcite , gemini-1.5-flash (gemini) @xcite , and gpt-4o-2024-08-06 (gpt-4o) @xcite . we find that, multi-matching and logicbased retrieval are really hard, nearly unsolvable for current lclms in long context. in contrast, multi-step-question retrieval proves to be much easier. in the main content we only show the results of gemini and gpt-4o in multi-matching and logicbased retrieval. the complete evaluation results of other models and direct retrieval or multi-stepquestion retrieval are in appendix b. ## using cot with sufficient steps since asking llms to directly do retrieval within one step leads to bad performance, it is natural to wonder if cot-like prompt might help. thus, we design special prompts to guide step-by-step reasoning, to test if this could help model solve these retrieval tasks. in experiments, we employ 4 types of prompts: as shown in therefore, we conclude that llm can indeed solve these context-difficult problems if we scale the reasoning process to n + n 2 +n 2 steps, with a well-designed prompt. however, this method has huge latency when the context is very long or there are too many items to retrieve. so far, we have not found a solution which achieves both accuracy and efficiency purely relying on llm itself (some other technologies may help, shown in appendix f), which indicates the original intention of lclms, to process massive information simultaneously, has not been fully achieved. however, we find the reasoning steps for these retrieval tasks are basically simple and parallel (not involving very complicated or interlocking logic), which indicates there may be methods to compress them to improve efficiency. ## conclusion in this paper, we use evaluation on synthetic datasets to demonstrate that lclms always struggle to solve some basic cases, such as multi-matching retrieval and logic-based retrieval. then, we find they must be addressed by a sufficiently long reasoning process guided by specific prompts. therefore, we highlight that merely extending the context window size and relying solely on the "long-context capability" of lclms cannot address all types of long-context tasks. on the other hand, although long reasoning process can improve accuracy, it is too time-consuming and inefficient. thus, we urge more novel perspectives and approaches to more efficiently and accurately exploit long contexts.