Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP (2024)

Abstract: Improvements in language models’ capabilities have pushed their applications towards longer contexts, making long-context evaluation and development an active research area. However, many disparate use-cases are grouped together under the umbrella term of “long-context”, defined simply by the total length of the model’s input, including - for example - Needle-in-a-Haystack tasks, book summarization, and information aggregation. Given their varied difficulty, in this position paper we argue that conflating different tasks by their context length is unproductive. As a community, we require a more precise vocabulary to understand what makes long-context tasks similar or different. We propose to unpack the taxonomy of long-context based on the properties that make them more difficult with longer contexts. We propose two orthogonal axes of difficulty: (I) Diffusion: How hard is it to find the necessary information in the context? (II) Scope: How much necessary information is there to find? We survey the literature on long-context, provide justification for this taxonomy as an informative descriptor, and situate the literature with respect to it. We conclude that the most difficult and interesting settings, whose necessary information is very long and highly diffused within the input, is severely under-explored. By using a descriptive vocabulary and discussing the relevant properties of difficulty in long-context, we can implement more informed research in this area. We call for a careful design of tasks and benchmarks with distinctly long context, taking into account the characteristics that make it qualitatively different from shorter context.

## introduction the ability to deal with ever-longer contexts has been one of the most notable trends among the emerging capabilities of large language models (llms). starting with a few hundred tokens as the * equal contribution figure 1 : a taxonomy of long context tasks based on the distribution of the needed information in the text. tasks with larger scope and higher dispersion are more difficult (indicated by shade) and more indicative of the long context capabilities of large language models. maximal input length of the first attention-based llms @xcite , contemporary models are -technically -able to process up to 128k and even 1m tokens @xcite . the demand to evaluate llms in this setting has led to a line of research on designing long-context tasks and benchmarks, in order to systematically understand models' capabilities and drive their development. however, the field has generally a sole recurring descriptor to define such measurements by -simply, the length of the context. for example, long-context benchmarks group tasks mostly by length in words (e.g., @xcite @xcite . this leads to qualitatively different measurements being conflated together, with conclusions about long-context capabilities being extended from one class of tasks to others. the community is, of course, aware that, for example, tasks which require a small part of the input are different from tasks that require a large part of it. but we ask the more general ques-tion: what are the properties that differentiate tasks when conditioned on their context length? what can we accomplish with such a distinction? in this position paper, we claim that the current landscape of works on long-context evaluation will greatly benefit from a more fine-grained characterization of long-context task design. we argue that judging llms by their ability to process long sequences, while disregarding the task they process them for, overlooks the characteristics that make longer inputs more difficult, and interesting to research, to begin with ( §2). for example, needle in a haystack tasks (niah; @xcite involve queries whose main challenge is finding the relevant information in a long context, without requiring much further processing. synthetic niah datasets are, of course, easier than their natural equivalents @xcite , but the "natural vs. artificial" classification is not informative in our setting, since it applies equally for tasks regardless of context length. what, then, is an informative property? what makes long-context tasks different from each other? for example, multiple-needle variants of niah @xcite , or those that position the "needles" closer or farther apart @xcite . evidently, "the number of tokens in the input" is not a sufficient descriptor. to resolve this roadblock, we present a taxonomy of long-context tasks for the different factors that make them harder when controlling for context length ( §3). this taxonomy is derived by surveying the long-context literature and surfacing the most salient points of distinction between various tasks. we focus on (i) how difficult it is to find and extract the required information from the input (its dispersion in the input), and (ii) the absolute quantity of required information to solve the task (its scope). see figure 1 for a summary. to understand this categorization and its utility, we review the literature on long-context evaluation and position the works with respect to those factors. we find that the most challenging setting, in which a large quantity of required information is present in a dispersed manner that is difficult to extract, is significantly under-explored ( §4). finally, acknowledging the inherent and legitimate reasons behind the focus on context length as the sole descriptor of difficulty, we discuss possible paths forward for designing more reliable measurements of long-context capabilities when utilizing a more nuanced vocabulary ( §5). ## task design in long context evaluating the performance of nlp models over very long contexts is a fast-changing area of research @xcite . measurements are regularly updated to account for new capabilities which improve with extrapolation architectures @xcite and training data @xcite . evaluators were tasked with designing measurements of long-context capabilities cheaply, efficiently, and quickly, while matching realistic use cases as much as possible. the most common way of differentiating long-context tasks, besides the context's length, is whether they are naturallyconstructed or synthetically-constructed @xcite @xcite . ## challenging long context is under-explored revisiting the works surveyed in §2, they clearly differ with respect to both scope and dispersion. with respect to dispersion. the information needed for tasks ranges from easily accessible to highly dispersed and difficult to detect. on low dispersion we have niah @xcite and a myriad of factual single-hop qa datasets @xcite @xcite in which the answer is relatively accessible. adding more snippets of information separated by distractors, either in the form of several needles @xcite or of hops in a multi-hop question @xcite , complicates the information detection due to the need to find at least two snippets @xcite , thereby increasing dispersion. dispersion can also be increased by making the detection of the information less straightforward (e.g., pang et al., 2022) or re-quiring aggregation @xcite . lastly, summarization tasks are of a very high dispersion @xcite , as they require the non-trivial detection of salient facts that are interwoven with the irrelevant text. with respect to scope. tasks overwhelmingly target relatively small scope. in addition to the aforementioned niah tasks and their variants, many qa datasets apply as well @xcite @xcite . a somewhat higher scope is achieved by datasets for querybased summarization @xcite , and qa datasets with more obfuscated answers that require reading the text surrounding the answer for its verification @xcite . although much higher on the scope ladder, book summarization is still limited in its scope as datasets include texts that are only of up to 20k tokens @xcite @xcite . currently, tasks with the highest scope, requiring information across the entire input length, are artificial and of low dispersion, like common words extraction @xcite . ## conclusions we present a taxonomy of factors that make longcontext tasks more challenging compared to short ones. this is in contrast with the existing literature that refers only to the length of the input as the hallmark of long context, and as a result ends up conflating tasks of different character when assessing the ability of models to understand longer text. we reviewed works on evaluation in a longcontext setting and found that the most challenging setting, in which the information needed is of large scope and is highly dispersed within the input, is under-explored. finally, we suggested some leads for future work to tackle this imbalance towards a more informative long context evaluation.