In-Context Learning with Long-Context Models: An In-Depth Exploration (2025)

Abstract: As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets. We study the behavior of in-context learning (ICL) at this extreme scale on multiple datasets and models. We show that, for many datasets with large label spaces, performance continues to increase with thousands of demonstrations. We contrast this with example retrieval and finetuning: example retrieval shows excellent performance at low context lengths but has diminished gains with more demonstrations; finetuning is more data hungry than ICL but can exceed long-context ICL performance with additional data. We use the ICL setting to study several properties of both in-context learning and long-context models. We show that long-context ICL is less sensitive to random input shuffling than short-context ICL, that grouping of same-label examples negatively impacts performance, and that the performance boosts do not arise from cumulative gain from encoding many examples together. We conclude that long-context ICL can be an effective tool, and may not require long-context attention for encoding the demonstration set at all.

## introduction when a few examples are provided in-context, large language models can perform many tasks with reasonable accuracy. while questions remain about the exact mechanism behind this phenomena @xcite , this paradigm of in-context learning (icl) has seen widespread adoption in both academic and industry applications, thanks to its ease of implementation, relatively small computational cost, and ability to reuse a single model across tasks. however, most work has focused on models where the maximum number of demonstrations is severely limited by context length. as more and more methods are developed to adapt language models to extreme context lengths @xcite ; @xcite , inter alia), in-context learning over large quantities of data becomes a potential alternative to finetuning. the properties of icl in this regime are not well-understood; and as the cost of inference over many thousands of tokens can be steep, the efficiency and performance tradeoff between many-shot icl and finetuning on the same data is complex. we conduct a systemic study of long-context in-context learning. namely, we consider: a) the performance of prompting the base model naively, b) retrieving examples to use in-context for each test example, c) finetuning the base model (both full and parameter-efficient finetuning), and d) using models trained to adapt to longer contexts. performance continues to increase past 2000 demonstrations (see figure 1 ), approaching and sometimes exceeding the performance of models finetuned on thousands of examples from the same dataset ( § 3). john will buy the goat cheese tracy liked, milk, a couple of grainy rolls, and tissues. ## experimental setup we consider 5 classification datasets: trec @xcite , trec-fine @xcite , nlu @xcite , banking-77 @xcite , and clinic-150 @xcite ; and 1 generation dataset: samsum @xcite . we compare icl performance across several long-and short-context models, including variants of llama-2 with 4k @xcite , 32k (to-getherai, 2023), and 80k @xcite context windows, mistral-7b-v0.2 @xcite , and qwen 2.5-7b @xcite . for more details on models, see appendix h. ## long-context icl we consider four common methods for using a large dataset. ## properties of long-context icl we compare the properties of long-context icl with the known properties of short-context icl. 7 we primarily use classification, not generation, tasks to study these properties to avoid confounding effects from the difficulty of evaluating generated texts. is it best to use the entire context? prior work suggested that, for some simple tasks, providing ad-ditional input can reduce performance @xcite . however, we observe monotonically increasing performance on nearly every dataset; in cases where the performance curve begins to flatten, small variation occurs, but no significantly lower performance occurs at higher example counts. while using the full context window is computationally costly and may be unnecessary to achieve high performance on some datasets, it is minimally not harmful to performance. sensitivity to example order many models exhibit strong sensitivity to example order in-context @xcite . we examine this by measuring the percentage of predictions that change when the input is reordered (averaged over 3 shuffles). figure 4 shows that, while some sensitivity to order persists, this effect weakens substantially with longer context. across all datasets, the percent of labels flipped by shuffling in 1000-shot icl is less than half the percentage flipped in 10-shot icl. label sorting we also consider an adversarial case for example ordering: we sort the examples so that examples with the same label appear together. at small numbers of examples, this has very little impact; if the average number of examples per class is low, label sorting is similar to a random sort. however, as the number of examples grows, label sorting begins to have a dramatic impact on performance. figure 5 shows the performance of llama2-32k on clinic-150 with and without label sorting. as the number of examples in-context increases, the penalty for input sorting increases as 10 0 10 1 10 2 10 3 number of examples in-context 0 20 40 60 80 100 accuracy original order sorted labels figure 5: by contrast, sorting examples by label has an increasingly negative impact on performance in longer context regimes. results on llama2-32k with clinic-150. tasks where long context does not help concurrently to our work, @xcite identify a set of tasks where long context is not uniformly helpful. however, we observe that the tasks that show this trend either have near-0 performance at short demonstration lengths or also display an inverse performance trend on the short context scale (e.g. for tacred @xcite , we observe that performance decreases from 1 to 10 total demonstrations; on discovery @xcite , we observe that performance is near-zero at all demonstration lengths and decreases from 5 to 10 total demonstrations). while these are important failure modes of language models, we restrict our analysis to tasks without these confounding issues. in banking-77, the one dataset that our works share, both papers observe similar trends of improved performance with additional context. with respect to-that is, the same demonstration may have a different impact on the prediction if it is the only demonstration in-context versus if it is the 90th demonstration to appear in-context, because the demonstration will be better-contextualized. we hypothesize that long-context icl is primarily beneficial because of retrieval in-context-the idea that attention is used to select examples, rather than the model aggregating a complex decision boundary by better contextualizing across examples. this means that the primary benefit of longcontext icl is the number of demonstrations, not the quality of contextualization that each individual demonstration receives. if this is the case, then there should exist some sparse attention pattern that severely restricts long-range attention between demonstrations, but has limited or no impact on icl performance. 8 in all cases, the current test example can attend back to all demonstrations, in the same style as @xcite . ## conclusion we have demonstrated that icl with large demonstration sets can be surprisingly effective, and shed light on a few surprising properties in its behavior. namely, long-context icl exhibits a reduced dependence on example selection, relatively stable performance with respect to example order, and performance often approaching or exceeding parameter-efficient finetuning on the same data, all properties that make this an appealing option for a variety of tasks. we have also shown that long-context icl's effectiveness is largely due to retrieval from the long context during prediction, rather than cross-attention within the large demonstration set during encoding. our work also highlights that our understanding of icl remains incomplete. though much work has studied the potential mechanisms behind icl, these works have largely focused on simple tasks with small (< 10 examples) demonstration sets; as our work demonstrates that icl's properties shift in the long context regime, more work is necessary to validate hypotheses about icl at larger scales. while prior work has focused on two strategies for performing inference on a new task-either finetuning on task-specific data or selecting a subset of that data to use in-context-our results points to a potential third paradigm: adapting the model to fit as much of that data in-context as possible, caching and reusing the encoding of the long demonstration set. while finetuning with full datasets is still a powerful option if the data vastly exceeds the context length, our results suggest that longcontext icl is an effective alternative. icl trades finetuning-time cost for increased inference-time compute, and increasing the amount of inferencetime compute by using more examples in-context is an effective strategy to improve performance. as the effectiveness and efficiency of long context models continue to increase, we believe longcontext icl will be a powerful tool for many tasks. 12127 our work focuses on open-source models (and predominately on the llama-2 family); more work is necessary to establish whether this trend holds across model families, although we are encouraged by the strong results @xcite observed for many-shot icl on gemini 1.5. additionally, we do not consider other non-lora peft methodologies; it is possible that some of these methodologies may outperform in-context learning. finally, we focus primarily on classification tasks, and care should be taken in generalizing to new tasks.