Revisiting In-Context Learning with Long Context Language Models (2025)

Abstract: In-Context Learning (ICL) is a technique by which language models make predictions based on examples provided in their input context. Previously, their context window size imposed a limit on the number of examples that can be shown, making example selection techniques crucial for identifying the maximally effective set of examples. However, the recent advent of Long Context Language Models (LCLMs) has significantly increased the number of examples that can be included in context, raising an important question of whether ICL performance in a many-shot regime is still sensitive to the method of sample selection. To answer this, we revisit these approaches in the context of LCLMs through extensive experiments on 18 datasets spanning 4 tasks. Surprisingly, we observe that sophisticated example selection techniques do not yield significant improvements over a simple random sample selection method. Instead, we discover that the advent of LCLMs has fundamentally shifted the challenge of ICL from that of selecting the most effective examples to that of collecting sufficient examples to fill the context window. Specifically, in certain datasets, including all available examples does not fully utilize the context window; however, by augmenting the examples in context with a simple data augmentation approach, we substantially improve ICL performance by 5%.

## introduction in-context learning (icl) has emerged as a powerful paradigm in natural language processing that enables language models (lms) to learn, adapt, and generalize from examples provided within their input context, eliminating the need for extensive training and parameter updates @xcite @xcite . however, due to the limited context lengths of earlier lms (which accommodate only a few thousand tokens), much of previous icl work has focused on * this work was conducted during an internship at google. optimizing sample selection strategies @xcite @xcite @xcite . with the advent of long context language models (lclms), which are capable of processing over a million tokens in a single context window, these constraints are significantly relaxed as it enables including a large number of examples to be used in icl, known as many-shot icl @xcite . this expansion of context length raises an important question: do previous sample selection strategies, designed for shorter context windows in earlier lms, generalize to the many-shot icl regime? to answer this, we systematically revisit existing sample selection strategies by conducting extensive experiments across 18 datasets spanning diverse tasks (namely, classification, translation, summarization, and reasoning) with multiple lclms. our experiments include multiple types of sample selection methods: relevance, diversity, and difficultybased sample selection, as outlined in @xcite . from these experiments, we uncover novel and surprising findings: contrary to prevailing expectations that carefully selected icl demonstrations would yield performance improvements, they are similarly effective with a simple random selection approach, offering no statistically meaningful improvements in almost all cases (figure 1 ). an additional reason to prefer the naive sample selection approach is that it enables greater efficiency through key-value caching of in-context examples (as the same examples can be reused across multiple queries), unlike sophisticated sample selection methods where the examples vary for each sample. while the expanded context length in lclms allows us to focus less on selecting optimal subsets of examples, it introduces a new challenge: effectively utilizing this expanded capacity when the number of examples is limited. specifically, in scenarios where available data is sparse (such as r a n d o m r e le va n ce d iv e rs it y c u rr ic u lu m h a rd 0.29 0.30 0.31 0.32 performance with gemini pro averaged summarization results r a n d o m r e le va n ce d iv e rs it y c u rr ic u lu m h a rd a u g m e n ta ti o n 0.54 0.55 0.56 0.57 averaged translation results r a n d o m r e le va n ce d iv e rs it y c u rr ic u lu m h a rd a u g m e n ta ti o n 0.61 0.63 0.65 0.67 0.69 0.71 averaged reasoning results r a n d o m r e le va n ce d iv e rs it y c u rr ic u lu m h a rd a u g m e n ta ti o n 0.52 0.53 0.54 0.55 0.56 0.57 ## related work lclms the field of language modeling has witnessed remarkable advancements with language models (lms) @xcite @xcite . however, earlier lms were constrained by relatively short context windows, typically handling only a few thousand tokens at a time, which limits their applicability in advanced tasks requiring broader context comprehension, such as document-level summarization or complex reasoning @xcite . to address this, recent efforts have led to the development of lclms, designed to process much larger contexts, sometimes accommodating over a million tokens within a single prompt @xcite . to mention a few, models like longformer and bigbird @xcite incorporate sparse attention mechanisms to efficiently handle extended contexts without compromising on computational feasibility. also, longrope extends the context window of lms to 2m tokens by interpolating their specific positional embeddings @xcite . in-context learning in-context learning (icl) is a recent paradigm that enables language models to learn from examples provided within their input context and then perform given tasks @xcite @xcite . since its introduction, previous studies have concentrated on developing the strategies to optimize the quality and arrangement of in-context examples to maximize performance, especially given the limitations of early lms on context length. for example, these approaches include selecting examples that maximize relevance to the target query @xcite , ensuring diversity among examples to cover a range of possible cases @xcite , strategically ordering examples to improve model adaptation @xcite , and prioritizing examples by their ease of learning based on their difficultly @xcite . yet, as the context capacity expands with lclms, these conventional selection strategies warrant re-evaluation, particularly in manyshot settings; thus, we focus on revisiting them. ## conclusion we explored icl in the context of lclms, investigating whether traditional sample selection strategies remain effective in many-shot scenarios and observing that they offer minimal to zero performance gains over simple random selection. we also highlighted the emerging challenge of underutilized context in low-resource tasks due to limited example availability, and proposed a data augmentation strategy, which substantially boosts performance by increasing context utilization of lclms. lastly, we analyzed the behavior of lclm-enabled icl when operating with extremely long context and in the presence of noisy examples, and found that while performance improves with added examples, it plateaus and even declines when the context becomes too long, with increased vulnerability to noise in complex tasks. this suggests promising future directions in making lclms more robust to lengthy context and noise examples alongside the direction of extending their context length.