I nfinite ICL : Breaking the Limit of Context Window Size via Long Short-term Memory Transformation (2025)

Abstract: In-context learning (ICL) is critical for large language models (LLMs), but its effectiveness is constrained by finite context windows, particularly in ultra-long contexts. To overcome this, we introduce **InfiniteICL**, a framework that parallels context and parameters in LLMs with short- and long-term memory in human cognitive systems, focusing on transforming temporary context knowledge into permanent parameter updates. This approach significantly reduces memory usage, maintains robust performance across varying input lengths, and theoretically enables infinite context integration through the principles of context knowledge elicitation, selection, and consolidation. Evaluations demonstrate that our method reduces context length by 90% while achieving 103% average performance of full-context prompting across fact recall, grounded reasoning, and skill acquisition tasks. When conducting sequential multi-turn transformations on complex, real-world contexts (with length up to 2M tokens), our approach surpasses full-context prompting while using only 0.4% of the original contexts. These findings highlight InfiniteICL’s potential to enhance the scalability and efficiency of LLMs by breaking the limitations of conventional context window sizes.

## introduction in-context learning (icl) has emerged as a cornerstone capability of large language models (llms), allowing training-free and user-friendly customization @xcite @xcite . this ability is critical for real-world deployment in scenarios such as deep research integrating web information and lifelong learning @xcite through user interactions. however, the effectiveness of icl is constrained by the finite context windows of the transformer architecture @xcite , typically 8k-128k tokens. this limitation manifests through two compounding factors: (i) the quadratic complexity scaling of attention mechanism and the linear growth of kv cache memory @xcite , which impose prohibitive hardware infrastructure requirements for long-context deployment, and (ii) diminishing performance returns observed in tasks requiring extended contexts, such as many-shot learning @xcite and cross-document reasoning @xcite . these constraints create a paradox: expanding context capacity inflates computational costs disproportionately while delivering marginal accuracy gains. in response to these challenges, we propose a novel perspective that frames the dichotomy between context and parameters in llms as analogous to short-term and long-term memory in human cognitive systems @xcite . specifically, we posit that contexts function as short-term memory, capturing transient information relevant to the current input, while parameters serve as longterm memory, encoding accumulated knowledge over time. building on this analogy, our approach focuses on transforming temporary context knowledge into permanent parameter updates, as depicted in figure 1 . this transformation draws inspiration from the human circadian rhythm, where daily ex-periences are consolidated into long-term memories during sleep @xcite . it is further supported by recent studies demonstrating that icl can be interpreted as meta-gradient updates @xcite . additionally, our method falls within the scope of test-time compute scaling @xcite @xcite , as it enhances model capabilities through strategic allocation of compute for test-time training. compared to conventional strategies of extending context windows, our approach offers three key advantages: (i) it reduces gpu memory usage by converting context into compact parameter updates; (ii) it potentially maintains robust performance across varied input lengths, mitigating the scaling issues of attention mechanisms; and (iii) it theoretically enables infinite context integration by continuously updating long-term memory, eliminating the need for ever-expanding context retention. to implement this cognitive-inspired paradigm, we develop a framework with three principled design choices: (i) context knowledge elicitation establishes a unified strategy for diverse scenarios through hybrid prompting -systematically steering the model to generate both task-specific interactions (e.g., summarization, multi-step reasoning chains) and open-ended contextual expansions. this process constructs a transfer set t that comprehensively encodes contextual knowledge. (ii) path selection optimizes t into t k by retaining topk interactions with maximum perplexity discrepancy between context-aware and context-free generations, prioritizing knowledge-critical pathways. (iii) memory consolidation transforms these temporary contextual insights into permanent model parameter updates through knowledge distillation. our framework is rigorously evaluated through a systematic protocol designed to comprehensively assess its effectiveness across diverse scenarios. we first examine our approach in the single transformation setting, where the model undergoes a onetime conversion of context into long-term memory. when transforming 90% of the context into parameter updates, our method maintains 84% performance for fact recall on natural questions, 117% in grounded reasoning (counterfact and mquake), and 98% in 300-shot icl (trec and nlu), all compared to full-context prompting. these results collectively yield an average performance recovery of 103% relative to full-context prompting, consistently outperforming state-of-the-art context com-pression methods, with average scores of 70.4 versus 10.4 for llmlingua-2 (pan et al., 2024) and 7.3 for snapkv @xcite . we further validate our framework in sequential transformation scenarios, where multi-turn parameter updates are conducted with consecutive contexts. on longbench v2 @xcite , our method achieves superior results to full-context prompting (up to 128k) while utilizing merely 0.4% of the original contexts. extended analysis reveals that existing context compression/distillation baselines exhibit progressive performance degradation as context lengths increase (e.g., relative performance declines range from 28% to 82% as context lengths grow from 8k to 64k tokens), with most baselines experiencing performance collapse at 2m tokens. in contrast, our method maintains largely stable performance throughout this length scaling trajectory. additionally, we carry out systematic ablations to study the choices and effects of the three components (context knowledge elicitation, selection, and consolidation) in our framework. ## related work context compression some research focuses on reducing input context length to ensure it remains within a manageable computational budget. @xcite ; @xcite employ summarization models to generate concise scripts. other approaches remove redundant tokens based on information entropy @xcite @xcite or downstream task performance @xcite , the latter requiring task-specific training. some works focus on learning soft task prompts that encodes contexts into trainable vectors @xcite @xcite @xcite . additionally, there is a line of work that compresses context from the kv perspective, such as reducing kv activations along the length dimension @xcite . although these methods effectively shorten input text to enhance efficiency, they do so at the cost of diminished context integrity, with the loss becoming more pronounced as input texts lengthen. context distillation some existing works have explored injecting knowledge in prompt to model parameters. however, many of these focus on short contexts @xcite @xcite . others tend to concentrate on specific types of context such as conversation history @xcite , sports documents @xcite , and task instructions @xcite , or they require additional parameters and depend on specialized pretraining @xcite , which limits their applicability. additionally, temp-lora @xcite addresses long-form text generation by storing context information through training the model on the prompt and previously generated content. in contrast, we aim to present a general approach that is applicable to a wide range of scenarios. our evaluation covers a variety of tasks, especially involving long contexts, to rigorously assess the effectiveness of context distillation. ## evaluation setup to validate the effectiveness of our approach, we perform controlled experiments under two circumstances: single transformation and sequential transformations. while the former isolates the efficacy of individual memory transformation, the latter assesses operational viability under real-world streaming contexts where inputs are gradually received and processed. ## conclusion this work parallels llms' context and parameters with human memory systems, establishing a framework to transform temporary context knowledge into permanent parameter updates through elicitation, path selection, and memory consolidation. in single transformation tasks, our method remarkably surpasses full-context prompting performance, achieving an average 103% recovery rate. further analysis reveals that baselines exhibit brittle performance under high compression ratios, whereas our method maintains robust performance, even maintaining 62.25 performance at extreme ratios where most baselines collapse to no-context performance. our framework's generalizability is further evidenced by its consistent performance across varying context lengths and task difficulties in sequential transformation tasks, highlighting its applicability in real-world scenarios.