Distilling Many-Shot In-Context Learning into a Cheat Sheet (2025)

Abstract: Recent advances in large language models (LLMs) enable effective in-context learning (ICL) with many-shot examples, but at the cost of high computational demand due to longer input tokens. To address this, we propose cheat-sheet ICL, which distills the information from many-shot ICL into a concise textual summary (cheat sheet) used as the context at inference time. Experiments on challenging reasoning tasks show that cheat-sheet ICL achieves comparable or better performance than many-shot ICL with far fewer tokens, and matches retrieval-based ICL without requiring test-time retrieval. These findings demonstrate that cheat-sheet ICL is a practical alternative for leveraging LLMs in downstream tasks.

## introduction in-context learning (icl; @xcite has emerged as a novel paradigm for leveraging large language models (llms) on downstream tasks @xcite . unlike the predominant finetuning approach, icl does not involve updating the model parameters. instead, it provides a few taskspecific examples, known as demonstrations, and a test input together, enabling llms to infer based on this context. due to context window limitations, icl has been typically used in few-shot settings. building on the extended context windows afforded by recent advancements in llms, @xcite and @xcite have demonstrated the superior performance of many-shot icl, wherein a larger number of demonstrations are provided, over the conventional few-shot setup. although many-shot icl requires a larger number of demonstrations, it retains key advantages over fine-tuning: it is training-free and can be applied to state-of-the-art proprietary models, which often limit or do not support fine-tuning. nevertheless, this paradigm introduces the increased computational costs associated with providing substantially longer input contexts. a common approach to efficient icl, given a large pool of demonstrations, is demonstration retrieval, in which demonstrations are selected based on their similarity to test inputs @xcite . while effective, this method requires a retrieval operation for each inference. in this study, we explore an alternative to existing approaches. our key idea is that llms, with their advanced language understanding, can extract the knowledge needed for a task as a textual summary. rather than requiring llms to infer hidden patterns from demonstrations at each inference, we propose distilling many-shot icl knowledge into an explicit textual format. figure 1 presents an overview of our method. we term our approach cheat-sheet icl, by analogy to how students summarize key points on a single sheet for exams. experimental results show that this remarkably simple method achieves performance comparable to, or even surpassing, many-shot icl on challenging reasoning tasks. furthermore, cheat sheets are interpretable and allow easy intervention for further improvements. moreover, cheat-sheet icl matches demonstration retrieval in performance, indicating its viability as an alternative. we consider this alternative paradigm a promising direction, and anticipate further improvements as llms' language understanding advances. ## conclusion we introduced cheat-sheet icl, which uses concise textual summaries distilled from many-shot demonstrations to leverage llms. this approach matched or exceeded the performance of many-shot icl and retrieval methods on challenging reasoning tasks, while remaining more efficient and interpretable. we believe that cheat-sheet icl provides a simple and practical alternative for leveraging llms.