Focused Large Language Models are Stable Many-Shot Learners (2024)

Abstract: In-Context Learning (ICL) enables large language models (LLMs) to achieve rapid task adaptation by learning from demonstrations. With the increase in available context length of LLMs, recent experiments have shown that the performance of ICL does not necessarily scale well in many-shot (demonstration) settings. We hypothesize that the reason lies in more demonstrations dispersing the model attention from the query, hindering its understanding of key content, which we validate both theoretically and experimentally. Inspired by how humans learn from examples, we propose a training-free method FocusICL, which conducts triviality filtering to avoid attention being diverted by unimportant contents at token-level and operates hierarchical attention to further ensure sufficient attention towards current query at demonstration-level. We also design an efficient hyperparameter searching strategy for FocusICL based on model perplexity of demonstrations. Comprehensive experiments validate that FocusICL achieves an average performance improvement of 5.2% over vanilla ICL and scales well with many-shot demonstrations.

## introduction the rapid development of large language models (llms) has facilitated the emergence and enhancement of their in-context learning (icl) abilities @xcite . as a trainingfree method, icl can achieve fast model adaptation on specific tasks based on several demonstrations prefixed to the query, formally denoted as icl(response|demos, query). intuitively, more demonstrations can help llms better understand the task and increase the likelihood of finding demonstrations that aid in responding queries, thus leading to better performance. theoretically, a similar conclusion can be drawn. previous studies @xcite @xcite oswald to understand this gap, we revisit the derivation of @xcite that formally equates icl with finetuning and identify that their approximation of standard attention operation as linear attention operation will ignore the competition for attention between demonstrations and the query when generating the response. since this approximation is key to the equivalence of icl and finetuning, we hypothesize that the reason why icl does not adhere to the scaling law like finetuning is that more demonstrations can divert attention away from the query. inadequate attention and understanding of the query can naturally lead to inferior response. to verify our hypothesis, we first conduct experiments confirming that increasing the number of demonstrations does lead to a decrease in model attention towards queries (figure 1 ). we further experiment by adding blank spaces within the demonstrations and confirm that: the more blank spaces added, the more attention towards queries distracted by blanks, resulting in lower response accuracy (figure 2 ). inspired by the way humans benefit from ignoring irrelevant contents and integrating insights from multiple examples when solving problems, we propose focusicl to avoid the attention dispersion issue faced by icl. specifically, at the token-level, focusicl conducts triviality filtering by adaptively masking unimportant tokens of demonstrations based on attention distribution, allocating the attention to more important contents. at the demonstration-level, focusicl performs hierarchical attention mechanism by dividing demonstrations into multiple batches and respectively conducting intra-batch and inter-batch attention operations. the limited demonstration number within each batch ensures sufficient attention to the query, while inter-batch attention integrates the benefits from a larger number of demonstrations. we further introduce an efficient hyperparameter searching strategy for focusicl according to model perplexity of demonstrations. our experiments across three llms on five benchmarks confirm that focusicl achieves an average performance improvement of 5.2% over icl by avoiding attention dispersion, with lower inference overhead. this demonstrates the effectiveness, efficiency, and generalizability of focu-sicl. furthermore, we observe that focusicl achieves performance scaling with the number of demonstrations by maintaining attention on critical parts, making demonstration number a possible scaling direction for llm-based agi. finally, we propose a unified perspective to understand the divergent phenomena observed in previous studies, where more demonstrations lead to either improved @xcite or deteriorated @xcite performance in icl. based on experimental results, we conclude that the performance of icl initially benefits but subsequently suffers from more demonstrations. the weaker the model and the closer the relationship between samples, the later the sweet spot for the number of demonstrations occurs. our contributions are summarized as follows: ## revisiting in this section, we explore what impedes llms from becoming stable many-shot learners. ## methodology to mitigate the impact of llms' attention being dispersed by many-shot demonstrations, we propose focusicl. the core idea behind focusicl is to allocate model attention to more important contents at token-level by triviality filtering ( §4.1) and at demonstration-level by hierarchical attention ( §4.2), as shown in figure 3 . ## experiments centered around focusicl, we will empirically demonstrate its performance on different llms and tasks in §5.2, verify whether it can help llms scale well with demonstration number in §5.3, and delve into its working mechanism in §5.4. we also investigate the choice of hyperparameters in appendix §a.1. ## conclusions noticing that the performance of llms under many-shot icl does not consistently improve with more demonstrations, we analyze and validate the