Batch- ICL : Effective, Efficient, and Order-Agnostic In-Context Learning (2024)

Abstract: In this paper, by treating in-context learning (ICL) as a meta-optimization process, we explain why LLMs are sensitive to the order of ICL examples. This understanding leads us to the development of Batch-ICL, an effective, efficient, and order-agnostic inference algorithm for ICL. Differing from the standard N-shot learning approach, Batch-ICL employs N separate 1-shot forward computations and aggregates the resulting meta-gradients. These aggregated meta-gradients are then applied to the forward computation of a zero-shot query to generate the final prediction. This batch processing approach renders the LLM agnostic to the order of ICL examples. Through extensive experiments and analysis, we demonstrate that Batch-ICL consistently outperforms most permutations of ICL examples. In some cases, it even exceeds the performance of the best order for standard ICL, all while reducing the computational resources required. Furthermore, we develop a novel variant of Batch-ICL featuring multiple “epochs” of meta-optimization. This variant implicitly explores permutations of ICL examples, further enhancing ICL performance.

## understanding in-context learning from a meta-optimization perspective many works @xcite @xcite have demonstrated the similarity between the linear attention and the linear layers optimized by gradient descent. in this section, we briefly review this similarity and then introduce the in-context learning from a meta-optimization perspective. ## method analysis we delve into various aspects of batch-icl, including the impact of shot number (n ), aggregation layer index (k), the impact of the order of icl examples, and the optimization "epochs." this exploration aims to provide a deeper insight into both the efficacy of batch-icl and the icl itself. ## related works in addition to pcw we compared in §4, our study is also related to the findings of @xcite and @xcite . these works discovered that in few-shot learning tasks, the representation in deep layers carries the task information. @xcite focused on identifying and aggregating the outputs of induction heads @xcite . they emphasized the content analysis of aggregated hidden states. in contrast, we focus on a simpler yet effective method for enhancing the icl and do not differentiate specific heads. when compared to @xcite , our method demonstrates greater efficacy and surpasses the original few-shot learning performance by using zero-shot learning, an achievement not realized by @xcite . theoretically, our work offers an explanation for the "training set compression" proposed by @xcite and elucidates why averaged attention activation conveys icl task information @xcite . ## conclusion from a meta-optimization perspective, we explain icl example orders' impact on performance. building on our insights, we introduce batch-icl, an efficient and effective algorithm for icl inference. batch-icl processes icl examples in batches, aggregating their meta-gradients. aggregated meta-gradients are then applied to a zero-shot forward computation for final predictions. due to the batch processing, batch-icl is agnostic to the order of icl examples, surpassing the average performance of all order permutations across various tasks, and supports much more examples. we expand batch-icl by developing multi-epoch variants that implicitly enumerate permutations of icl examples, which fosters better interaction between inputs and further improves our method.