Effective In-Context Example Selection through Data Compression (2024)

Abstract: In-context learning has been extensively validated in large language models. However, the mechanism and selection strategy for in-context example selection, which is a crucial ingredient in this approach, lacks systematic and in-depth research. In this paper, we propose a data compression approach to the selection of in-context examples. We introduce a two-stage method that can effectively choose relevant examples and retain sufficient information about the training dataset within the in-context examples. Our method shows a significant improvement of an average of 5.90% across five different real-world datasets using four language models.

## introduction drawing inspiration from recent research that regards large language models (llms) as an efficient means of compressing pre-training datasets, and the notion that in-context learning (icl) can be seen as fine-tuning on example datasets @xcite , we assume that llms can achieve data re-compression through icl. in other words, an effective training dataset compression method can aid in the selection of in-context examples. looking at the matter from another perspective, it is evident that fine-tuning the entire dataset would yield the best results. however, in the case of icl, we typically choose only a few examples as llm prompts due to the limitations of input window length. by employing data compression techniques, we can ensure that the majority of data information is preserved in the in-context examples, which is also the aim of dataset pruning. based on the aforementioned analysis, we propose to utilize the influence function @xcite , which has exhibited efficacy in dataset pruning, to select examples for icl. however, recent studies on icl have revealed that the relevance between the query source and in-context examples is critical for icl. furthermore, the influence function requires the gradient of parameters, which is computationally expensive and inefficient. to tackle the aforementioned issues, we suggest a two-stage method (i 2 cl). firstly, relevant examples for the query input are recalled, which ensures the correlation between the examples and the query source. secondly, our meta-gradient-based influence function is utilized to calculate the influence score for each recalled example. finally, based on the influence score, in-context examples are selected from the recalled examples. notably, our framework compresses important information from the training set into the in-context examples, thereby enhancing the performance of icl. additionally, our framework is data-independent, relies solely on a small number of model parameters, and does not require the training of any additional models. numerous experiments indicate that i 2 cl shows a significant improvement of an average of 5.90% on five different real-world datasets using multiple language models. ## experiments in this section, we empirically verify the efficiency of our approach. the source code and all experiments have been shared at https://github.com/ ke-01/icl . ## conclusion in summary, inspired by llms and icl's potential, we devised a two-stage method using the influence function for optimal in-context example selection. our approach ensures relevance with the query source and efficiently determines influence scores. the result is an enhancement in icl performance, with our experiments validating our model's effectiveness. our framework stands out due to its data-independent nature and minimal re-4 874 liance on model parameters.