Finding Support Examples for In-Context Learning (2023)

Abstract: In-context learning is a new learning paradigm where a language model observes a few examples and directly outputs the test input’s prediction. Previous works have shown that it is sensitive to the provided examples and randomly sampled examples probably cause inferior performance. In this paper, we propose finding “support examples” for in-context learning: Given a training dataset, it aims to select one permutation of a few examples, which can well characterize the task for in-context learning and thus lead to superior performance. Although for traditional gradient-based training, there are extensive methods to find a coreset from the entire dataset, they struggle to find important in-context examples, because in-context learning occurs in the language model’s forward process without gradients or parameter updates and thus has a significant gap with traditional training. Additionally, the strong dependence among in-context examples makes it an NP-hard combinatorial optimization problem and enumerating all permutations is infeasible. Hence we propose **LENS**, a fi**L**ter-th**EN**-**S**earch method to tackle this challenge in two stages: irst we filter the dataset to obtain individually informative in-context examples. Specifically, we propose a novel metric, InfoScore, to evaluate the example’s in-context informativeness based on the language model’s feedback, and further propose a progressive filtering process to filter out uninformative examples. Then we propose diversity-guided example search which iteratively refines and evaluates the selected example permutations, to find examples that fully depict the task. The experimental results show that LENS significantly outperforms a wide range of baselines and further analyses show that each component contribute critically to the improvements and shed light on the principles of supporting examples and in-context learning.

## introduction in-context learning (icl) is a new paradigm using the language model (lm) to perform many nlp tasks @xcite @xcite . in icl, by conditioning on a few training examples, lm can directly output the prediction of a given test input without parameter updates. restricted by lm's max input length, it is typical to randomly sample a small set of examples from the entire dataset for in-context learning @xcite . however, in-context learning is sensitive to the provided examples and randomly sampled in-context examples show significant instability and probably cause inferior performance @xcite . in this paper, we propose to select a small list of examples that are informative and representative for the entire dataset as in-context examples. inspired by the traditional machine learning method, support vector machine (svm) @xcite , where a few support vectors are closest to the decision boundary and provide crucial discriminative information for svm, we name the selected examples for icl as support examples since they provide crucial task information for the lm and their quantity is usually limited, too. there is a similar problem in traditional gradientbased deep learning like fine-tuning @xcite , typically called coreset selection @xcite , which aims to select a set of representative training examples for the dataset to benefit many downstream scenarios like data-efficient learning @xcite , active learning @xcite , neural architecture search @xcite , etc. however, it is challenging for these coreset selection methods to select important in-examples acc the casting of raymond j. barry as the 'assassin' greatly enhances the quality ... it was @xcite 82.0 context examples because there is a significant discrepancy between traditional training and icl. as shown in figure 1 , the "learning" paradigms of model training and icl are highly different. traditional training depends on back-propagation's gradients to update parameters while icl occurs in lm's forward process without gradients and parameter updates. existing coreset selection methods are always coupled with the training procedure, i.e., they usually depend on gradients or run with the training procedure. for example, @xcite select informative examples by their gradients' norm. @xcite evaluate each example's importance by counting how many times it is forgotten, i.e., the example is misclassified after being correctly classified in the previous epoch. additionally, coreset selection methods mainly depend on the example's gradients as the feature of example selection @xcite @xcite . however, lm performs icl through inference, which does not rely on gradients or parameter updates. hence, the gap between gradient-based training and icl makes these methods struggle to effectively select informative examples for in-context learning. another challenge is the strong dependency among in-context examples. previous work @xcite shows that even the same example set with different orderings can result in drastically different performance from random-guess level to state-of-the-art. here we also conduct an additional case study to shed light on examples' combinatorial dependency in to tackle these challenges, we propose lens, a filter-then-search that finds support examples in two stages: in the first stage, we filter the dataset to obtain informative in-context examples individually. specifically, we propose infoscore to evaluate the example's in-context informativeness based on the lm's feedback, and further propose a progressive filtering process to filter out uninformative examples; in the second stage, we propose a diversity-guided example search method that iteratively refines and evaluates the selected examples to find support examples that can fully depict the task. we summarize our contributions as follows: ## background:in-context learning in this section, we introduce the definition of incontext learning. we focus on text classification's in-context learning using the causal language model @xcite . given a language model g, n examples {x i , y i } n @xmath0, the prediction of x test is generated as: as eq.( 1 ) shows, g receives the task's supervision only from the concatenated {x i , y i } n @xmath1. typically, n is limited by the max input length of g, so it is typical for researchers to randomly sample a small set of samples from the entire dataset d @xcite . however, icl is sensitive to the provided examples and random in-context examples show significant instability and probably cause inferior performance @xcite . in this paper, we focus on selecting a small list of support examples that are informative for the task and performant for in-context learning, from the entire dataset d. ## method the strong dependency among in-context examples makes selecting support examples essentially an np-hard combinatorial optimization problem. enumerating all combinations and evaluating them is infeasible due to the combinatorial explosion. in this section, we propose lens, a filter-then-search method to find support examples: 1. we first filter the training dataset to obtain informative examples individually, 2. then we search the example permutation that fully depicts the task from them. in this paper, we instantiate the two stages as a novel example metric with progressive filtering and diversity-guided example search, we leave the development of more powerful components as future work. we introduce these two stages below. ## conclusion in this paper, we propose a two-stage filter-thensearch method to find support examples for incontext learning from the annotated dataset: first we propose infoscore to select informative examples individually with a progressive filtering process. then we propose diversity-guided example search which iteratively refines and evaluates the selected examples, to find the example permutations that fully depict the task. the experimental results show that our method significantly outperforms extensive baselines, and further analyses show that each component contributes critically to the improvements and shed light on the principles of support examples and in-context learning.