Active Example Selection for In-Context Learning (2022)

Abstract: With a handful of demonstration examples, large-scale language models demonstrate strong capability to perform various tasks by in-context learning from these examples, without any fine-tuning. We demonstrate that in-context learning performance can be highly unstable across samples of examples, indicating the idiosyncrasies of how language models acquire information. We formulate example selection for in-context learning as a sequential decision problem, and propose a reinforcement learning algorithm for identifying generalizable policies to select demonstration examples. For GPT-2, our learned policies demonstrate strong abilities of generalizing to unseen tasks in training, with a 5.8% improvement on average. Examples selected from our learned policies can even achieve a small improvement on GPT-3 Ada. However, the improvement diminishes on larger GPT-3 models, suggesting emerging capabilities of large language models.

## introduction large language models demonstrate the capability to learn from just a few examples @xcite @xcite . the possibility to train a model without any parameter update has inspired excitement about the in-context learning paradigm. intuitively, high in-context learning performance should require carefully chosen demonstration examples, but a recent line of work suggests otherwise -that demonstration examples are not as important as we expected, and that few-shot performance can be largely attributed to the model's zeroshot learning capacity @xcite , across gpt-2 and gpt-3. this insight is corroborated by a parallel line of work that brings significant improvements to in-context learning performance without example selection, for example, by reordering randomly selected examples and using calibration @xcite @xcite . another notable approach is to use best-of-n sampling, which requires a labeled set for validation @xcite . our contribution in this paper is twofold. first, we revisit the effect of example selection on incontext learning. we show that even with reordering and calibration, we still observe a large variance across sets of demonstration examples, especially for gpt-2, while calibration reduces the variance for gpt-3 models. the high variance needs further investigation, as we take it as evidence that large language models are still not capable of efficiently and reliably acquire new information in-context. understanding what makes good demonstration examples sheds some light on the mechanisms that large language models use to process information. second, we seek to discover general trends in example selection for in-context learning across different tasks. concretely, we use reinforcement learning to optimize example selection as sequential decision making problem. we argue that active example selection from unlabeled datasets is the most appropriate setting for in-context learning because fine-tuning with an existing labeled set leads to great performance with low variance. for gpt-2, we validate our learned policy on a seen task with labeled dataset and observe a 12.1% improvement over a max-entropy active learning baseline. moreover, our learned policy is able to generalize to new tasks with 5.8% improvement, suggesting that the policy is able to capture systematic biases in how gpt-2 acquires information. examples selected from our learned policies can even achieve a small improvement on gpt-3 ada. however, the improvement diminishes on larger gpt-3 models. we provide further analyses to understand the properties of useful examples. overall, our work explores how large language models process information through the perspective of example selection and formulate active ex-ample selection as a sequential decision making problem. we investigate divergent behaviors between gpt-2 and gpt-3, which echoes the emerging abilities of large language models, and suggest that researchers in the nlp community should collectively build knowledge and research practice in the era of large language models. 1 ## the effect of example selection in this section, we demonstrate the instability of incontext learning performance due to the selection of demonstration examples. we further show that existing methods (e.g., calibration, reordering) are insufficient for addressing this stability for gpt-2. in comparison, the variance of gpt-3 models can be mitigated with calibration. ## active example selection by rl given a set of unlabeled examples, can we choose the right ones to be annotated as demonstration examples? in this section, we formulate the problem of active example selection for in-context learning. following the definition of in-context learning in §2.1, constructing a prompt for in-context learning boils down to choosing a sequence of demonstration examples. we emphasize that by selecting from unlabeled examples, our setup is analogous to active learning, where we select examples to label. we think that this is the most appropriate setting for in-context learning because fine-tuning can lead to great performance with low variance if we already have a moderately-sized labeled set (e.g., 100 instances). as in-context learning uses a small number of examples, we formulate active example selection as a sequential decision making problem, where prompt is constructed by selecting and annotating one demonstration example at a time. we use a markov decision process (mdp) to formalize the problem, discuss our design of the reward function, and introduce our solution to example selection using reinforcement learning (rl). ## results in this section, we investigate the performance of our learned policies for gpt-2. due to the significant costs of generating episodes, we only apply the policies learned from gpt-2 and examine direct transfer results on gpt-3. baselines, oracles and our method have access to the same underpinning calibrated gpt-2 model. ## related work our paper builds on top of prior work that uses rl to solve the active learning problem @xcite , and is made possible by the recent advances in pre-trained language models @xcite @xcite . in-context learning, the observation that lms @xcite @xcite can "learn" to perform a task when conditioned on a prompt. xie et al. (2022) explains the emergenece of in-context learning by inferring the shared latent concept among demonstration examples, while @xcite finds the success of in-context learning is largely independent of access to gold labels. a variety of issues with in-context learning is discovered, including surface form competition, the phenomenon that multiple words referring to the same concept fighting for probability mass @xcite , and sensitivity of lms due to changes in prompt @xcite , instruction @xcite , or ordering of demonstration examples @xcite . to optimize the performance of in-context learning, methods with varying levels of granularity are proposed. such methods include prompt tuning @xcite @xcite , and instruction optimization @xcite . @xcite approaches the example selection problem by searching for nearest neighbors of test examples in the embedding space, while @xcite uses a scoring lm for example retrieval. ## discussion inspired by @xcite , we adopt a q&a format to discuss the implications of our work. q: are gpt-2 results still relevant? a: we believe that it is relevant for three reasons. first, gpt-2 is public and economically feasible options for many researchers. our knowledge about gpt-2 is far from complete and expanding this understanding is useful on its own. second, in the long term, it is unclear that everyone will have access to large models or that it is appropriate to use the largest model available in every use case. models of moderate sizes are likely still useful depending on the use case. third, it is important to highlight the emerging abilities over different sizes of language models. by understanding the phase change, i.e., when emerging abilities happen, we will better understand the behavior of large-scale language models. that said, one should caution against making generalizing claims based on results from gpt-2, because the results may not generalize to gpt-3 @xcite . this is why we present negative results from gpt-3. differing results between gpt-2 and gpt-3 or more generally models of different sizes will be a reality in nlp for a while. it is important for the nlp community to collectively build knowledge about such differences and develop the future ecosystem of models. q: why did you not experiment with gpt-3-davinci? a: the goal of this work is twofold: 1) assessing the ability of large-scale language models to acquire new information and 2) exploring whether reinforcement learning can identify reliable strategies for actively selecting examples. our results are generally positive on gpt-2. meanwhile, we observe relatively small variance after calibration with gpt-3-babbage, so it does not seem economically sensible to experiment with even bigger models. q: why did you choose @xmath0 a: our experiments are limited by the context window of gpt-2 (1024 tokens) and gpt-3 (2048) tokens. using k beyond 4 would frequently leads to demonstration examples overflowing the token limit and need to be truncated. additionally, prior work @xcite shows diminishing improvements of in-context learning performance by adding the number of demonstration examples beyond 4. therefore, we believe experimenting with @xmath1. we are optimistic that our framework and method can generalize to different shots. ## conclusion in this work, we investigate how large language models acquire information through the perspective of example selection for in-context learning. in-context learning with gpt-2 and gpt-3 is sensitive to the selection of demonstration examples. in order to identify generalizable properties of useful demonstration examples, we study active example selection where unlabeled examples are iteratively selected, annotated, and added to the prompt. we use reinforcement learning to train policies for active example selection. the learned policy stablizes in-context learning and improves accuracy when we apply it to a new pool of unlabeled examples or even completely new tasks unseen during training for gpt-2. our analyses further reveal that properties of useful demonstration examples can deviate from human intuitions. examples selected from gpt-2 can still lead to a small improvement on gpt-3 ada, however, the gain diminishes on larger models (i.e., babbage and curie). our results highlight the challenges of generalization in the era of large-scale models due to their emerging capabilities. we believe that it is important for the nlp community to collectively build knowledge about such differences and develop the future ecosystem of models together.