Representative Demonstration Selection for In-Context Learning with Two-Stage Determinantal Point Process (2023)

Abstract: Although In-Context Learning has proven effective across a broad array of tasks, its efficiency is noticeably influenced by the selection of demonstrations. Existing methods tend to select different demonstrations for each test instance, which is time-consuming and poses limitations in practical scenarios. Therefore, this study aims to address the challenge of selecting a representative subset of in-context demonstrations that can effectively prompt different test instances in a specific task. We propose that this representative subset should be of high quality and diversity. Our empirical analyses confirm that demonstrations that meet these criteria can indeed bolster model performance. To satisfy these criteria, this paper further introduces a two-stage Determinantal Point Process (DPP) method designed to incorporate both quality and diversity in the process of demonstration selection, thereby obtaining representative in-context demonstrations. Through comprehensive experimentation, we have confirmed the efficacy of our proposed method, paving the way for more practical and effective In-Context Learning.

## introduction the emergence of large language models (llms) has swept various natural language processing (nlp) tasks and brought new research paradigms @xcite @xcite . in-context learning (icl) @xcite is the primary paradigm for leveraging llms, which enables the model to generalize rapidly from a few examples without parameter update. recent studies have shown that icl has significantly improved over the zero/few-shot setting and even surpassed the performance of full data fine-tuning @xcite . however, icl is insufficiently stable and robust in performance when compared to traditional finetuning, and exhibits sensitivity to numerous factors, including demonstration selection @xcite train set instance 1 … … retrieval-based in-context demonstration selection representative in-context demonstration selection test set train set test set instance 2 instance k instance 1 instance 2 instance k 2022; @xcite , demonstration formats @xcite , demonstration labels @xcite , and demonstration order @xcite . among these factors, demonstration selection is the most significant, as they provide only information about the task @xcite . therefore, selecting suitable demonstrations is necessary. existing demonstration selection methods usually adopt retrieval-based solutions, which aim to select different demonstrations for each test instance @xcite @xcite . however, these retrieval-based methods can be costly in practice. in detail, these methods can only perform inference on one sample at a time, which incurs high costs in terms of token usage, particularly when considering leveraging batch prompting @xcite . besides, performing demonstration selection for each testing example is obviously time-costing ( §4.3). based on this, as shown in figure 1 , this paper aims to select a representative demonstration subset from a pool of training examples for all test instances in a specific task rather than each test instance. to select a representative demonstration subset for a task, we argue that there are two criteria that should be satisfied: quality and diversity. firstly, in icl, the quality of an instance is defined as the degree that it can help the llm to make correct predictions. intuitively, high-quality instances can improve the llm's performance on as many test examples as possible. secondly, diversity means that the elements in the selected demonstration subset should be mutually dissimilar and represent the training set as comprehensively as possible. moreover, in this paper, we argue that diversity can be refined into semantic diversity and influence diversity. semantic diversity means that the selected examples should be diverse enough to cover more semantics or expressions in the given training set, which is consistent with existing representative subset selection in traditional deep learning @xcite . influence diversity means that each selected example should help the model correctly classify a diverse set of testing samples, enabling the total demonstration set to support the correct classification of more test instances for the given llm. pilot experiments ( §2) also verify that the demonstration subsets satisfying these criteria could improve icl performance. in this paper, we make the following efforts to meet the above criteria. to achieve the quality criterion, we resort to recent instance-based explanation methods @xcite . different from them, this paper extends existing instance-based explanation methods to icl and defines the influence score to measure the contribution of a training example to the correct classification of another example. to achieve the diversity criterion, this paper seeks the help of the determinantal point process (dpp), which is a probabilistic model that could maintain high diversity among different instances by pairwise repulsion between instances @xcite . moreover, to incorporate both high quality and diversity for demonstration selection, this paper further proposes a two-stage dpp selection method. in specific, in the first stage ( §3.2.1), we utilize the semantic information measure instance similarity and then leverage dpp to obtain a subset of instances with high semantic diversity. in the second stage ( §3.2.2), we first compute the quality score of each instance in this subset based on our defined influence scores ( §2.2), which could reduce a significant amount of meaningless inference time compared to computing the quality for each training example. then we utilize the obtained influence scores to measure influence similarity between instances and construct another dpp. besides, we also introduce the quality scores ( §2.2) of these examples to this dpp. as a result, both quality and influence diversity could be optimally introduced in the selection process. this two-stage dpp selection method enables the selected demonstration subset could not only cover more semantics in the training set but also help the given llm classify more examples with better performance. our contributions can be summarized as follows: ## pilot experiments this section explores whether the proposed criteria could improve the performance of icl. in specific, we measure the impact of three factors: semantic diversity, instance quality, and influence diversity. we choose sst-2 @xcite as the test dataset and conduct 4-shot experiments on gpt2xl (1.5b) @xcite , gpt-j (6b) @xcite and gpt-neox (20b) @xcite . ## method in §3.1, we illustrate the formulation of in-context learning and the background of the determinantal point process. then, we introduce the implementation of our method in §3.2, which could introduce the three factors in the whole selection process. ## conclusion this paper aims to address the challenge of selecting a representative demonstration subset and proposes two criteria (quality and diversity) that such demonstrations should satisfy. and we further propose a two-stage dpp method to incorporate both high quality and diversity in the selection process. extensive experimental results show the effectiveness of our method and the significant advantages compared to retrieval-based methods.