Revisiting Demonstration Selection Strategies in In-Context Learning (2024)

Abstract: Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL), where a few examples are used to describe a task to the model. However, the performance of ICL varies significantly with the choice of demonstrations, and previous research usually focuses on the data aspect ignoring the model’s effect. In this work, we first revisit the factors contributing to this variance from the model aspect, and find that the demonstration choice is both data- and model-dependent. We further propose a conjecture that the performance of a demonstration positively correlates with its contribution to the model’s understanding of the test samples, and accordingly propose a data- and model-dependent demonstration selection method, TopK + ConE. Empirically, our method yields consistent improvements in both language understanding and generation tasks with different model scales. Further analyses confirm that, besides the generality and stability under different circumstances, our method provides a unified explanation for the effectiveness of previous methods. Code is publicly available at https://github.com/Romainpkq/revisit_demon_selection_in_ICL.

## introduction large language models (llms, @xcite have achieved widespread success across many nlp tasks @xcite @xcite due to their remarkable emergent abilities @xcite . one of the most exciting emergent abilities is in-context learning @xcite , which utilizes only a few input-output examples to help llms make better predictions @xcite . icl has shown its effectiveness in eliciting llms' advanced capabilities and has (almost) become a common practice in tackling complex tasks. however, prior work @xcite has found that icl is very sensitive to the choice of in-context examples and their order in the prompt, and even small changes can result in large variance @xcite . the sensitivity of icl motivates researchers to explore methods to identify stable and highperforming demonstrations. influenced by the success of leveraging a retrieval module to augment neural networks @xcite , the retrieval module has become a standard module in the icl framework for retrieval demonstrations from a dataset @xcite . extensive research has been conducted to search for demonstrations similar to the test samples @xcite @xcite . for example, @xcite proposed to select the samples that are closer to the test sample in the embedding space as in-context examples, and @xcite found that choosing the high word-overlap samples can also improve the icl performance. despite empirical success to some extent, the above methods usually only focus on the test data, overlooking the impact of models. to figure out what factors influence the choice of demonstrations, we revisit the performance of icl from the model aspect, and accordingly propose a conjecture to understand the effective demonstrations. specifically, we investigate icl performance across different retrieval modules and inference models in §2.1. experimental results show that the icl performance can largely vary with different models even with the same demonstrations (see figure 1 as an example), indicating that the choice of demonstration is not only dependent on test data but also on the retrieval modules and inference models. we further propose a corresponding conjecture that effective demonstrations are those that enhance the inference model's understanding of the test input, and the comparison results between shuffled test input and original test input demonstrate that the icl performance positively correlates with model's understanding of the test samples. based on the above conjectures, we accordingly propose a demonstration selection method, denoted as topk+cone. specifically, we initially employed the topk @xcite ) method to narrow down the pool of demonstration candidates, followed by ranking these candidates based on the conditional entropy (estimated by the model itself) of the test sample input. extensive experiments demonstrate the effectiveness of our method across different model scales. further analyses show the universality and robustness, and provide a unified view of why previous demonstration selection methods work. our contributions are summarized as follows: ## revisiting demonstrations selection while in-context learning (icl, @xcite has shown its impressive few-shot performance, recent work has found that llms are very sensitive to the selected examples leading to large variances in performance @xcite . although many advanced icl strategies @xcite @xcite have been proposed to select effective demonstrations, why these demonstrations work and what factors influence their selection have not been fully studied. in this section, we first explore the influencing factors to the demonstration selection and correspondingly propose a conjecture to understand the effective demonstrations. ## method based on the above conclusions, we propose a simple and effective data-and model-dependent demonstration selection method, named topk + cone. our method is based on the conjecture in section §2.2, which implies effective demonstrations excel in reducing the conditional entropy of the test input under the inference model. it is noteworthy that we compute the conditional entropy of the test input rather than labels. mathematically, we find the best demonstrations c * by solving the following optimization problem: figure 3 : the performance of different inference models with three randomly sampled demonstrations for sst-2 and sst-5 datasets. model1, model2, model3 represent gpt-j-6b, llama2-7b, and llama2-13b, respectively. the impact of various demonstrations varies depending on the specific inference models. the lower the h θ (x|c) is, the better the understanding is. the equation ( 1 ) can be reformulated as in the practical implementations, considering the huge search space generated by a large number of combinations, enumerating all combinations is infeasible. we adopt the selection-rerank framework proposed in @xcite . specifically, we first use the selection module to select the candidate demonstrations and then use our method to rank each candidate to get effective demonstrations. ## experimental setup models. we perform experiments across different sizes of models, including gpt2-xl (1.5b) @xcite , gpt-j-6b (6b) @xcite , llama2-7b (7b) and llama2-13b (13b) @xcite , which are decoder-only dense lms. we also conduct experiments on extensive alignment models, e.g., llama2-7b-chat and llama2-13b-chat @xcite , vicuna-7b, vicuna-13b and deepseek-7b-chat (deepseek-ai, 2024) to verify the generalizability of our approach. datasets. we conduct a systematic study across 7 natural language understanding (nlu) tasks, including binary, multi-class classification tasks (sst-2, sst-5 @xcite , cr, subj @xcite ) and natural language inference tasks: mnli @xcite and qnli @xcite . we also evaluate our method in 4 machine translation tasks, extracted from flores-200 @xcite dataset, which contains 1012 examples for each language. baselines. we mainly compare our method with five widely used methods that do not require additional training. evaluation metrics. we adopt different evaluation methods for different tasks. for classification, we report the performance with the accuracy. for the translation tasks, we adopt the mostly used language model-based metrics comet @xcite since they have demonstrated a high correlation with human evaluation and are resilient to domain shift. specifically, we use the referencebased metric comet-20 (wmt20-comet-da) and comet-22 (wmt22-comet-da) for evaluation, and use the default parameters of "comet-compare" for the significance test 1 . ## analysis to further demonstrate the effectiveness and generality of our method, we conduct further analyses on nlu tasks (with the gpt2-xl model) and nlg tasks (with llama2-7b). our cone method is complementary to previous approaches. to further explore the generality of our method, we combine cone with different selection methods, e.g. random and bm25, in binary and multi-choice classification tasks. the results in figure 6 (a , b ) show that cone can further significantly improve the baseline performance in different types of tasks. especially in sst-2 tasks with the random method, cone brings +7.5 score improvement, which indicates that cone is complementary to previous approaches and can further improve their performance. we can also notice that topk + cone achieves better performance compared with other methods, hence we choose topk as our selection method because of its simplicity and effectiveness. our method works for mix-domain demonstration pools. previous results have shown the superior performance of our method in single-domain demonstration pools. now, we evaluate the effectiveness of our method in mixed demonstration pools, which have demonstrations from different domains. specifically, we evaluate the performance of our method in three domains, e.g., e-commerce, news and social, with a mix-domain demonstrations pool in wmt22 translation task foot_0 . experimental results in our method works for aligned chat models. can achieve consistent improvement in different models, demonstrating that our method also works for instruction-tuned and safety-enhanced models. ## impact of hyperparameter in this section, we conduct ablation studies on the hyperparameters in our method. ## discussion whether our method can partially explain why previous icl methods work? intuitively, enhancing the model's understanding to test input is one of the reasons why previous methods work. to prove this, we calculate the conditional entropy of the test input with respect to previous baselines across three classification tasks. the results presented in figure 6 (c) show that the previous methods will also reduce the conditional entropy of test samples in all three tasks, which demonstrate that previous icl methods can also be explained by our conjecture. these results show the universality of our conjecture. whether our method is sensitive to the demonstration order? previous studies have proven that icl is very sensitive to the order of in-context examples @xcite . to explore the sensitivity of our methods for the order of in-context examples, we randomize the order of our chosen demonstrations on three classification tasks and compare the stability with random and topk methods. results on ## related work despite that large language models have shown their surprising zero-shot performance in various tasks, even including complex reason/ agent tasks @xcite @xcite . recent works show that icl can effectively elicit their capability and further improve llms' performance @xcite . besides effectiveness, icl may provide format guidance to alleviate the prompt bias during language model inference @xcite . 0 1 2 4 8 16 number of in-context examples 50 55 60 65 70 accuracy (a) random topk ours 0 1 2 4 8 16 number of in-context examples 55 60 65 70 75 accuracy (b) random topk ours 8 10 20 30 50 number of candidates 65.0 65.5 66.0 66.5 67.0 67.5 68.0 68.5 69.0 accuracy (c) topk ours figure 7: the average performance of ablation experiments. (a, b) impact of the number of in-context examples for gpt2-xl and llama2-7b; (c) impact of the number of candidates selected by the topk method. method sst-2 subj cr avg. var. avg. var. avg. var. however, the performance of icl is unstable @xcite , and the small change of in-context examples and their order can result in a large variance in performance. motivated by the instability of the icl performance, in-context example selection methods have been widely investigated. @xcite first propose a validation-free corpus-level method for determining the optimal order of in-context examples. however, they only investigate the influence of order without proposing how to better select in-context examples. inspired by the success of retrieval modules in augmenting neural networks, @xcite find examples that are close to each test sample in embedding space that can serve as a good choice for icl. following the finding of @xcite , @xcite subsequently extended their method by incorporating increased diversity in the selection of in-context examples. however, why these methods work is still unclear and the methods only consider the influence from the data aspect. unlike the data-dependent demonstration selection methods, model-dependent methods are rarely explored. @xcite proposed a demonstration rank method grounded in the minimum description length principle, which utilizes the inference model to select the optimal in-context example organization. however, their ranked in-context organizations are randomly sampled, which may limit their performance. @xcite proposed a novel framework to iteratively train dense retrievers to identify high-quality in-context examples for llms. however, they need additional training, which is costly for practitioners. furthermore, both methods neglected to investigate whether and how the inference model affects icl performance. on the other hand, although some previous methods @xcite @xcite have emphasized the significance of understanding the test samples, their primary emphasis lies in the confidence of test labels, neglecting that of test input. for instance, @xcite searches the demonstrations capable of losslessly compressing testing labels, and @xcite identify the in-domain demonstrations through the cross-entropy difference of test labels computed by the small model fine-tuned in demonstrations. while @xcite propose to reweight label anchors to improve icl performance. @xcite found that using perplexity could be a good heuristic for prompt selection, while the effect for icl has not been investigated. ## conclusion in this paper, we take the first step to investigate the factors that influence the choice of demonstrations in icl from the model perspective, and find that the demonstration selection is both data-and modeldependent. based on the findings, we conjecture that effective demonstrations can improve the inference model's understanding to test input, and correspondingly propose a data-and model-dependent selection method. empirical results suggest that our method can significantly outperform the previous icl method. further analysis confirms the generalization of our method and our approach can provide a unified explanation for previous studies.