What Makes a Good Order of Examples in In-Context Learning (2024)

Abstract: Although large language models (LLMs) have demonstrated impressive few-shot learning capabilities via in-context learning (ICL), ICL performance is known to be highly sensitive to the order of examples provided. To identify appropriate orders, recent studies propose heuristic methods to evaluate order performance using a set of unlabeled data. However, the requirement of in-domain data limits their utility in real-world scenarios where additional annotated data is challenging to acquire. Additionally, these dataset-based approaches are prone to being sub-optimal for a lack of consideration for individual differences. To address the problems, we first analyze the properties of performant example orders at both corpus level and instance level. Based on the analysis we propose **DEmO** to adaptively identify performant example order for each instance without extra data. DEmO works by filtering out a subset of orders featuring label fairness, then selecting the most influential order for each test instance. The employment of a content-free metric makes DEmO independent of in-domain data. Extensive experiments indicate the superiority of DEmO over a wide range of strong baselines. Further analysis validates the generalizability across various settings.

## introduction large language models (llms) demonstrate impressive abilities @xcite @xcite across a variety of natural language processing tasks through in-context learning (icl). in icl, provided with prompts containing several labeled examples, models are asked to make predictions on a new example @xcite . in contrast to traditional learning paradigms of full data fine-tuning, icl requires no model pa- † corresponding author. rameter update and has emerged as a prevailing paradigm for using llms. however, the performance of icl shows notable differences across various prompts @xcite . to enhance icl performance, a line of studies has predominantly focused on selecting highquality examples from a large dataset @xcite @xcite . nevertheless, access to a large in-domain dataset can be particularly challenging in certain fields such as medicine and finance @xcite . hence, recent studies have focused on organizing examples in a reasonable order since the order of icl can highly affect the performance @xcite . as illustrated in figure 1 , we conduct experiments on 4 classification task to show that the accuracy of different orders varies a lot across different models. one straightforward yet effective strategy is to directly evaluate the performance of different orders on a labeled validation set and select the most effective one. to reduce reliance on labeled data, several studies leverage heuristic metrics such as mutual information @xcite , global entropy and local entropy @xcite to evaluate the order performance on an unlabeled dataset. nevertheless, these methods still heavily rely on supplementary in-domain data for order assessment and fail in realistic few-shot scenarios, as defined by @xcite , where only a handful of examples are available. in addition, since these methods operate at the corpus level without considering differences among instances, they tend to be sub-optimal @xcite . in this paper, we aim to identify effective example orders for each test instance under realistic few-shot scenarios. initially, we explore a fundamental question: what constitutes a good example order for icl? we consider this question at both the corpus level and the instance level. we systematically evaluate the impact of corpus-level prediction distribution and instance-level influence on icl performance. specifically, we consider 50 random orders and evaluate them across four widely used text classification datasets to explore the correlations. pilot experiments reveal that a reasonable example order is supposed to ideally (1) exhibit label fairness at the corpus level and (2) facilitate influential prediction at the instance level. label fairness means that the demonstration examples will not bias models towards certain labels. influential prediction means that the test instance should be dominant in the final prediction of models. motivated by these two insights, we introduce a novel two-stage ordering framework named demo (dataset-free example ordering) that can be effortlessly implemented without extra data. to achieve label fairness while mitigating data dependency, a filtering stage is applied to pick out a candidate set of orders with balanced label distributions. in this process, a content-free input that replaces the test instance with a meaningless token is employed as a proxy for corpus-level prediction. then to produce influential prediction at the instance level, we adaptive select candidate order that maximizes the output influence of the test instance during inference. to verify the effectiveness, we conduct comprehensive experiments and analysis across a spectrum of established classification tasks. experimental results reveal that our framework consistently outperforms existing dataset-based methods and exhibits generalability across various settings. furthermore, our framework demonstrates the potential to be enhanced with access to a labeled validation set. to summarize, our contributions include: with a validation set available, our framework further boosts the performance by 2.1%. order sensitivity is observed universally across llms. with a fixed set of demonstration examples, the performance of different orders can vary from random guess to state-of-the-art (lu et al., 2022). recently, several studies have tried to determine performant orders. their methods can be roughly categorized by whether an in-domain dataset is required. typically, datasetbased approaches show superior performance to dataset-free methods by utilizing domain information but incur more computational overheads. dataset-free search: to counter the recency bias in prompts (i.e. llms tend to repeat the answer in the last example), liu et al. (2022) organize demonstration examples by their embedding space similarity to the test instance, placing more similar examples closer to the end of prompts. inspired by solomonoff's general theory of inference (solomonoff, 1964), wu et al. (2023) employs an information-theoretic strategy to minimize the code length required to transmit task labels. however, these methods are tailored to specific demonstration selection techniques and lack general applicability. dataset-based search: sorensen et al. (2022) opt for prompts with larger mutual information between the input and model output, fostering confident and diverse predictions. lu et al. (2022) define local entropy and global entropy metrics to sequence examples, aiming for balanced predictions across pseudo samples generated by the llm itself. in spite of the effectiveness of dataset-based methods, they exhibit notable drawbacks of high dependency on data and are limited in real-world scenarios. compared with them, our framework shows superior performance without the need for additional data. in contrast, our framework additionally incorporates corpus-level information to identify performant orders. in this section, we explore the properties of performant orders at the corpus level and instance level respectively. we conduct our pilot experiments on four widely used classification datasets: sst-2 @xcite , subj @xcite , trec @xcite , and agnews @xcite . following a standard 4-shot setting, we consider 50 random permutations and evaluate them on a validation set containing 256 instances. ## methodology based on the aforementioned insights in preliminary experiments, we propose a general two-stage ordering framework for icl in realistic few-shot settings. we start by employing a content-free entropy metric to filter out permutations that are prone to generating imbalanced corpus-level predictions, yielding a refined set of candidate permutations. subsequently, during the inference stage, we adaptively select the permutation with the highest influence for each test sample. figure 4 gives an overview of our framework. the full algorithm is demonstrated in algorithm 1.