Learning to Select In-Context Demonstration Preferred by Large Language Model (2025)

Abstract: In-context learning (ICL) enables large language models (LLMs) to adapt to new tasks during inference using only a few demonstrations. However, ICL performance is highly dependent on the selection of these demonstrations. Recent work explores retrieval-based methods for selecting query-specific demonstrations, but these approaches often rely on surrogate objectives such as metric learning, failing to directly optimize ICL performance. Consequently, they struggle to identify truly beneficial demonstrations. Moreover, their discriminative retrieval paradigm is ineffective when the candidate pool lacks sufficient high-quality demonstrations. To address these challenges, we propose GenICL, a novel generative preference learning framework that leverages LLM feedback to directly optimize demonstration selection for ICL. Experiments on 19 datasets across 11 task categories demonstrate that GenICL achieves superior performance than existing methods in selecting the most effective demonstrations, leading to better ICL performance.

## introduction large language models (llms) have demonstrated remarkable capabilities across a wide range of tasks. in-context learning (icl), introduced by @xcite , enables llms to perform tasks with only a few examples as demonstration, without requiring parameter updates @xcite . although icl is promising, the selection of incontext demonstrations is crucial, as it significantly impacts llm performance @xcite . many existing approaches rely on retrieval-based methods, either using off-the-shelf figure 1 : the distribution of the useful example ratio across test sets from different datasets. the x-axis represents the ratio # useful examples # total examples , where a 'useful example' is one that helps the llm generate an accurate output with in-context learning. the ratio remains low for most test queries, i.e., the majority of demonstration examples are ineffective for the llm. however, these methods face several challenges. the most critical issue is the misalignment between the surrogate learning objective of the retriever and the intrinsic optimization goal of icl. specifically, a discriminative model, trained with a metric learning objective to approximate the relevance score between a demonstration candidate and a given query, does not necessarily indicate the candidate's effectiveness as an in-context demonstration for an llm. furthermore, the scarcity of effective demonstration candidates in the retrieval pool poses another challenge for training a discriminative retriever. figure 1 illustrates the distribution of test queries regarding the ratio of useful examples that help the llm achieve accurate output using icl. the figure shows that most candidates are ineffective for most queries, making retriever optimization particularly challenging. this issue is further exacerbated by the properties of hard labels for llm feedback, as noted by @xcite . to address these challenges, we propose genicl, a novel generative preference learning framework that directly optimizes demonstration selection for icl using llm feedback. specifically, we reformulate icl as a generative bayesian optimization problem, introducing a latent variable to bridge demonstration selection and llm inference. genicl then optimizes this objective through preference learning, which focuses more on the relative effectiveness between demonstration samples, allowing it to capture finer-grained information in scenarios where effective demonstrations are scarce. this optimization procedure on demonstration selection more closely aligns with the intrinsic objective of icl. through this approach, genicl can better capture the llm's demonstration preference and identify the truly effective demonstrations (preferred by the llm). to validate the effectiveness of genicl, we conducted experiments on a diverse set of datasets covering classification, multiple-choice, and generation tasks. the results demonstrate that genicl achieves superior performance compared to existing methods. our contributions can be summarized as follows: