O pti S eq: Ordering Examples On-The-Fly for In-Context Learning (2025)

Abstract: Developers using LLMs and LLM-based agents in their applications have provided plenty of anecdotal evidencethat in-context-learning (ICL) is fragile. In this paper, we show that in addition to the quantity and quality of examples, the order in which the in-context examples are listed in the prompt affects the output of the LLM and, consequently, their performance. While prior work has explored improving ICL through dataset-dependent techniques, we introduce , a purely inference-time, dataset-free optimization method that efficiently determines the best example order. OptiSeq leverages log probabilities of LLM-generated outputs to systematically prune the search space of possible orderings and recommend the best order(s) by distinguishing orderings that yield high levels of accuracy and those that underperform. Extensive empirical evaluation on multiple LLMs, datasets, and prompts demonstrates that OptiSeq improves accuracy by 5.5 - 10.5 percentage points across multiple tasks.

## introduction the use of in-context learning (icl) with large language models (llms) has become a popular approach to achieve impressive performance in many nlp tasks @xcite . in icl, models are prompted during inference with task-specific examples that help condition the generated output. unlike fine-tuning, it does not require updates to the model parameters, which offers many benefits with ever-increasing model sizes and capabilities @xcite . it has been shown that prompting llms without finetuning is often sensitive to prompt design @xcite . in particular, the quality, quantity, and permutation of examples can all significantly impact performance @xcite . to address this, previous work has primarily focused on selecting high-quality examples from a candidate set @xcite and determining the optimal number of these examples @xcite . existing solutions to mitigate prompt sensitivity caused by example ordering at inference-time are limited. figure 1 illustrates this using an api sequence generation task on the toolbench dataset @xcite . given three in-context examples, llm predictions vary significantly across the six possible orderings, with only one specific order (order 2) yielding the correct answer. this variability in precision and recall underscores how reordering examples alters the input context, influencing token probabilities and ultimately affecting model performance. most prior approaches rely on a precomputed strategy that assumes access to a predefined set of examples and a fixed label space @xcite . however, our setting is inherently online, requiring dynamic ordering decisions at inference time without training or validation data. since orderings cannot be precomputed, the limited number of examples further complicates the problem. a common approach to selecting examples at inference-time is to generate embeddings of candidate examples using a model like sentence-bert @xcite and retrieve the top-k most similar examples for a given test instance, ranking them based on distance or similarity. however, there is a distinction between ranking examples (determining how relevant they are to our test case) and ordering them (deciding how to arrange them in the prompt). while finding relevant examples through ranking is valuable, it does not tell us the best way to order them in the prompt. furthermore, top-k is dependent on the quality of embeddings and can lead to suboptimal performance if the distances are too close. recent efforts leverage additional in-domain validation datasets to perform offline evaluation of different orders, which is often not feasible in real-world scenarios where limited data is available @xcite . they also develop mutual information-or entropybased heuristics @xcite @xcite on these validation datasets, but are computationally limited to tasks like singleclass classification assuming label-balanced tasks and do not generalize to generation tasks. moreover, as we show in section 2, the optimal order of the examples varies for the test samples within and between tasks and between different llms, finding and selecting this optimal order is very challenging in production settings. an effective real-world solution needs to be (a) non-reliant on the availability of additional validation/training data, (b) generalizable to different tasks, and (c) performant across different llms and number of examples. in this paper, we introduce optiseq, a novel approach for selecting the optimal order of in-context examples at run-time and make the following contributions: ## sensitivity of icl to example ordering in this section, we present an analysis of the impact of in-context example ordering on performance under inference-time settings. naive icl fails to distinguish between correct and incorrect outputs. figure 2 illustrates an example from the toolbench dataset, where naive icl is applied using the standard prompt structure. we evaluate all six in-context example permutations using llama-3-8b-instruct and compare their generated api sequences against the ground truth. orders 1, 5, and 6 produce correct sequences, while orders 2, 3, and 4 fail. to analyze these failures, we compute the logarithmic probabilities of each output sequence as given by the llm. in figure 2 we can see naive icl assigns similar log probabilities to both correct and incorrect sequences across all orders. ideally, an llm should assign higher probabilities to correct sequences, effectively distinguishing them from incorrect ones. choose the appropriate sequence of apis using their descriptions that satisfy the provided utterance [list of available apis and their descriptions] example #1: …. example #2: …. 3 in-context @xmath2: …. utterance: my company is hosting a charity event and needs items for a gift auction. can you find popular, highly-rated products in the id region with details on price, image, and seller location? • order 1: shopeeapi.search.products, shopeeapi.product.details • order 2: realtor_data_api.realtorpropertylist, asin_data.product • order 3: realtor_data_api.realtorpropertylist, demo.get_product • order 4: realtor_data_api.realtorpropertylist, asin_data.product • order 5: shopeeapi.search.products, shopeeapi.product.details • order 6: shopeeapi.search.products, shopeeapi.product.details optimal order varies across instances and models. we evaluate six example orderings using three ag news @xcite articles as in-context examples across three models. figure 3 shows that optimal ordering is not universal-it varies across test instances and models. a fixed order fails to generalize, necessitating an adaptive approach without relying on training data or precomputed orderings. moreover, an order that performs well for one model (e.g., llama-3-8b-instruct) may underperform for another (e.g., granite-20b-code-instruct), highlighting the need for a model-aware, instancespecific ordering strategy without assuming dataset-wide fairness or requiring labeled data. impact of choosing the wrong order can be significant depending on the dataset or task. we evaluate five datasets (section 4.1) across two tasks-api sequence generation and text classification-by measuring accuracy variations across all possible example orderings in a three-shot setting. figure 4 demonstrates that example ordering plays a crucial role in performance, with accuracy fluctuating by ≈ 12 percentage points for api generation trrobhqfk rhvwgpt ag_news sst-5 trec 'dwdvhw @xmath0ffxudf\ 7rro%hqfkjudqlwhefrghlqvwuxfw 5dqgrp 7rs. 2swl6htvkrw vkrw vkrw vkrw @xmath1ffxudf\ 5hvw*37judqlwhefrghlqvwuxfw 5dqgrp 7rs. 2swl6htvkrw vkrw vkrw vkrw $ffxudf\ 5hvw*37pl[wudo[elqvwuxfw 5dqgrp 7rs. 2swl6htvkrw ing a random order and top-k. on average, for the api sequence generation task, optiseq performs better than random and top-k selection by 5.07% points and 2.1% points for classification task (shown in figure 12 ). adding more examples does not guarantee better performance, especially given context-length limitations. exceeding the model's input window can lead to prompt truncation and degraded performance at inference-time. @xcite demonstrates that more examples may introduce noise or redundancy, limiting generalization. optiseq offers a robust solution for icl by focusing on order optimization, which remains effective even when adding more examples is infeasible. ## related work methods for optimizing example ordering: @xcite formulates example ordering as an optimization problem. using label proportion, it improves accuracy and reduces model miscalibration across classification tasks. @xcite batch-icl aggregates meta-gradients from independent computations, making the model agnostic to example order while improving performance and reducing computational costs. @xcite proposes a select-then-rank framework for self-adaptive icl, achieving significant performance gains by dynamically optimizing example orders. inspired by how humans learn, @xcite gradually increases example complexity, improving instance and corpus-level performance through curriculum ordering. unlike batch or curriculum-based approaches, optiseq performs instance-specific optimization rather than applying a general rule. example selection and ranking techniques: @xcite selects diverse and informative examples using bertscore-recall, significantly outperforming independent ranking methods. demo @xcite identifies optimal example orders for individual instances through label fairness and content-free metrics. @xcite formulates example selection as a sequential process using beam search to optimize inter-relationships and diversity among examples. explora @xcite improves taskspecific exemplar selection for complex reasoning tasks by efficiently estimating scoring function parameters, reducing computational cost while enhancing performance. ceil @xcite models example selection as a subset selection problem using determinantal point processes and contrastive learning to optimize example interactions across diverse nlp tasks. optiseq goes beyond static or sequential ranking by dynamically testing every possible example order and using log probabilities to determine the best sequence. theoretical insights and adaptive strategies in icl @xcite demonstrates that dynamically adjusting the number of in-context examples improves task-specific performance over fixed hyperparameters. @xcite examines the limitations of icl for instruction-following tasks and identifies key parameters for alignment. @xcite employs adversarial learning to iteratively refine prompts, significantly improving performance across diverse tasks. optiseq avoids the complexity of adversarial learning or fine-tuning, providing an inference-time method for icl. ## conclusion this study highlights the impact of in-context example ordering on llm performance. optiseq significantly enhances accuracy by optimizing example orderings. by evaluating all possible orderings and selecting the highest confidence score based on input log probabilities, optiseq consistently improved accuracy by 5.5 to 10.5 percentage points over baselines. this improvement was observed across various generation and text classification tasks for three different model families with diverse parameter ranges, demonstrating optiseq's robustness and versatility in enhancing icl.