NICE : To Optimize In-Context Examples or Not? (2024)

Abstract: Recent work shows that in-context learning and optimization of in-context examples (ICE) can significantly improve the accuracy of large language models (LLMs) on a wide range of tasks, leading to an apparent consensus that ICE optimization is crucial for better performance. However, most of these studies assume a fixed or no instruction provided in the prompt. We challenge this consensus by investigating the necessity of optimizing ICE when task-specific instructions are provided and find that there are many tasks for which it yields diminishing returns. In particular, using a diverse set of tasks and a systematically created instruction set with gradually added details, we find that as the prompt instruction becomes more detailed, the returns on ICE optimization diminish. To characterize this behavior, we introduce a task-specific metric called Normalized Invariability to Choice of Examples (NICE) that quantifies the learnability of tasks from a given instruction, and provides a heuristic to help decide whether to optimize instructions or ICE for a new task. Given a task, the proposed metric can reliably predict the utility of optimizing ICE compared to using random ICE. Our code is available at [https://github.com/microsoft/nice-icl](https://github.com/microsoft/nice-icl).

## introduction with the scaling up of large language models (llms) in terms of model size and training data, recent work demonstrates their emergent in-context learning ability across a wide range of tasks @xcite @xcite . specifically, the in-context learning (icl) paradigm studies ability of an llm to perform a new task given a few example demonstrations. the selection of good in-context examples (ice) from a large pool of candidates is a non-trivial optimiza-* these authors contributed equally. in-context learning ability in llms has motivated a growing body of literature on how to select the ice to be added to an llm's input prompt @xcite @xcite . these works propose various ice selection techniques and demonstrate significant improvements in llm performance across a range of tasks, leading to an apparent consensus in the literature that ice optimization is critical for better icl performance of llms. however, with the advent of instruction-tuning @xcite @xcite , llms are expected to follow natural language instructions, making it possible to solve a task in a zero-shot manner, i.e., learning with just the instruction and no ice. as a result, a typical prompt to an llm contains both the instruction and ice. a natural question that follows is how instructions and ice interact and whether they can be jointly optimized. would ice optimization still matter once a detailed task instruction is given? more practically, given a task, is optimizing the instruction more effective or optimizing ice? using state-of-the-art llms, our main finding is that returns on ice optimization diminish as more detailed instructions are added to the input prompt. we find that a detailed instruction with randomly chosen ice surpasses or matches the accuracy of a prompt with ice selected using criteria from past work @xcite @xcite across diverse tasks. these tasks include sst-2 and sst-5 for 2 and 5-way sentiment classification respectively @xcite , mnli for natural language inference @xcite , trec for 6-way question classification @xcite , medqa @xcite and finqa for domainspecific question-answering @xcite . moreover, given a detailed instruction, even using ice with incorrect labels yields the same accuracy as using correct labels on most tasks. on the other hand, there exists a different class of tasks, especially generative tasks that expect the output to follow specific schema, where ice selection matters. for example, for tasks such as nl2bash for bash command generation @xcite , mtop @xcite for semantic parsing, and break @xcite for question decomposed meaning representation (qdmr) generation, a prompt with chosen examples outperforms a prompt containing a descriptive instruction with random ice. therefore, contrary to prior work, we argue that ice optimization per query may not always be the most effective strategy for improving performance, especially if the task instructions are much simpler to write (e.g., using our proposed template in section 3.1). to characterize the tasks where ice optimization may or may not matter, we introduce a metric called normalized invariability to choice of examples (nice) that measures, for a given task and an instruction, the invariability of an llm's task performance to ice. we partition the candidate pool of examples into a relatively small number of query-dependent bins (e.g., based on cosine similarity of examples to the given query), and compute a bin-wise score given by the task performance of an llm when the given task instruction is augmented with random ice picked only from a particular bin. our nice metric compares the maximum score against the average score over all bins. we show that this simple metric can effectively capture the dependence of tasks on ice optimization. schema-based generative tasks such as mtop and break have low nice values (<0.4) whereas all other tasks have high nice values (>0.85). further, the nice value increases when we include more detailed task instructions, and suggests diminishing returns of ice optimization for better instructions. as a practical implication, our work provides a method to decide the best option for improving an llm's task accuracy: ice optimization or improving the instruction (see fig. 1 ). to this end, we propose a template for creating a detailed instruction for any task and provide an efficient procedure to compute the nice metric. given the task and a template instruction, if the nice value is high, then it is more effective to improve the instruction and random ice suffice. otherwise, if the nice value is low, it is more effective to do ice optimization. to summarize, our main contributions are: • contrary to prior work, we show that ice optimization does not always lead to performance improvement for llms. across a range of tasks, we find that the performance of prompts improves with more detailed instructions, and this improvement remains constant even when the true labels in ice are replaced with random labels. ## problem formulation optimizing ice amounts to selecting the best k examples from a given pool of candidate examples that should be included in the prompt. however, optimizing the instruction is not a straightforward optimization because the space of possible instruc-tions can be extremely large. hence, we first provide a instruction template covering five types of instructions that we consider in this paper, and then define our research questions. ## nice: measuring invariability to ice in addition to answering the research questions above, we provide a practical metric to assess the effectiveness of optimizing ice. that is, to decide whether to optimize instruction or in-context examples, we develop a task-specific metric for a given instruction that measures the invariability of task performance to ice selection. we assume that each task is associated with a training set from which ice are selected for a given query. given a new query, we assume access to a scoring function f of the candidate examples and the query. to study invariance to ice selection, for each query in the test set, we partition available training data of candidate examples using the function f into multiple bins. subsequently, for each bin, given a test sample of queries, we compute the model's performance using a pre-specified instruction i and random ice from the bin for each query (and then averaging the resultant accuracy). this yields the score s for a bin, ## conclusion our paper challenges the existing consensus that optimizing in-context examples (ice) is universally beneficial. through extensive experimentation on llms, we demonstrate that ice optimization yields diminishing returns in the presence of task-specific instructions. we introduce a novel and efficient metric, nice, to gauge the utility of optimizing ice for a task. nice measures the invariability of task performance with respect to the choice of ice. this helps in efficient utilization of data and compute resources. a nice score near 1 for a task implies minimal impact of ice choice on model performance, allowing the use of static ice that avoids query-dependent retrievals at inference time and improves computational efficiency. thus for certain tasks, we show that a given llm api budget may be more judiciously used to optimize the prompt instruction than to finetune available retriever models. while we focus on using nice to evaluate the effectiveness of ice optimization for a task, the same can also be used to compare different instructions by optimizing a weighted sum of nice scores (see more details in appendix a.2.1). for simplicity, we use uniform weights in this work.