I nstruct E val: Systematic Evaluation of Instruction Selection Methods (2024)

Abstract: In-context learning (ICL) performs tasks by prompting a large language model (LLM) using an instruction and a small set of annotated examples called demonstrations. Recent work has shown that precise details of the inputs used in the ICL prompt significantly impact performance, which has incentivized instruction selection algorithms. The effect of instruction-choice however is severely underexplored, with existing analyses restricted to shallow subsets of models and tasks, limiting the generalizability of their insights. We develop InstructEval, an ICL evaluation suite to conduct a thorough assessment of these techniques. The suite includes 13 open-sourced LLMs of varying scales from four model families, and covers nine tasks across three categories. Using the suite, we evaluate the relative performance of seven popular instruction selection methods over five metrics relevant to ICL. Our experiments reveal that using curated manually-written instructions or simple instructions without any task-specific descriptions often elicits superior ICL performance overall than that of automatic instruction-induction methods, pointing to a lack of generalizability among the latter. We release our evaluation suite (at https://github.com/princeton-nlp/InstructEval) for benchmarking instruction selection approaches and enabling more generalizable methods in this space.

## introduction one of the most effective insights in nlp research in recent years has been that large language models trained to perform next-token prediction show emergent in-context learning (icl) abilities @xcite @xcite . while the bulk of research interest has shifted away from task-specific models and towards creating "foundation models" to perform a variety of tasks using appropriately constructed 1 code: https://github.com/princeton-nlp/ instructeval prompts, the performance of icl remains sensitive to the precise details of prompt construction. prompt engineering remains critical for achieving optimal icl performance @xcite @xcite . in practice, icl typically involves prompting a language model using a concatenation of a taskspecific instruction, a short sequence of annotated in-context examples known as demonstrations, and a test example (figure 2 ). much of the research interest surrounding in-context learning has focused on understanding the optimal selection, ordering of demonstrations, and label-space choices @xcite @xcite @xcite @xcite . however, instruction choice remains a relatively underexplored aspect of prompt engineering despite its established significance @xcite on downstream performance. even among recent works exploring automatic instruction selection @xcite @xcite , the use of different evaluation protocols makes the comparison of their relative performances difficult. existing studies typically limit their analyses to specific models or tasks; for example, @xcite focus on a single model, and while @xcite consider multiple model scales, they all belong to a single model family. moreover, evaluations often span disparate task selections with minimal overlap and are primarily dominated by classification tasks, neglecting other task types like multiple-choice qa or generation. lastly, most previous works studying automatic instruction selection tend to emphasize zero-shot accuracy, overlooking other pertinent icl metrics such as fewshot accuracy and robustness measures. to address these issues, we build instructeval, an evaluation suite for the comprehensive evaluation of instruction selection methods. the suite covers a diverse collection of 13 open-sourced autoregressive llms from four model families and nine tasks spanning three task types. additionally, it also incorporates three accuracy metrics and two sensitivity metrics that are of interest to icl. we perform evaluations of seven popular instruction selection methods including trivial instruction baselines, manually curated instructions, and sophisticated automatic methods using our suite. overall, we find that the relative effectiveness of these approaches varies significantly across different models and task types. we discover that curated manually-written instructions and task-agnostic instructions can elicit better aggregated performance (over models) than automatically induced ones, highlighting the lack of generalizability of the latter. we also find that including instructions in few-shot prompts usually tends to hurt icl performance at the model scales we consider. our findings suggest that it may be optimal for icl practitioners to omit instructions in few-shot settings and use curated manually-written instructions in zero-shot settings, rather than contemporary automatic induction techniques that require substantial computation and hyperparameter tuning to achieve competitive performance. we release the evaluation suite we develop to aid the systematic study of even more questions regarding prompt engineering that we do not explicitly address in our work. ## related work in-context learning and existing benchmarks as language models have scaled, in-context learning has emerged as a popular paradigm and remains ubiquitous among several autoregressive llm families @xcite @xcite @xcite . benchmarks like bigbench @xcite and helm @xcite have been created for the holistic evaluation of these models. bigbench focuses on few-shot abilities of state-of-the-art large language models, while helm extends its evaluation to consider metrics like robustness and bias. however, these benchmarks focus on evaluating and ranking language models, and do not address the systematic evaluation of prompting methods. although contemporary work by @xcite also aims to perform a similar systematic analysis of prompting methods, they focus on simple probability-based prompt selection while we evaluate a broader range of methods including trivial instruction baselines, curated manually selected instructions, and sophisticated automated instruction selection. ## experimental setup we perform experiments evaluating 3 families of instruction selection methods (listed in ## results we tabulate the mean relative gain values over accuracy metrics in ## conclusion we conduct the broadest attempt to our knowledge, to systematically study the generalizability of popular instruction selection and induction methods for icl in llms. we find that simpler approaches such as using task-agnostic instructions, expert manual instructions, or even omitting instructions entirely tend to show good performance more consistently when evaluating across a wide variety of tasks and models. our work indicates the need for more systematic and consistent evaluations in the instruction induction space. to facilitate such analyses, we release the instructeval suite which provides coverage over 13 diverse autoregressive llms and 9 tasks spanning classification, multiplechoice qa, and generative qa.