Instruction Induction: From Few Examples to Natural Language Task Descriptions (2023)

Abstract: Large language models are able to perform a task by conditioning on a few input-output demonstrations - a paradigm known as in-context learning. We show that language models can explicitly infer an underlying task from a few demonstrations by prompting them to generate a natural language instruction that fits the examples. To explore this ability, we introduce the instruction induction challenge, compile a dataset consisting of 24 tasks, and define a novel evaluation metric based on executing the generated instruction. We discover that, to a large extent, the ability to generate instructions does indeed emerge when using a model that is both large enough and aligned to follow instructions; InstructGPT achieves 65.7% of human performance in our execution-based metric, while the original GPT-3 model reaches only 9.8% of human performance. This surprising result suggests that instruction induction might be a viable learning paradigm in and of itself, where instead of fitting a set of latent continuous parameters to the data, one searches for the best description in the natural language hypothesis space.

## introduction large language models (lms) can perform unseen tasks by conditioning on a few labeled examples, effectively inferring the underlying tasks through a process known as in-context learning @xcite . however, task inference is implicit, and the ability of models to explicitly reason about it remains unexplored. in this work, we show that lms can explicitly describe an underlying task, in natural language, given a few labeled examples. we introduce the instruction induction challenge, in which a model is provided with a few inputoutput demonstrations, and is requested to generate a natural language instruction describing the connection between the input-output pairs. in our experiments, inducing instructions is done in a zeroshot manner by simply prompting the models to explain a small set of given demonstrations, as shown in figure 1 ; we do not perform fine-tuning or use any labeled instruction induction data. we examine instruction induction on 24 tasks, ranging from morphosyntactic tasks to style transfer and sentiment analysis. since our goal is to shed light on the phenomenon of instruction induction, we focus on tasks that have clear and simple instructions. as a basic evaluation protocol, we collect human annotations and use them as gold-standard references; the generated instructions are then compared to these references using bertscore @xcite . moreover, we suggest a novel evaluation metric for instruction induction: execution accuracy. the execution accuracy of a generated instruction is measured by testing whether lms can correctly perform the task in a zero-shot manner by using the generated instruction alone, without any demonstrations. our experiments reveal a surprising ability at generating correct instructions. the bestperforming model, instructgpt @xcite , achieves an average bertscore of 44.4, compared to human performance of 60.0; when measuring execution accuracy, the model reaches 43.6, with human-written instructions reaching 66.4. for some tasks, the model's performance is on par or even better than human performance. when qualitatively examining the generated instructions, we often observe accurate instructions, even for some of the more challenging tasks. for instance, in the task of formality style transfer, generated instructions include "translate the inputs into more formal language" and "use formal language". for semantic text similarity, the generated instructions include "for each input, rate the similarity of the two sentences on a scale of 0 to 5, with 5 being a perfect match" and "determine whether figure 1 : an example of instruction induction for the task of formality style transfer. left: the standard in-context learning setting; given five demonstrations, complete the sixth. right: instruction induction; the language model is prompted to generate a natural language instruction that describes the demonstrations. model completions are in blue, prompt templates are in pink. despite these impressive results, we find that this ability is currently unique to instructgpt @xcite , which is both very large (175b parameters) and was especially fine-tuned to follow instructions. ablations on smaller versions of in-structgpt as well as the original 175b-parameter gpt-3 @xcite yield dramatically weaker performance. these findings are in line with recent work showing that increasing model size unlocks new capabilities @xcite , and serves as additional evidence for the strength of instruction tuning @xcite @xcite , perhaps even pointing to the necessity of complementing standard next-word prediction with additional objectives. the fact that models can induce natural language instructions suggests that instruction induction may serve as a learning paradigm of its own, where the optimization goal is to find the best natural language description that fits the observations. in this ambitious view of instruction induction, natural language can function as the hypothesis space, and a model is required to learn a natural language rule describing the relation between inputs and outputs in the training examples, rather than a set of uninterpretable parameters. while we currently provide a proof-of-concept for that idea, extending it by grounding models in natural language has the immediate benefit of human interpretability, explainability, and verifiability, while potentially alleviating overfitting and other issues associated with spurious correlations. ## instruction induction we begin by formulating the task of instruction induction. given a sequence of n demonstrations {x k , y k } @xmath0,...,n} , the goal is to generate a single natural language instruction, such that for each x k , following the instruction results in y k . this format is similar to in-context learning @xcite , only here the desired output is an instruction describing the relation between the inputs and outputs of the demonstrations. we require models to perform this in a zero-shot setting, without any fine-tuning on labeled data. figure 1 illustrates the difference between standard in-context prompting and instruction-induction prompting. to elicit models to generate instructions, we consider prompts that would elicit humans to do so. we design a meta-prompt presenting instruction induction as a challenge puzzle and verify its clarity in a human study ( §3.3). the prompt is presented in figure 1 (right side, in pink). 2 while prior work already shows that large lms are often able to infer a latent task from a given set of demonstrations, this has been largely based on their ability to execute the task on a held-out exam-ple. instruction induction requires that the model describe the underlying task in natural language. ## data we evaluate on 24 tasks. example tasks are listed in ## evaluating generated instructions as a standard text generation metric, we report bertscore @xcite . however, the instruction induction challenge has a unique property, which does not usually hold for other text generation tasks: the instructions are executable. their correctness can therefore be measured directly by utilizing them as prompts. ## analysis to gain further insight into the successes and failures of instruction induction prompting, we manually analyze the model-generated instructions of 5 tasks. instructgpt obtains high, or close to human execution accuracy scores for three of these tasks (first letter, sentence similarity, pluralization). indeed, the instructions for both first letter and sentence similarity accurately describe the task. however, the instruction generated for pluralization is not entirely precise, since it dismisses other forms of pluralization such as -es, -ies, and irregulars. although the instruction only asks to add an "s", the execution model often ignores the specifics and produces the correct plural form; in one case, the input word was "life" and the output was "lives". while this particular instruction accounts for 24% of the induced instructions in the pluralization task, some predictions do explicitly mention pluralization, though not always accurately, e.g., "add -s to the end of each word to make it plural". for some tasks, instructgpt fails to produce accurate instructions, even if it is able to solve via in-context learning (see overall, gpt-3 did not exhibit any instruction induction abilities, although it did often phrase outputs in imperative language. one relatively common prediction was the generic instruction "write an output for every input". because these empty instructions are in the right format, they tend to have some overlap with the reference instructions, which inflates their bertscore. execution accuracy, on the other hand, is robust to this phenomenon, and typically assigns gpt-3's outputs very low scores. ## discussion this work demonstrates that large lms can not only infer new tasks based on a handful of demonstrations, but also describe them in natural language. we provide evidence of this ability on a diverse set of language tasks, and show that while instruction induction abilities are limited to a single state-ofthe-art model, this model does indeed approach human performance on about half the tasks. it is not unreasonable to assume that models in the near future will be even better at processing human-generated instructions, and it is therefore interesting to discuss the potential applications of instruction induction. in particular, we envision a use case in which instruction induction serves as a machine learning approach; instead of converting a dataset into a set of continuous parameters, we could produce a natural language instruction that best describes the data. grounding the model in concise natural language has the advantage of interpretability, and has the potential to solve fundamental issues pertaining to spurious correlations. while it is still too early to determine whether this approach is viable, we view it as an intriguing direction for future research. ## limitations since our primary goal is to study the phenomenon of instruction induction under lab conditions, we focus on tasks that have simple instructions. future work may extend instruction induction research by including tasks with more complex instructions. these tasks are expected to pose a greater evaluation challenge, especially when considering reference-based methods. evaluating through execution accuracy, however, may mitigate some of that challenge. additionally, only one model showed instruction induction abilities, i.e., textdavinci-002. the exact implementation details of the model and its training data are not publicly available, thus we are unable to investigate the reason behind the emergence of this ability. however, we note that our goal is to present the phenomenon of instruction induction and to raise the ambitious possibility of instruction induction as a learning paradigm. thus, our goal is not to focus on specific models but rather to shed light on this unexplored phenomenon. finally, we point to a limitation of the execution accuracy metric, namely assuming the existence of a good-enough instruction-tuned model. due to recent interest and progress in instruction tuning, we believe this to be a reasonable assumption.