Reframing Instructional Prompts to GPT k’s Language (2022)

Abstract: What kinds of instructional prompts are easier to follow for Language Models (LMs)? We study this question by conducting extensive empirical analysis that shed light on important features of successful instructional prompts. Specifically, we study several classes of reframing techniques for manual reformulation of prompts into more effective ones. Some examples include decomposing a complex task instruction into multiple simpler tasks or itemizing instructions into sequential steps. Our experiments compare the zero-shot and few-shot performance of LMs prompted with reframed instructions on 12 NLP tasks across 6 categories. Compared with original instructions, our reframed instructions lead to significant improvements across LMs with different sizes. For example, the same reframed prompts boost few-shot performance of GPT3-series and GPT2-series by 12.5% and 6.7% respectively averaged over all tasks. Furthermore, reframed instructions reduce the number of examples required to prompt LMs in the few-shot setting. We hope these empirically-driven techniques will pave the way towards more effective future prompting algorithms.

## introduction prompting language models (lms) @xcite has made nlp modules accessible to nonexpert users through plain text instructions foot_0 of nlp tasks. such task instructions written by non-expert users are often long and contain abstract descriptions which are not easy to follow for lms, as evident by their low performance @xcite . however, it is not quite clear whether this is due to the inherent difficulty of the target tasks or an artifact of the complex phrasing of their language instructions. you are given passages that contain mentions of names of people, places, or things. your job is to write questions that evaluate one's understanding of pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. ## related work our work is related to designing discrete prompts and tuning continuous prompts in recent literature. ## prompt reframing this section describes our reframing principles and then describes the guidelines to operationalize them. reframing principles are obtained by probing instructions of various tasks in the training split of natural instructions @xcite to understand different failure modes associated with prompting in gpt3. motivation from gpt3's failures we observe that gpt3 fails to follow instructions when it is provided with long prompts that often contain repeated information, abstract notions, analogies, complex statements requiring human commonsense and their domain knowledge (see examples in ## experimental setup dataset we evaluate the proposed reframing techniques on the evaluation tasks from natural instructions @xcite , which consists of 12 tasks categorized into 6 categories. following the original setup, we use rouge-l @xcite as the evaluation metric in our experiments. essential-terms @xcite models for evaluation we use various models of the gpt family: gpt2, gpt2large, gpt2xl, gpt3 and gpt3-instruct @xcite foot_3 and bart-base @xcite . we evaluate the models according to the following setups: gptk w/ raw instructions: we follow the setup of @xcite who experiment with gpt3instruct on their raw instructions. overall the prompts provided to the model consist of three segments (in this order): (a) task instructions, (b) examples (input and outputs) and (c) a new input for which we expect model's response. we experiment with three different variants of the baselines, depending on the number of examples in their prompts: (i) few-shot: we experiment with 5 examples foot_4 which is a more realistic few-shot setup. (ii) max. ex.: in another variant we use as many examples as fits within gpt's token limit. (iii) zero-shot: in this setup, we do not incorporate any example while prompting the models with the instructions. finally, we build variants of these baselines by conducting 'schema selection' where we experiment with 12 different encodings of the instruction @xcite and select the best performing one for each task. gptk w/ reframed instructions: the model designer applies various reframing techniques (section 3.2) on tasks in natural instructions. similar to the raw instructions baseline, we use 5 examples in our reframed tasks. in our setup, model designer is an author who follows the guidelines ( §3.2) by observing 5 examples in the development set and reframes instructions. this process was done in interaction with gpt3-instruct via the development examples. this took roughly 15 minutes per task and per reframing type. similar to the setup with raw instructions, the ultimate encoded prompts contained a concatenation of the following (in this order): reframed instructions, positive examples and the instance input. gptk w/ calibration: this method extends the recent calibration approach introduced by @xcite , which involves compensating for various model-specific biases in a few-shot setup, such as recency bias and majority bias. @xcite perform calibration by masking input instances with 'n/a' tokens, estimating the bias using model prediction probabilities and then compensating the bias while feeding the input instance during prediction. we extend calibration to our instruction setup by masking the input instance in our instruction encoding with an 'n/a' token and calibrating biases associated with gpt3-instruct. supervised baseline: while the conventional setup of supervised learning has been successful for reasonably sized models, it is prohibitively expensive for large models like gpt3. we train mediumsized lms (e.g., bart-base @xcite on 5k examples of each task and evaluate on unseen instances of the corresponding task. ## concluding remarks inspired by gpt3's poor performance in following task instructions, we study reframing them. we introduce five approaches that reformulate task instructions to make them easier, while maintaining their human readability. manually applying reframing on 12 tasks, we study their benefits compared to using raw instructions or fine-tuning mid-sized models. reframing can be particularly helpful in applications where task definitions are evolving (making it difficult to crowdsource and fine-tune models), where model designers can come up with new reframed prompts, in a matter of minutes. we hope that this study will inspire further investigation of potentially-unconventional approaches to exploit the knowledge harnessed by increasingly large lms where fine-tuning and its alternatives are prohibitively expensive.