Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2022)

Abstract: Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. It is commonly argued that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. In this study, we experiment with over 30 prompts manually written for natural language inference (NLI). We find that models can learn just as fast with many prompts that are intentionally irrelevant or even pathologically misleading as they do with instructively “good” prompts. Further, such patterns hold even for models as large as 175 billion parameters (Brown et al., 2020) as well as the recently proposed instruction-tuned models which are trained on hundreds of prompts (Sanh et al., 2021). That is, instruction-tuned models often produce good predictions with irrelevant and misleading prompts even at zero shots. In sum, notwithstanding prompt-based models’ impressive improvement, we find evidence of serious limitations that question the degree to which such improvement is derived from models understanding task instructions in ways analogous to humans’ use of task instructions.

## introduction suppose a human is given two sentences: "no weapons of mass destruction found in iraq yet." and "weapons of mass destruction found in iraq." they are then asked to respond 0 or 1 and receive a reward if they are correct. in this setup, they would likely need a large number of trials and errors before figuring out what they are really being rewarded to do. this setup is akin to the pretrain-andfine-tune setup which has dominated nlp in recent years, in which models are asked to classify a sentence representation (e.g., a cls token) into some similarly, reformatting nlp tasks with prompts such as the underlined text above has dramatically improved zero-shot and few-shot performance over traditional fine-tuned models @xcite @xcite . such results naturally give rise to the hypothesis that the extra prompt text included within each input example serves as semantically meaningful task instructions which help models to learn faster, in the way task instructions help humans to learn faster. this hypothesis is implicitly assumed by many and explicitly argued by @xcite , @xcite @xcite . while last years saw a gold rush of papers (summarized in §2) that proposed automatic methods for optimizing prompts, @xcite compare a representative sample of these newly proposed methods and report that @xcite 's manually written prompts still on average outperform the automatically searched prompts across a range of superglue tasks @xcite . such findings suggest that expert-crafted prompts are among the best, if not the best, which reinforces the above hypothesis that models benefit from meaningful instructions. in this paper, we test this hypothesis by evaluating various models on nli in zero-shot and fewshot settings using more than 30 manually written templates and 13 sets of lm target words for a total of over 390 prompts. we find that in most cases models learn identically as fast when given irrelevant or misleading templates as they do when given instructively good templates. further, models ranging from 235 million to 175 billion parameters all exhibit this behavior, as do the instructiontuned models, which are trained on hundreds of manually written prompts. while we confirm @xcite 's finding that instruction tuning substantially improves the performance and robustness of prompts, we also find that instruction-tuned models can be, in some sense, too robust and less sensitive to the semantics of the prompts, as compared to their non-instruction-tuned equivalents. finally, models are much more sensitive to the choice of the lm target words as opposed to the meaning of the instruction templates. in sum, despite promptbased models' dramatic improvement in zero-shot and few-shot learning, we find limited evidence that models' improvement is derived from models understanding task instructions in ways analogous to humans' use of task instructions. ## overall setup we implement a manual discrete prompt modelwhich in essence is the same as that of @xcite , except their implementation includes several augmentations such as self-labeling and ensembling of multiple prompts for competitive results. in order to focus on measuring the effect of prompts themselves, our implementation does not include those augmentations. following @xcite and @xcite , we evaluate by a rank classification of the target words. ## effect of templates our research question is whether models understand prompts as meaningful task instructions analogous to how humans would. for intuition, suppose an experimenter provides a human annotator with an informative instruction of a reasonably easy task. if the annotator understands the instruction, we expect them to perform better than when the experimenter provides intentionally misleading instructions, makes irrelevant chitchat, or says nothing at all. accordingly, we write various prompt templates that correspond to these different scenarios and evaluate models' performance with these templates in zero-shot and few-shot settings. ## conclusion in this study, we train several models with over 30 manually written templates and 13 sets of lm targets for nli. we find that models often learn equally fast with misleading and irrelevant templates as they do with instructive ones, and that the choice of the target words overrides the meaning of the overall prompts. although models do not entirely ignore the meaning of the prompts, our results contradict a hypothesis commonly assumed in the literature that models use prompts as semantically meaningful task instructions in ways analogous to humans' use of instructions.