Are Language Models Worse than Humans at Following Prompts? It’s Complicated (2023)

Abstract: Prompts have been the center of progress in advancing language models’ zero-shot and few-shot performance. However, recent work finds that models can perform surprisingly well when given intentionally irrelevant or misleading prompts. Such results may be interpreted as evidence that model behavior is not “human like’. In this study, we challenge a central assumption in such work: that humans would perform badly when given pathological instructions. We find that humans are able to reliably ignore irrelevant instructions and thus, like models, perform well on the underlying task despite an apparent lack of signal regarding the task they are being asked to do. However, when given deliberately misleading instructions, humans follow the instructions faithfully, whereas models do not. Thus, our conclusion is mixed with respect to prior work. We argue against the earlier claim that high performance with irrelevant prompts constitutes evidence against models’ instruction understanding, but we reinforce the claim that models’ failure to follow misleading instructions raises concerns. More broadly, we caution that future research should not idealize human behaviors as a monolith and should not train or evaluate models to mimic assumptions about these behaviors without first validating humans’ behaviors empirically.

## introduction prompting has emerged as the default way of using large language models @xcite @xcite @xcite . however, a collection of recent papers show that models perform surprisingly well when given misleading or irrelevant prompts @xcite @xcite , corrupted in-context examples @xcite , or corrupted explanations or chain-of-thought @xcite . such results raise questions about whether language models' ability to follow instructions is analogous to humans' ability to do so. in this paper, we investigate what humans do in such settings. we follow the experimental setup used by @xcite to study how instructions in prompts affect lms' performance. w&p manually write a set of prompts for various natural language inference (nli) and coreference resolution datasets. these prompts cover three main categories: instructive, irrelevant, and misleading. for example, given {sentence1} , can we assume it is true that {sentence2} is an instructive prompt for nli, whereas {sentence1} does the above passage express a positive sentiment? {sentence2} is a misleading instruction for nli. (see instead, w&p find that t5 (lm-adapted, 11b, @xcite , t0 @xcite , and @xcite do not exhibit the above patterns. rather, in both zero-shot and fewshot settings, models perform roughly the same on instructive, misleading, and irrelevant prompts, violating a1 and a2 above. models do, however, perform better given any type of instructions than they do with no instructions (i.e., a3 holds). therefore, w&p conclude that while prompts do confer substantial empirical benefits, the fact that models are so good at inferring the gold labels under various pathological prompts casts doubts on whether models understand or use instructions in ways similar to how humans do. in this paper, we revisit w&p's assumptions on how humans behave with pathological prompts. ## experiment overview following w&p, we use natural language inference (nli) as the primary task for our experiments. (that is, in our results, we always report human and model performance with respect to the nli task.) when necessary, in designing stimuli for the misleading prompt condition (discussed in detail below), we additionally draw examples from 7 other tasks: lexical overlap, lexical identity, paraphrasing identification, grammatical acceptability, summarization acceptability, topic classification, and language identification (see appendices h.2 and h.3 for details); we refer to these collectively as surface tasks in this paper. we define an example to be a pair @xmath0, sentence2>. for nli, sentences 1 and 2 are the premise and hypothesis, respectively. 1 we define an item as a unique 3-@xmath1, instruction, sentence2>, i.e., an example fitted within a prompt template, which can be instructive (w.r.t. nli), misleading, irrelevant, or empty (null). 2 1 for other tasks, if they do not need a sentence pair (e.g., sentiment analysis), our instructions always unambiguously ask for judgment of sentence1. crucially, when the instruction is misleading, we manually select examples such @xmath2, misleading instruction, @xmath3, nli instruction, @xmath4. assuming participants are competent at nli as well as at each of the relevant surface tasks foot_0 used in our experiments, this design enables us to distinguish whether the participant is performing the nli task or surface task when given misleading instructions. procedure to ensure this experiment is as zeroshot as possible, each participant receives only one test item, followed by four additional items which we use as controls to ensure that all tasks and examples are fair (appendix e). for each item, subjects choose between"yes" or "no". they do not receive any feedback throughout the experiment. ## results figure 2 show the zero-shot performance of humans compared to that of instruction-tuned that our participants are sufficiently competent. 4 we add a line break after the premise text for better human readability, and we edit all prompts to be clearer by making use of the line break (e.g. "the above passage"). models. the overall performance of t0++, flan-t5 and gpts by themselves is consistent with the model performance reported by w&p. but, with the exception of gpt-4, models show very different patterns from that of humans when given misleading prompts. as expected, when humans are explicitly asked to do a task other than nli (e.g., sentiment analysis, grammaticality), they tend to do the specified task (leading to low accuracy when measured against the nli task). in contrast, models often appear to behave as though they have been instructed to do nli even though they are instructed to do some other surface tasks (leading to high accuracy when measured against the nli task). for example, when given a misleading instruction (e.g., paraphrasing identification as opposed to nli) {sentence1} does that have the same meaning as "{sentence2}"?, humans do indeed perform the paraphrasing task and thus receive a score of 0 on nli, whereas models tend towards performing nli, with t0++ / gpt-4 receiving an nli score of 1 and flan-t5 / gpt-3.5 a score of 0.6. (full results on all prompts in §g.1.) this pattern confirms w&p's assumption a1 that models perform 'too well' on misleading prompts, i.e., better than humans would under similar conditions. however, when we consider assumption a2 (@xmath5, we see a different story. when given irrelevant instructions, we see that all models and humans exhibit similar patterns. in fact, humans show far less variance than models in performing the nli task when figure 2: zero-shot accuracy of human annotators vs. instruction-tuned models. in the misleading condition, lower nli accuracy is preferred as it means higher accuracy in following the instructions to perform the surface tasks. medians are displayed in the bars. each scatter point represents the accuracy on a instruction within the semantic category. for models, this is calculated as the mean accuracy over all the test items constructed from the examples for the instruction (i.e., 5 for each misleading instruction, 12 for each null, irrelevant and instructive instruction); for humans this is the mean accuracy over all participants who received items with that instruction. the instructions are irrelevant, suggesting that humans are more likely to perform nli as some kind of "default" task absent of useful instructions. for example, given the irrelevant instruction {sentence1} inflections are annoying and thank god that middle english got rid of most of them. {sentence2} humans, t0++, flan-t5, gpt-3.5 and gpt-4 all score similarly at 0.79, 0.83, 0.83, 0.67 and 0.75 respectively ( §g.1). ## discussion our findings show that, in a zero-shot setting, humans appear largely faithful to prompt instructions. they perform well given instructive prompts and poorly given misleading prompts (as expected). however, we observe that t0++, flan-t5, and (to a lesser extent) gpt-3.5 are inclined to perform nli in a zero-shot setting regardless of what is in fact being instructed. gpt-4, however, does seem to exhibit a more human-like pattern in following the misleading instructions (albeit with high variance). it is possible that a combination of pretraining flops, fine-tuning data quality, and rlhf may have bridged some discrepancies between gpt and human behaviors, whereas smaller and supervised fine-tuning-only models fail to do so, but it is impossible to conclude given that little is known about gpt-3.5/4's technical details. when no useful instructions are provided in the irrelevant prompt setting, our results show that humans still tend to perform the nli task surprisingly well. contrary to w&p's criticism, models' ten-dency to do the same with irrelevant prompts is likely more a feature than a bug (cf. @xcite . such idiosyncrasies in human behavior are often difficult to anticipate and even more difficult to codify in a way that lends itself well to benchmarks. thus, we echo recent work @xcite @xcite @xcite in emphasizing the difficulty of evaluating models when "human-likeness" is the intended gold standard. as nlp models become increasingly advanced, and evaluation tasks increasingly complex, we are likely to face increasing challenges in determining whether models' behavior is "aligned" with that of humans. this study contributes to the line of work which underscores the importance of empirically measuring, rather than presupposing, human behavior in such settings, as humans in practice routinely evade basic intuitions. appendix b further discusses how to design a fairer comparison between humans and models. ## conclusion in this work, we measure human behaviors when given misleading and irrelevant instructions for various nlp tasks. we show that our prior work used oversimplified assumptions of human behavior to evaluate nlp models. our results underscore the need to empirically validate assumptions about human behavior, which is often more complex in reality than our intuitions would lead us to believe.