Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning (2023)

Abstract: Recent works on instruction tuning (IT) have achieved great performance with zero-shot generalizability to unseen tasks. With additional context (e.g., task definition, examples) provided to models for fine-tuning, they achieved much higher performance than untuned models. Despite impressive performance gains, what models learn from IT remains understudied. In this work, we analyze how models utilize instructions during IT by comparing model training with altered vs. original instructions. Specifically, we create simplified task definitions by removing all semantic components and only leaving the output space information, and delusive examples that contain incorrect input-output mapping. Our experiments show that models trained on simplified task definition or delusive examples can achieve comparable performance to the ones trained on the original instructions and examples. Furthermore, we introduce a random baseline to perform zeroshot classification tasks, and find it achieves similar performance (42.6% exact-match) as IT does (43% exact-match) in low resource setting, while both methods outperform naive T5 significantly (30% per exact-match). Our analysis provides evidence that the impressive performance gain of current IT models can come from picking up superficial patterns, such as learning the output format and guessing. Our study highlights the urgent need for more reliable IT methods and evaluation.

## introduction recently, instruction tuning(it) has drawn much attention in the nlp communities, with the rapid growth of new models @xcite @xcite and datasets @xcite @xcite @xcite . models trained with task instructions demonstrate impressive zero-shot cross-task generalization ability. despite the remarkable results, tk-inst t0 flan alpaca vicuna training # of tasks 756 39 38 # of instructions 756 390* 380 52k 70k testing # of tasks 119 11 24 # of instructions 119 110* 240 252 252 ✔ ✔ ✔ ✗ ✗ it models @xmath0. --testing on unseen tasks? prior works have raised the question of whether models really learn to follow the instructions or just capture spurious correlations. @xcite , @xcite showed that the current large language models (llms) can achieve similar performance with misleading instructions(prompts) in in-context learning(icl) and few-shot learning scenarios. @xcite analyze how model utilize examples in icl. they observed that (1) input-output mapping in examples is not important and(2) output space information is crucial. besides icl and few-shot prompt-tuning, some works raise concerns about instruction following in the instruction tuning field @xcite @xcite , with a focus on test-time analysis. in contrast, we focus on analyzing how the models utilize instructions during the training process. we compare our analyzing methods and observation with prior works in appendix a.1. in this work, we conduct controlled experiments on natinst-v2 @xcite , the largest opensource instruction learning dataset includes 800+ english tasks with diverse task types, to study how models utilize instructions during it. note that existing research on it can be categorized into two ## background recently, many instruction tuning work train and test the models with instructions to achieve better zero-shot generalizability toward unseen tasks/instructions. we categorize these works by their objectives: generalize to unseen tasks and generalize to unseen instructions, and show the comparison in since instruct-gpt is not open-sourced and distilled it models such as alpaca and vicuna come up after our submission, we focus our analysis on the first category using the tk-instruct model and natinst-v2 dataset. however, we also conduct additional experiments and discuss the alpaca model's instruction following ability in ## experimental setup dataset. we conduct experiments on the natinst-v2 @xcite , the largest open-source instruction learning dataset, including over 800+ english tasks with diverse task types. the instructions include human-crafted human-readable task definition, positive task examples, negative task examples, and explanation. we focus on studying task definition and task examples, which were shown to be most useful in the original paper. model. we conduct experiments on tk-instruct, the current sota model provided in natinst-v2 paper. the model significantly outperformed previous sota models, such as t0 (62.0 v.s. 32.3 rouge-l for 11b model). we follow the seq-to-seq instruction-tuning method used in tk-instruct, and train a t5-large-lm-adapt (770m parameters) model (raffel et al., 2020) with performance comparable to the larger model (3b parameters) reported in wang et al. (2022). 1 evaluation metrics. for task definition, we separately evaluate classification and generative tasks using exact match and rouge-l respectively. for 10 20 50 200 800 15 20 25 30 35 40 45 50 55 60 original example for testing stage original delusive empty t5 w/o it 10 20 50 200 800 delusive example for testing stage # of instances per task overall score(rouge-l) controlled experiments for task example task examples, we follow @xcite to report the overall rouge-l score for both classification and generative tasks. to understand the impact of training examples, we report model performances with varying numbers of training instances per task (i.e., 10, 20, 50, 200, 800). ## discussion do alpaca better follow the instruction on natinst-v2 dataset? after our submission, new instruction tuning models, like alpaca and vicuna, are trained on distilled data from chat-gpt and exhibit behavior closer to it. to investigate their instruction utilization, we conduct the "altered task definition" experiment on llama-7b @xcite and alpaca-7b models using the natinst-v2 test set. in ## conclusion we constructed controlled experiments on natinst-v2 to compare model training with altered vs. original instructions (task definitions and examples). our findings indicate that some current it models do not fully utilize instructions, and the impressive performance gains of it may come from models learning superficial patterns, such as the output space and format. we suggest future research on instruction tuning to analyze their performance gains with more comprehensive evaluation and benchmark against trivial baselines. 1321 while our analysis suggests that it models do not fully utilize instructions but instead learn superficial patterns from instructions, there are some limitations to our experiments. first, we only analyze a sota it method on the natinst-v2 dataset and t0 dataset. though @xcite showed that their model can outperform other large models such as instruct-gpt @xcite and t0 @xcite , we did not analyze other it methods, such as rlhf (reinforcement learning from human feedback) in instruct-gpt. secondly, since our analysis is conducted in the training stage, we cannot analyze private models such as chat-gpt. also, we did not explore models larger than 7b parameters due to our computation resource limitation. this may miss some emergent abilities of large language models (llms) @xcite . lastly, while we observe the models do not utilize the majority of the instructions by it, a certain degree of instruction understanding may already exist in pre-trained llms, which we did not study in this work. in conclusion, our work is a concentrated analysis to illuminate the potential vulnerability of the current it models and evaluation metrics. we encourage future works to conduct more comprehensive studies on larger models and propose more reliable it methods and evaluation frameworks.