Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2023)

Abstract: Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions. These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions. In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor. We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth. This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs. Experiments show that despite containing a fair amount of noise, training on Unnatural Instructions rivals the effectiveness of training on open-source manually-curated datasets, surpassing the performance of models such as T0++ and Tk-Instruct across various benchmarks. These results demonstrate the potential of model-generated data as a cost-effective alternative to crowdsourcing for dataset expansion and diversification.

## introduction instruction tuning enables pretrained language models to generalize to unseen tasks in a zero-shot setting @xcite . one way to collect examples of instructions and their execution is to reformulate existing nlp datasets in an explicit instruction-input-output format via prompt engineering @xcite . however, the resulting data is limited to existing academic benchmarks, even though the instruction paradigm can describe any text-based task @xcite . alternatively, @xcite collect user-generated prompts and manually annotate their expected outputs, reflecting a different (and arguably more desirable) distribution of the instruction space, but requiring a live application with existing users and major investments in human annotation. can we create a large dataset of instructions that is diverse in tasks, content, and phrasing, without human labor? we introduce unnatural instructions, a dataset of natural language instructions and their corresponding inputs and outputs. inspired by recent work on utilizing language models for data generation @xcite @xcite , we collect data in a fully automatic manner by prompting a pretrained language model with three examples from the super-natural instructions foot_0 dataset @xcite and asking the model to generate a fourth (figure 1 ). we repeat this process with 5 different seeds -i.e. the entire process requires only 15 instruction examples -to automatically produce 64,000 diverse triplets of instructions, inputs, and outputs. 2 we further diversify the dataset's format by generating additional natural language paraphrases of each instruction, while preserving the contents of any input arguments and outputs, expanding the dataset to approximately 240,000 examples. although the dataset contains noise, our analysis reveals that more than 50% of generated examples are indeed correct, and that even incorrect examples typically contain valuable information for instruction tuning. at the same time, we find that unnatural instructions contains highly creative tasks -some of which are very different from "classic" nlp tasks -and has a more diverse set of instructions than super-natural instructions. experiments show that fine-tuning an 11bparameter t5 model @xcite on unnatural instructions can outperform both t0++ @xcite and tk-instruct @xcite across several benchmarks, including super-natural instructions @xcite , big-bench hard @xcite , and lmentry @xcite . when controlling for all variables besides the data, we find that a model trained on unnatural instructions performs competitively with a baseline model trained on super-natural instructions. in particular, we observe an 18-point gain on big-bench hard (original task formulation) and a 16-point gain on lmentry, suggesting that unnatural instructions is particularly useful for generalizing to instructions that deviate from the distribution of classic nlp tasks. these improvements become even more pronounced when the cost of generating examples is amortized; in this case, training on unnatural instructions substantially outperforms our baseline on all benchmarks. we observe a log-linear relationship between the number of generated examples and downstream task performance, suggesting that performance of models trained on unnatural instructions can further be improved simply by increasing its size. beyond the immediate implications on instruction tuning, this work demonstrates the viability of automatic dataset expansion using language models as an alternative to crowdsourcing. unnatural instructions highlights the ability of language models to produce creative and diverse data, a trait that is difficult to obtain with crowd workers, who lack the intrinsic motivation to create novel examples and typically collapse into predictable heuristics to form annotation artifacts @xcite . at the same time, language models are faster and cheaper than human labor, opening up new possibilities for scaling up data annotation. ## data collection we introduce unnatural instructions, a dataset of 240,670 diverse natural language instructions. each example contains a natural language instruction as input and its expected execution as output. unnatural instructions is collected in a completely automatic process, requiring a seed of only 15 manually-constructed examples, which can be produced in about one hour of human labor. we first collect a core set of 68,478 examples ( §2.1) by prompting a pretrained language model m with a seed of 3 manually-annotated examples to produce a new (fourth) example. this phase uses a structured instruction format and filtering heuristics to ensure data quality. we then expand the core dataset by rephrasing the structured instructions in example 1 instruction: you are given a science question (easy-level) and four answer options (associated with "a", "b", "c", "d"). your task is to find the correct answer based on scientific facts, knowledge, and reasoning. do not generate anything else apart from one of the following characters: 'a', 'b, 'c', 'd'. there is only one correct answer for each question. ## data analysis we first demonstrate the creativity of unnatural instructions, and then manually analyze 200 randomly-sampled examples from our core dataset, focusing on correctness and diversity. we also compare our data's distribution to super-natural instructions, and find our inputs to be more diverse. creativity a major challenge when creating an instruction dataset is task creativity. crowd workers typically collapse into predictable heuristics to form annotation artifacts @xcite . while the high performance of models trained on unnatural instructions (see §5) suggests that it is indeed diverse and creative, we also present in correctness when evaluating correctness, we test whether (1) the generated instructions are logical and executable, (2) the input arguments correspond to the task described in the instruction, and (3) the outputs are correct, given the instruction and input. although our data filtering process is minimal, 113 of the 200 analyzed examples (56.5%) are correct. of the 87 incorrect examples, 9 (4.5%) had incomprehensible instructions, 35 (17.5%) had an input that did not match the task description, and 43 (21.5%) had incorrect outputs. while the amount of noise in the data may raise concerns regarding its usability, many of the examples that were marked as incorrect can still be considered informative. for example, one erroneous example had the instruction "in this task, you will be provided with a list of countries and their corresponding capital cities. you are also given a list of clues...for each clue, determine which country it is referring to and write down that country's name..." the input argument was "clue 1: this capital city is on two different continents." this example is incorrect since the input does not conform with the format described by the instruction -a list of countries and their capitals is not provided, only a clue. however, the output is istanbul, turkey, which indeed lies in both europe and asia and therefore corresponds with the input clue. in §5 we show that, despite being noisy, unnatural instructions provides a highly informative training signal. ## experimental setup we describe model fine-tuning on unnatural instructions and our evaluation protocol. ## results our main results are shown in ## generative model ablations as a data generation model, we used text-davinci-002, an instruction-tuned variant of gpt-3 @xcite . however, our approach is not limited to this specific model. we experiment with original (untuned) gpt-3 model by using it as the model m in both the input generation and output generation phases (see §2). we train models for 1,500 steps using 2,000 examples and evaluate the super-natural instructions validation set performance as a proxy, averaged across three different random seeds. other work suggested creating datasets entirely automatically, without the need for labeled data. @xcite and @xcite propose to leverage pretrained language models to generate entire labeled datasets from scratch, for a given, predefined task. @xcite use pretrained language models to automatically construct multilingual qa data using only five examples per language. ## conclusion we introduce unnatural instructions, an automatically generated dataset of natural language instructions and their corresponding inputs and out-puts. to the best of our knowledge, this is the first general-purpose nlp dataset that was automatically generated. our experiments show that models trained on unnatural instructions outperforms models trained on manually annotated datasets across several benchmarks. unnatural instructions is not only cost-effective, we also provide evidence of enhanced diversity in the instructions produced and a high level of creativity in the tasks devised, a trait difficult to obtain with crowd workers. ablations show that even weaker models without instruction tuning can generate useful instructions, though they may struggle with producing the corresponding outputs. however, coming up with interesting tasks and writing diverse instructions is arguably the main challenge of the data collection process, whereas given instructions and inputs, outputs are often far easier to annotate through crowdsourcing. our findings incentivize utilizing models for general-purpose data generation, which we view as an intriguing direction for future research.