Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation (2023)

Abstract: Instruction tuning has emerged to enhance the capabilities of large language models (LLMs) to comprehend instructions and generate appropriate responses. Existing methods either manually annotate or employ LLM (e.g., GPT-series) to generate data for instruction tuning. However, they often overlook associating instructions with existing annotated datasets. In this paper, we propose Dynosaur, a dynamic growth paradigm for the automatic curation of instruction-tuning data. Based on the metadata of existing datasets, we use LLMs to automatically construct instruction-tuning data by identifying relevant data fields and generating appropriate instructions. By leveraging the existing annotated datasets, Dynosaur offers several advantages: 1) it reduces the API cost for generating instructions (e.g., it costs less than $12 USD by calling GPT-3.5-turbo for generating 800K instruction tuning samples; 2) it provides high-quality data for instruction tuning (e.g., it performs better than Alpaca and Flan on Super-NI and Longform with comparable data sizes); and 3) it supports the continuous improvement of models by generating instruction-tuning data when a new annotated dataset becomes available. We further investigate a continual learning scheme for learning with the ever-growing instruction-tuning dataset, and demonstrate that replaying tasks with diverse instruction embeddings not only helps mitigate forgetting issues but generalizes to unseen tasks better. Code and data are available at https://github.com/WadeYin9712/Dynosaur.

## introduction instruction tuning @xcite @xcite enables large language models (llms) @xcite @xcite to provide appropri-ate output according to input instructions. existing approaches compile instruction-tuning datasets mainly by 1) manual annotations or 2) distillate from a larger size of llm. for example, super-naturalinstruction (super-ni) @xcite and dolly @xcite recruit experts to manually annotate task instructions and related task data. despite their high quality, this approach is labor-intensive and costly @xcite . recent efforts @xcite leverage gpt-series to distill instruction tuning data to train smaller models. however, subsequent studies @xcite argue that these methods merely help smaller models learn to mimic the style of teacher llms without inheriting their true capabilities, such as factuality and problem solving skills. we suspect it is mainly due to the instructions not ground to actual data. in this paper, we propose dynosaur, a dynamic growth paradigm to convert high quality annotations from dataset repositories into instructiontuning data. in particular, dynosaur generates instructions based on the metadata of existing datasets in the dynamically growing huggingface datasets platform @xcite . as shown in figure 1 , metadata covers essential information about a dataset, including dataset description ("a collection of ... ebooks ..."), dataset name ("gutenburg_english"), data fields ("title", "text", ..., "issued") and dataset annotations. guided by metadata, our method can generate multiple tasks applicable for forming instruction-tuning data with instances in nlp datasets. we leverage llms to harvest task instructions and their corresponding input/output fields with a single prompt. prompted with dataset description involving ebooks and data fields about the book published information, llms can synthesize instructions such as "given a gutenburg passage, generate its title" and "predict the year when the book is published based ## collection of dynosaur data in this section, we introduce how to construct the dynosaur dataset. as shown in figure 1 , we first collect metadata from existing datasets, then prompt llm to create tasks based on the metadata, and filter out invalid ones. ## experiments we conduct two sets of experiments to evaluate the quality of dynosaur. we first evaluate models trained with dynosaur on super-ni and longform to examine its ability to solve nlp tasks. then we run a human evaluation to examine if dynosaur helps in user-oriented situations. ## continual learning with dynamically growing datasets as dynosaur can expand over time as new tasks come in, an important question is how to adapt an instruction-tuned model to new tasks without suffering from catastrophic forgetting. in this section, we examine continual learning as an approach for learning instruction-following models with dynamically growing datasets. we focus on one of the common continual learning techniques @xcite , replay methods, which select previously trained tasks for further training stage. we aim to provide an analysis of how to most effectively select the tasks to replay. we want to answer the following questions: 1) do we need to replay history tasks? 2) shall we replay tasks based on instructions or data? 3) which tasks to replay?. 2 replay methods. we compare the following replay strategies: 1) no replay: train models without any replay tasks; 2) instr. diverse: replay last stage's tasks that diverge most from ones in the current stage based on instruction representations; 3) instr. similar: replay last stage's tasks that are most similar to tasks in the current stage; 4) instr. support: replay the most representative tasks in the last stage; 5) data diverse: replay diverse tasks based on similarity of example data. suppose there are l tasks in the current stage, and k tasks in the previous stage, we use sentence transformer @xcite based on roberta-large @xcite to obtain the instruction representation matrix i @xmath0, where d is the representation dimension. then, we compute the cosine similarity between i c and i p , and i p itself: s @xmath1, i p ) ∈ r l×k , s @xmath2, i p ) ∈ r k×k . then, instr. diverse replays the tasks with the least column sum in s cp . instr. similar replays the tasks with the largest column sum in s cp . instr. support replays the tasks with the largest row sum in s pp . ## related works llms can be empowered to follow instructions via instruction tuning @xcite @xcite @xcite @xcite @xcite @xcite . they fine-tune llms with the training data and instructions of diverse upstream training tasks and enable them to do inference on unseen tasks. one branch of instruction-tuning data are constructed with existing human annotations. the instructions in promptsource @xcite and flan @xcite are created with human-designed templates for limited task categories. ni @xcite and super-ni @xcite are annotated by nlp practitioners from github and nlp courses. most recent attempts distill instruction-tuning data from llms. the methods proposed in self-instruct @xcite and unnatural instruction @xcite generate novel tasks by prompting llms with seed instructiontuning tasks. other works @xcite study instruction generation upon input/output data. there are another type of works simply using structured metadata as instructions @xcite . different from those works, when we generate dynosaur instructions, the inputs/outputs for the generated tasks are unknown to llms. llms need to generate instructions from metadata and determine which part of the dataset annotations are task inputs/outputs simultaneously. ## conclusions we propose dynosaur, an automatic paradigm for instruction data construction. we utilize metadata from existing nlp datasets and generate various tasks upon them. dynosaur generation costs significantly lower than other methods, while models trained on dynosaur data outperform models trained on existing human-curated and machinegenerated instruction datasets on super-ni and longform. taking advantage of the dynamic growth nature of dynosaur, we further explore specific replay methods for instruction tuning that are effective in mitigating forgetting.