Data Diversity Matters for Robust Instruction Tuning (2024)

Abstract: Recent works have shown that by curating high quality and diverse instruction tuning datasets, we can significantly improve instruction-following capabilities. However, creating such datasets is difficult and most works rely on manual curation or proprietary language models. Automatic data curation is difficult as it is still not clear how we can define diversity for instruction tuning, how diversity and quality depend on one other, and how we can optimize dataset quality and diversity. To resolve these issue, we propose a new algorithm, Quality-Diversity Instruction Tuning (QDIT). QDIT provides a simple method to simultaneously control dataset diversity and quality, allowing us to conduct an in-depth study on the effect of diversity and quality on instruction tuning performance. From this study we draw two key insights (1) there is a natural tradeoff between data diversity and quality and (2) increasing data diversity significantly improves the worst case instruction following performance, therefore improving robustness. We validate the performance of QDIT on several large scale instruction tuning datasets, where we find it can substantially improve worst and average case performance compared to quality-driven data selection.

## introduction large pre-trained language models have demonstrated a remarkable ability to perform a wide range of natural language processing tasks @xcite @xcite @xcite . although these models are powerful, pre-trained models such as gpt-3 can be quite difficult to work with and often do not follow user instructions @xcite . to unlock instruction-following capabilities, researchers have turned to instruction tuning, in which language models are trained to follow instructions on a small set of example instructionresponse pairs @xcite @xcite . instruction tuning (ift) has become extremely popular, as it provides a simple way for researchers to train powerful and aligned language models @xcite @xcite . although initial works apply instruction tuning to large-scale datasets, it has recently been found that a small set of well chosen instruction-response pairs is sufficient for good performance. in particular, @xcite showed that by training on only 1000 instructions manually selected or crafted by experts, superior performance can be achieved compared to training on larger datasets. training with a small dataset has the added benefits of lowering training costs and enabling faster iteration. although such manual data selection is not scalable, this work raises an important question: how can we automatically select an instruction tuning dataset? recent work on dataset curation have identified two characteristics that an instruction tuning dataset should have: (1) the instruction responses should be high quality @xcite and (2) the instructions should cover a wide range of tasks (i.e. be diverse) @xcite @xcite . to curate high quality datasets, researchers have used proprietary llms to measure the quality of each data point in the dataset and then select only the highest quality data points. to improve dataset diversity, researchers have manually selected instructions that cover a wide range of topics and formats @xcite . while both of these methods enhance instructionfollowing capabilities, it is not clear how we can select high quality and diverse datasets without relying on manual curation from human experts, a process that is time-consuming and expensive. in pursuit of this goal, we propose a new algorithm, qdit, to measure and optimize the diversity and quality of instruction tuning datasets. qdit measures diversity using the facility location function @xcite . the facility location function provides an intuitive measure of subset diversity, as it essentially measures how well represented each data point in the full dataset is by the data points in the selected subset. with this diversity function and quality functions from prior works, we then define a dataset's quality-diversity score as a simple linear combination of dataset quality and diversity. to optimize the quality-diversity score, qdit employs a greedy strategy, where the data point that will improve the joint qualitydiversity score the most is selected at each time step @xcite . this procedure is extremely efficient, and can easily scale to datasets with millions of instruction. qdit provides an effective way to control the diversity and quality of the instruction tuning dataset, allowing us to conduct an in-depth study of diversity and quality in instruction tuning. from this study we identify two key findings: (1) there is an inherent tradeoff between dataset diversity and dataset quality and (2) improving dataset diversity primarily improves the worst and average case instruction following ability, while not affecting best case instruction following ability much. based on these finding, we are able to use qdit to optimize the quality-diversity tradeoff, improving worst case performance while maintaining or improving best case and average performance for robust instruction following. we extensively validate our results on five large-scale instruction tuning datasets. ## related work there have been several works that attempt to improve the quality and diversity of ift datasets. ⋄ manual data selection. several works have shown that superior instruction-following capabilities can be unlocked by carefully selecting and writing instruction-response pairs @xcite @xcite . to select such data quality and diversity are emphasized, with data from various scientific fields and internet forums being selected. it is not clear how such datasets can be automatically selected. ⋄ distilling closed models. to reduce the human effort required for dataset creation, researchers have used powerful proprietary llms such as gpt-4 to create instruction tuning datasets @xcite @xcite . again researchers found dataset quality @xcite and diversity @xcite to be most important. although resulting in powerful datasets, the reliance on proprietary language models in these works is expensive and may raise legal concerns @xcite . ⋄ automatic data selection for instruction tuning. due to the aforementioned issues, in this paper we focus on automatic selection of smaller instruction tuning datasets from larger ones. @xcite al. (2023); dong et al. (2023b) show that by rating the quality of each data point and training on the highest quality data points, downstream performance can be significantly improved. @xcite propose to select instructions based on difficulty. wang et al. (2022a); liu et al. (2023) attempt to increase diversity by restricting the distance between selected points to be larger than a given threshold. we find that this method does not necessarily lead to a significant increase in diversity, and compare qdit to similar approaches in our experiments. our work is also related to several works that seek to increase diversity in nlp , but do not consider data quality @xcite @xcite . concurrently, @xcite ; @xcite analyze the efficacy of various diversity and uncertainty metrics for ift, but do not consider dataset quality. ## methodology before presenting qdit, we discuss how to quantify instruction diversity and instruction-response quality. ## experiments we first analyze how the qdit algorithm affects dataset quality and diversity and then present our main results and analysis. ## limitations and risks this paper seeks to understand how dataset diversity and quality impact instruction tuning performance. we consider several datasets, base models, and benchmarks. some limitations of our study are that we do not consider extremely large scale models (e.g. 70b), we assume a good measure of instruction quality (i.e. reward model) is available, and we only consider supervised finetuning, which is only one part of llm alignment. to further improve upon the papers limitations, we would like to try scaling our experiments to even larger model sizes (e.g. 70b). in addition, it would be interesting to conduct our study on more sophisticated alignment techniques such as rlhf. our work is largely foundational and we therefore do not see any direct risks stemming from our work. however, it is possible that our method could be adapted to select harmful data to train on. (a) 1k selected points (b) 5k selected points (c) 10k selected points figure 13: effect of α on dataset quality and diversity. the dataset is ultrachat 1.3m. (a) 1k selected points (b) 5k selected points (c) 10k selected points figure 14: effect of α on dataset quality and diversity. the dataset is lmsys 1m.