Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning (2024)

Abstract: Instruction tuning is critical to improve LLMs but usually suffers from low-quality and redundant data. Data filtering for instruction tuning has proved important in improving both the efficiency and performance of the tuning process. But it also leads to extra cost and computation due to the involvement of LLMs in this process. To reduce the filtering cost, we study Superfiltering: Can we use a smaller and weaker model to select data for finetuning a larger and stronger model? Despite the performance gap between weak and strong language models, we find their highly consistent capability to perceive instruction difficulty and data selection results. This enables us to use a much smaller and more efficient model to filter the instruction data used to train a larger language model. Not only does it largely speed up the data filtering, but the filtered-data-finetuned LLM achieves even better performance on standard benchmarks. Extensive experiments validate the efficacy and efficiency of our approach.

## introduction earlier works of instruction tuning on large language models (llms) @xcite @xcite focus on creating large, varied, and high-quality datasets of various tasks with responses curated by human experts @xcite @xcite @xcite , which can be bottlenecked by the intensive human labor. an alternative is to generate the data by a powerful teacher llm @xcite @xcite @xcite but the quality is hard to control and largely depends on the teacher. lima @xcite finds that a mere 1,000 human-crafted high-quality data could significantly * corresponding author improve an llm's instruction-following capability, based on which they posit that llms acquire most knowledge during the pretraining and thus a few data suffices for instruction tuning. to further free the human labor in data curation and accelerate the instruction tuning process, a line of recent works apply an extra filter algorithm to select data from the existing dataset. however, the model used in the filtering process usually needs to be as powerful as chatgpt @xcite or requires additional reward data training @xcite , or is the student model itself @xcite , which leads to additional expensive cost and latency due to the inference on these large filter models, especially when the original dataset is large while only a tiny fraction of data needs to be selected. these paradigms are presented in figure 1 (a) and (b). to reduce the filtering cost, we study superfiltering: can we use a smaller and weaker model as a filter to select instruction-tuning data for training a larger and stronger model? this was first studied for training small classification models by @xcite while the effectiveness on the open-domain instruction dataset is un-explored. recently, weak-to-strong generalization @xcite proposes to utilize a weaker chatgpt to generate data used to finetune a stronger gpt4 model, which shares a similar spirit with our superfiltering as depicted in figure 1(c ). in superfitering, we find that a smaller and weaker gpt-2 (124m) @xcite suffices to replace previously used large filter models and select high-quality instruction tuning data used to finetune a much larger llama2 (7b or 13b). this is motivated by our main discovery of filter models' consistency on two data statistical metrics, perplexity and instruction-following difficulty (ifd) score @xcite . despite the differences in scales across different filter models, their rankings of the same instruction tuning dataset are surprisingly consistent, as demonstrated by the large rank correlation coefficients evaluated on different models and datasets. our thorough empirical study implies that weaker language models possess a capability consistent with their stronger counterparts in comprehending and discerning the difficulty of diverse instructions, though they may differ in other skills like reasoning and generalization. in extensive experiments, our superfiltering strategy using gpt-2 as the filter, as exemplified on several widely used instruction datasets, brings significant speedups to data filtering for instruction tuning. by utilizing only 5% of the original data volume, superfilter allows us to attain llms comparable, and in some instances superior, to those achieved by training with full data. our main contributions can be summarized in three folds: ## weak-to-strong consistency in this section, we delve into the hypothesis that weak and strong language models share a relatively consistent capability in perceiving the difficulties of instruction tuning samples. ## conclusion in this paper, we reveal the consistency between weak and strong language models in perceiving instruction difficulties. based on the consistency, we present "superfiltering", a novel and efficient approach for data filtering in the instruction tuning of llms. by effectively utilizing weaker models as proxies for evaluating instructional data, we achieve a significant leap in efficiency, accelerating the data filtering process largely. the experimental results affirm that our method considerably reduces computational overhead while maintaining or even improving the instruction tuning performance of llms. it is astonishing that a weak language model like gpt-2 is able to filter the data responses generated by gpt-4. our superfiltering marks a substantial contribution by offering a scalable, resource-efficient, and effective strategy for the advancement of ai technologies.