Rethinking Data Selection at Scale: Random Selection is Almost All You Need (2025)

Abstract: Supervised fine-tuning (SFT) is crucial for aligning Large Language Models (LLMs) with human instructions. The primary goal during SFT is to select a small yet representative subset of training data from the larger pool, such that fine-tuning with this subset achieves results comparable to or even exceeding those obtained using the entire dataset. However, most existing data selection techniques are designed for small-scale data pools, which fail to meet the demands of real-world SFT scenarios. In this paper, we replicated several self-scoring methods—those that do not rely on external model assistance—on two million-scale datasets, and found that nearly all methods struggled to significantly outperform random selection when dealing with such large-scale data pools. Moreover, our comparisons suggest that, during SFT, diversity in data selection is more critical than simply focusing on high-quality data. We also analyzed the limitations of several current approaches, explaining why they perform poorly on large-scale datasets and why they are unsuitable for such contexts. Finally, we found that filtering data by token length offers a stable and efficient method for improving results. This approach, particularly when training on long-text data, proves highly beneficial for relatively weaker base models, such as Llama3. The code is available at https://github.com/xiatingyu/SFT-DataSelection-at-scale.

## introduction with the advent of large language models (llms) such as chatgpt, we have observed significant advancements in tasks involving instruction following @xcite , intent comprehension (lu † work done during the author's internship at the alibaba group * corresponding authors @xcite , and text generation @xcite . one of the primary objectives of developing llms is to harness their potential for generalizing to unseen natural language processing (nlp) tasks. to achieve this aim, many llms focus on precisely aligning with human instructions. figure 1 : the discrepancy between each methods and random selection on bbh benchmark @xcite . the y-axis represents the differential score, which is computed by subtracting the random selection score from the scores obtained using various methods. recent studies indicate that supervised finetuning (sft) can customize llms for specific domains, tasks, or applications by utilizing wellcrafted data. according to the study in @xcite , it is feasible to fine-tune a pre-trained language model with a relatively small set of examples. building on this insight, several papers have explored data selection strategies for sft of llms @xcite , emphasizing the importance of enhancing the quality of instruction tuning (it) data or increasing data diversity. these strategies can be classified into two primary categories: (1) extenral-scoring methods, which require support from more sophisticated external models like to score the data for the subsequent selection @xcite @xcite @xcite ; existing sft data selection methods, both external-scoring and self-scoring, are evaluated using well-known it datasets like alpaca-gpt4 @xcite , dolly @xcite , flan @xcite , wizardlm @xcite , and sharegpt @xcite . these datasets are small and come from a single source. however, during sft, much larger data volumes, usually in the hundreds of thousands to millions, are often needed. for instance, qwen2 (qwe, 2024) used over 500,000 data samples in sft. thus, for effective llm utilization, large-scale instructionfollowing data is crucial in sft. furthermore, largescale data should not only be abundant but also diversified, including professional annotations, real user data, or model-generated data, across various types like code, math, conversations, and knowledge q&a. this disparity highlights a gap between current sft data selection and real-world applications. to study how dataset size impacts selection strategy performance, we compared outcomes from existing methods with random selection within datasets ranging from 10k-30k to 1m on llama3-8b (ai@meta, 2024). figure 1 shows that as dataset size grows to 1m, these methods perform worse compared to random selection. "data size 10k-300k" refers to sources from original method papers. "data size 1m" refers to the openhermes2.5-1m dataset @xcite . motivated by this discovery, we reconsider whether sft data selection methods are viable for large-scale it datasets. given the high costs of external-scoring techniques @xcite , we focus on self-scoring methods. referring to @xcite , we classify self-scoring techniques into data quality-based and data diversity-based methods. data quality-based methods prioritize algorithms and metrics to score data items, selecting based on these scores, while data diversity-based methods prioritize dataset diversity. to examine the impact of self-scoring methods on llms' performance with vast it data, we test recent methods on two benchmarks with millions of cases. our experiments highlight three key points: prior research @xcite shows benefits of long text training for subjective evaluations like mtbench @xcite and alpacaeval @xcite ; we confirm its positive effect on objective tasks like big-bench-hard @xcite . although not always optimal for every language model, token length is beneficial in training long texts, notably for a weaker base model such as llama3-8b. ## related work external-scoring method. @xcite introduced an open-set instruction tagging method called instag, which employed chatgpt to generate detailed tags to measure and examine the variety and intricacy of human instructions for llms during sft. @xcite presented the al-pagasus model that used chatgpt to evaluate each instruction and then selected various data based on a certain threshold. @xcite suggested a model-oriented instruction selection approach that not only considered the quality and coverage of instruction data but also incorporated the necessity of instructions according to the capabilities of specific llms. @xcite introduced deita, it used chatgpt to iteratively enhance the complexity or quality of each data sample across relevant dimensions and then requested chatgpt to evaluate these samples for their complexity or quality. these models exceed the performance of the basic foundation models trained on complete datasets. however, they heavily depend on highperforming external llms to score data. self-scoring method. @xcite introduced a self-directed technique for llms to identify relevant instruction pairs from open-source data, using an instruction-following difficulty (ifd) metric to expose discrepancies between expected and self-generated outputs. @xcite developed diverseevol, which allows the model to independently select diverse training subsets to boost performance without human or advanced llm oversight. @xcite proposed less, which selects pertinent instruction tuning data by leveraging a gradient datastore to find examples similar to few-shot instances representing a specific capability. @xcite found that model performance diminishes with higher compression ratios in training data and introduced zip, a selection method favoring low-compression-ratio subsets. @xcite created selectit, which uses llms' uncertainty at various levels to identify superior instruction tuning data efficiently. @xcite presented nuggets, using one-shot learning and a scoring system based on perplexity influence to choose high-quality instruction data for tuning. ## self-scoring strategies in this paper, we focus on self-scoring methods that do not rely on external advanced llms to score data. we refer @xcite 's work and categorize existing resourceful data selection methods into two main perspectives: data quality-based methods and data diversity-based methods. ## conclusion in this study, we observe that many sft data selection methods depend on small-scale data sets, which do not meet the actual needs in real-world scenarios. this finding makes us rethink whether sft data selection methods can work when they are required to handle large-scale it datasets. we reproduce some existing self-scoring data selection approaches that do not need external llms' support on two million-scale datasets and find that almost all present methods do not significantly surpass random selection when dealing with largescale datasets. moreover, our analyses show that during the sft phase, data diversity in data selection plays a more significant role than data quality. in addition, using token length as the quality metric is more appropriate for sft data selection compared to other carefully crafted quality metrics.