Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models (2025)

Abstract: Data selection for fine-tuning large language models (LLMs) aims to choose a high-quality subset from existing datasets, allowing the trained model to outperform baselines trained on the full dataset. However, the expanding body of research lacks a clear, unified framework, and the variability in experimental settings complicates systematic comparisons.While existing surveys comprehensively overview the stages and methods of data selection, they often overlook an in-depth exploration of the fine-tuning phase. In this paper, we conduct a focused review of recent data selection techniques for fine-tuning LLMs, analyzing a dozen key studies. We introduce a novel three-stage scheme—comprising feature extraction, criteria design, and selector evaluation—to systematically categorize and evaluate these methods. Additionally, we propose a unified comparison approach that incorporates ratio-based efficiency and ranking-based feasibility metrics to address inconsistencies across experiments. Our findings reveal that methods emphasizing more targeted quality measurement achieve higher efficiency but at the cost of feasibility. Finally, we discuss trends and highlight four key challenges in fine-tuning data selection, offering potential directions for future research.

## introduction supervised fine-tuning (sft) leverages small amounts of instruction-pair data to unlock large language models' instruction-following capabilities and improve generalization across various tasks @xcite @xcite @xcite . recent research highlights that data quality is more critical than data quantity for effective * the first two authors are equal contributions. ## feature extraction feature extraction plays a crucial role in the data selection process. some works preserve the original text data, believing that it contains the most comprehensive information @xcite , while others transform raw texts into more compact and representative feature sets (e.g., human-designed features and model-oriented features) to facilitate selection. the former approach aligns with human instinct by retaining linguistic structures, while the latter focuses on extracting features directly from the model to provide more targeted data representations. ## criteria design the source of the quality label is a critical factor in designing selection criteria, which can be categorized into internal and external information. internal information refers to quality signals inherent in the candidate dataset, while external information evaluates quality through external measures, such as preferences from external llms or the impact of samples on model performance. ## selector evaluation to evaluate a selector's effectiveness, existing works compare a selective-enhanced model (sem), trained on a selected subset of the candidate dataset, against the same foundation model finetuned on the full dataset (baseline model, blm) or other popular llms using win-tie-loss metrics or benchmark scores. candidate datasets. most studies @xcite use open-source datasets as candidate sources for fine-tuning, aiming to improve performance by selecting higher-quality samples. these datasets can be broadly categorized into two types: (1) typical datasets, which serve as common instruction-tuning benchmarks, such as alpaca @xcite , dolly @xcite , and flan @xcite ; (2) advanced datasets, which are refinements of typical datasets to improve quality further, such as wizardlm @xcite , ultrachat @xcite , etc. counterpart models. to evaluate sem's effectiveness, most works compare it against a blm, quantifying the improvement achieved through data selection. popular llama-series models @xcite and mistral models @xcite are commonly used as backbones. additionally, some studies @xcite ) benchmark sem against sota models (e.g., gpt-4, claude, llama-chat 7b) to assess absolute performance, further validating the selector's effectiveness. evaluation metrics. the evaluation metric includes both relative and absolute aspects. relative metrics, like win-tie-loss ratios judged by gpt-4, reflect sem's performance compared with its counterpart models. absolute metrics, instead, utilize either the model's response loss on test sets (mmlu) or gpt-4's score on specific benchmarks to evaluate the sem performance (mt-bench). ## discussions 6.1 trends figure 6 illustrates the current research trends, where we have categorized the existing work following three key aspects: candidate datasets, quality measurement, and selected features, arranged chronologically. based on this, we can identify three main trends: quality measurement pfm loss information llm score specific selected feature candidate dataset general timeline alpagasus instag superfiltering ifd less mods shed car deita instruction-mining instructiongpt-4 nuggets selectit jul 2023 aug nov dec feb 2024 apr model optimize indicator quality score indicator multi-dimensions indicator pfm loss indicator ## conclusion in this paper, we present a fine-grained survey of a dozen data selection methods for fine-tuning large-scale language models and establish a threestage data selection scheme to standardize the process. to address inconsistencies across different experimental setups, we also introduce a unified comparison approach using ratio-based and rankingbased indicators to quantify efficiency and assess feasibility, which has been overlooked in previous works. our comprehensive evaluations, considering both efficiency and feasibility, reveal that methods with more targeted designs tend to achieve higher efficiency, though often at the cost of feasibility. additionally, we outline the progression of existing research and identify four critical challenges for future work: balancing efficiency and flexibility, ensuring measurement objectivity, improving specific tasks/domain performance without compromising others, and satisfying multiple goals in data selection. these challenges point to key areas for future research, particularly in refining sample quality measurement techniques.