Stratified Selective Sampling for Instruction Tuning with Dedicated Scoring Strategy (2025)

Abstract: Recent work shows that post-training datasets for LLMs can be substantially downsampled without noticeably deteriorating performance. However, data selection often incurs high computational costs or is limited to narrow domains. In this paper, we demonstrate that data selection can be both—efficient and universal—by using a multi-step pipeline in which we efficiently bin data points into groups, estimate quality using specialized models, and score difficulty with a robust, lightweight method. Task-based categorization allows us to control the composition of our final data—crucial for finetuning multi-purpose models. To guarantee diversity, we improve upon previous work using embedding models and a clustering algorithm. This integrated strategy enables high-performance fine-tuning with minimal overhead.

## introduction large language models (llms) can perform a wide range of text-based tasks through chat interfaces. their generalist abilities stem from an extensive post-training phase, during which they are optimized to generate useful responses to user queries @xcite @xcite . the choice of training data in this phase has a major impact on model performance (e.g. @xcite @xcite @xcite . prior work identifies three key properties of effective instruction-tuning (it) data @xcite : i) difficulty/relevance, reflecting how much a query contributes to learning @xcite @xcite ; ii) quality, the usefulness and accuracy of responses @xcite @xcite ; and iii) diversity, the scope of withindomain variability and cross-domain coverage of 𝒟 deita quality @xcite math prm @xcite if-quality @xcite code-quality instructions are first classified into seven types, then routed and scored for difficulty (f ) and quality (q). their combined scores (p) are ranked, and top examples are sampled both within embedding clusters and overall, while preserving fixed category proportions. the data @xcite . although many approaches address one or more of these aspects, they often face limitations that make them difficult to apply in practice. most existing methods cover only a subset of criteria (e.g., diversity in @xcite . to our knowledge, deita @xcite is the only complete pipeline that integrates all three simultaneously. beyond completeness, prior work can be further refined in terms of generality and robustness. many heuristic-based methods cannot be applied to different tasks @xcite or domains @xcite . broader approaches-that do not limit themselves explicitly to a single domainare usually evaluated only on narrow domains or task-types (e.g., open-ended language generation), fine-tuned on a small set of models (typically from the llama or mistral family with 3b-8b parameters), or limited to a single source dataset. outside these settings, they often prove brittle, failing to consistently outperform random sampling or simple heuristics such as a longest baseline @xcite . ultimately, many methods are expensive as they rely on large or closed-weight llm judges @xcite . we address these limitations with laser (label-aware scoring and clustering), a robust, complete, and efficient data selection pipeline. 1 our key contributions are the following: (i) a novel, complete and robust data sampling strategy for instruction-tuning that integrates efficient scoring of difficulty and quality via routing while maintaining both inter-and intra-class diversity through stratified sampling and clustering; on the other hand, model-inherent criteria form another group of approaches that rely on signals from the target model itself, including: less @xcite , which estimates data influence using gradient information; selectit @xcite , which measures uncertainty via token probability, prompt variation and multiple models' assessments; shed @xcite , which clusters data and estimates impact using shapley values; instruction-following difficulty (ifd; @xcite , measuring discrepancies between the model's intrinsic generation capability and its desired response; and the approach by @xcite , which evaluates sample utility based on how much it reduces loss when used as an incontext example. our approach embraces external and modelinherent perspectives simultaneously: we use domain-specific llm-scorers to assess quality, and leverage various models' performances across benchmarks to estimate difficulty-that is, how likely an average model is to fail on a given instruction. for diversity, our approach closely follows car @xcite , which clusters data points and samples one representative from each cluster. ## methods our goal is to determine a subset of an arbitrarily large instruction source dataset that is effective when finetuning a given pre-trained base model. we define an effective dataset as one which achieves high performance on a large and general set of evaluation tasks, while requiring relatively few model parameter updates. formally, let @xmath0, y i ) be the full source dataset of size n , where x i represents an input sequence (i.e. an instruction) and y i represents the corresponding output (i.e. the desirable model response). we wish to select a subset d ′ ⊂ d of size m (with m ≪ n ) that is most effective for instruction tuning. previous research has shown that i) instruction difficulty (see e.g. @xcite @xcite ii) response quality (see e.g. @xcite @xcite and iii) diversity/composition (see e.g. @xcite are crucial for effective data selection for llm finetuning. we address these three aspects in the three-step laser pipeline as illustrated in figure 1: 1. classification. we train a lightweight classifier π c to categorise all inputs x i in d into one out of seven categories, which we denote as @xmath1. 2. scoring. each input-output pair (x i , y i ) is scored using a general-purpose difficulty scorer (yielding f i ) and a category-specific quality scorer (yielding q i ); the results are combined into an overall preference score p i . 3. clustering + ranking. ultimately, we select the samples with the highest p i while using a clustering approach to maintain diversity and minimise redundancies in d ′ . we detail each step in the remainder of this section. ## discussion and conclusion effective data selection for instruction tuning is increasingly important as we handle the increasing amount of available data of mixed quality. nevertheless, the results of data selection pipelines have often been brittle, struggling to generalise across training setups @xcite . in this paper, we address this challenge by explicitly covering the whole spectrum of possible instruction domains and tailoring scoring strategies that are apt to judge the utility of samples in those domains. we show the robustness of our approach by testing it across diverse settings (model families, scales, various shifts in the source data distribution) and evaluating it on a wide range of common benchmarks. our experiments provide further evidence for the necessity of data selection, as our sampling not only significantly reduces the amount of data required for training, but also outperforms setups trained on the full source data. despite its more elaborate design-including multiple scoring methods-our pipeline remains computationally efficient due to targeted scoring. while our pipeline outperforms baselines in almost all settings, it shows only little improvements when sampling large portions from weak source data (as shown in section 4.2.2). this may reflect a potential ceiling effect on possible performance given the weak source material, suggesting that improving the data itself-rather than relying solely on elaborate sampling techniques-might be necessary. in such cases, iterative data refinement seems promising, especially when guided by reliable scorers for assessing sample difficulty and quality. the modular setup of laser allows for further incremental improvements of different scorers in future work.