MIG : Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space (2025)

Abstract: Data quality and diversity are key to the construction of effective instruction-tuning datasets. With the increasing availability of open-source instruction-tuning datasets, it is advantageous to automatically select high-quality and diverse subsets from a vast amount of data. Existing methods typically prioritize instance quality and use heuristic rules to maintain diversity. However, this absence of a comprehensive view of the entire collection often leads to suboptimal results. Moreover, heuristic rules generally focus on distance or clustering within the embedding space, which fails to accurately capture the intent of complex instructions in the semantic space. To bridge this gap, we propose a unified method for quantifying the information content of datasets. This method models the semantic space by constructing a label graph and quantifies diversity based on the distribution of information within the graph. Based on such a measurement, we further introduce an efficient sampling method that selects data samples iteratively to Maximize the Information Gain (MIG) in semantic space. Experiments on various datasets and base models demonstrate that MIG consistently outperforms state-of-the-art methods. Notably, the model fine-tuned with 5% Tulu3 data sampled by MIG achieves comparable performance to the official SFT model trained on the full dataset, with improvements of +5.73% on AlpacaEval and +6.89% on Wildbench.

## introduction large language models (llms) have shown remarkable capabilities in following human instructions in a wide range of tasks @xcite . typically, llms first acquire general knowledge through large-scale pretraining and are subsequently refined through instruction tuning to better align with diverse human intentions @xcite @xcite on the tulu3 @xcite pool using llama3.1-8b @xcite , evaluated on (black) knowledge-based benchmarks and (red) human-preference benchmarks. see details in sec. 4.2. 2020; @xcite . instruction tuning utilizes instruction-response pairs to guide base models toward more accurate and contextually appropriate responses. recent studies @xcite emphasize the critical role of data engineering in instruction tuning, highlighting data quality rather than quantity as the key to effective instruction tuning. notably, lima @xcite demonstrates that just 1000 high-quality, human-curated instructions can achieve performance comparable to substantially larger datasets. however, manual curation of such datasets is inherently time-consuming and labor-intensive @xcite . more recently, a line of work @xcite @xcite proposes automatic selections of optimal subsets from extensive data pools by defining desirable data characteristics. these approaches @xcite ) posit that quality and diversity are crucial for an effective instructiontuning dataset. data quality is defined from multiple perspectives, such as instruction complexity @xcite , model perplexity and uncertainty @xcite , or scores assigned by advanced external models @xcite . however, diversity remains less explicitly quantified, often addressed via heuristic methods such as maximizing label set coverage @xcite , reducing redundancy through diversity filters @xcite , or enforcing fixed sample distributions per cluster @xcite . this narrow focus on diversity only during later selection stages, without a comprehensive view of the entire dataset, diminishes the global diversity and representativeness of the sampled data. alternative methods @xcite employing embedding-based facility location functions @xcite quantify diversity but require computationally intensive iterative pairwise distance calculations, making them impractical for large datasets. additionally, distance-based clustering in the embedding space may fail to capture the semantic intent of complex instructions accurately. to solve these issues, several essential questions are raised: 1) how can we effectively quantify diversity in semantic space while balancing quality and diversity in dataset evaluation? 2) how can we efficiently select data based on such evaluations? to this end, we propose an information-based measure for instruction-tuning datasets and introduce an efficient data selection algorithm that aims to maximize the information gain (mig). we model the semantic space as a label graph, with nodes representing labels and edges capturing semantic relationships. information in the dataset is distributed across this graph, with the total information being the sum of each label's information. each data point contributes to its associated labels in proportion to its quality. thus, the information of each data point measures local data quality, while the total of all label information measures the global diversity of the dataset. to balance quality and diversity, we apply a monotonically increasing but marginally diminishing function to compute label information, thereby promoting diversity and preventing excessive data concentration on particular labels. to better model information distribution in semantic space, we propagate information along label graph edges to address semantic correlations and annotation biases. leveraging the submodu-larity of our proposed information-based dataset measurement, we implement an efficient greedy algorithm that iteratively selects data points that maximize the information gain according to the current state of the label graph. through extensive experiments across data pools @xcite @xcite of varying quality and sizes, and llms of different families @xcite @xcite , mig consistently achieves superior performance on both humanpreference and knowledge-based evaluations. as shown in fig 1 , on the tulu3 @xcite pool with llama3.1-8b as the base model, mig achieves average improvements of +1.49% on six knowledge-based benchmarks @xcite @xcite @xcite and +1.96% on three human-preference benchmarks @xcite @xcite compared to previous state-of-the-art data selection methods @xcite . when combining both evaluations, mig achieves average improvements of +2.20% compared to the second-best method @xcite . notably, the model fine-tuned with 5% tulu3 data sampled by mig outperforms the official sft model trained on the full dataset by +1.73% (average on nine benchmarks), with a substantial boost of +4.59% in human-preference evaluations. mig also outperforms existing methods on the openhermes2.5 @xcite and x sota @xcite , further demonstrating its generalizability across different settings. additionally, mig significantly enhances sampling efficiency, reducing sampling time by over 100-fold on the tulu3 data pool compared to embeddingbased methods. in summary, our contributions are as follows: ## method as shown in fig. 2 (a), we begin by annotating the raw data pool with a tagger and scorer. next, mig constructs a label graph to measure dataset information (sec. 3.2) and selects a subset for subsequent sft by maximizing the information gain (sec. 3.3).