ROSE : A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning (2025)

Abstract: Instruction tuning has underscored the significant potential of large language models (LLMs) in producing more human controllable and effective outputs in various domains. In this work, we focus on the data selection problem for task-specific instruction tuning of LLMs. Prevailing methods primarily rely on the crafted similarity metrics to select training data that aligns with the test data distribution. The goal is to minimize instruction tuning loss on the test data, ultimately improving performance on the target task. However, it has been widely observed that instruction tuning loss (i.e., cross-entropy loss for next token prediction) in LLMs often fails to exhibit a monotonic relationship with actual task performance. This misalignment undermines the effectiveness of current data selection methods for task-specific instruction tuning. To address this issue, we introduce ROSE, a novel Reward-Oriented inStruction data sElection method which leverages pairwise preference loss as a reward signal to optimize data selection for task-specific instruction tuning. Specifically, ROSE adapts an influence formulation to approximate the influence of training data points relative to a few-shot preference validation set to select the most task-related training data points. Experimental results show that by selecting just 5% of the training data using ROSE, our approach can achieve competitive results compared to fine-tuning with the full training dataset, and it surpasses other state-of-the-art data selection methods for task-specific instruction tuning. Our qualitative analysis further confirms the robust generalizability of our method across multiple benchmark datasets and diverse model architectures.

## introduction while large language models (llms) are widely recognized for their strong generalization capabilities, many fields require enhanced domain-specific performance, e.g., health monitoring @xcite , legal question answering @xcite , and mathematics tutoring @xcite . instruction tuning has emerged as a popular method for adapting foundation models to specialized tasks, which typically involves curating a high-quality training dataset. although recent advancements in open-source datasets and synthetic data generation have facilitated the generation of large training datasets, it is widely acknowledged that the quality of training data is more crucial than its quantity in instruction tuning @xcite . consequently, practitioners must carefully select high-quality data to enhance the model's capabilities for specific tasks. this challenge is further compounded by the complexity of domain-specific requirements and the black-box properties of llms, rendering it nearly infeasible for humans to manually select the most suitable training set. therefore, developing more effective data selection methods is becoming increasingly crucial for reducing training costs and efficiently optimizing instruction tuning for specific tasks. despite various methods proposed for instruction tuning data selection @xcite @xcite , identifying high-quality instruction tuning data for target tasks remains a significant challenge. existing methods typically design specific similarity metrics to select a set of candidates samples whose distribution aligns with that of the target task data. for instance, dsir @xcite uses n-gram feature similarity to enhance the selection process of relevant data samples. moreover, rds @xcite and less @xcite calculate the similarity on the model embedding and gradient space to capture task-specific semantics and model characteristics, respectively. despite their successes, these similarity based methods are fundamentally limited. our analysis reveals that these strategies hinge on empirical risk minimization (erm), selecting training data that mirrors the target task data distribution by minimizing training loss, particularly next-token prediction loss. however, it is widely acknowledged that next-token prediction loss often fails to accurately reflect a model's real-world performance (e.g., alignment degree with human preference, reasoning ability on complex math problem) on target task @xcite . this significant gap between the implicit theoretical foundation of similarity-based methods and practical task performance limits the effectiveness of these methods in instruction tuning for task-specific fine-tuning. in response to these insights, we introduce a novel reward-oriented instruction data selection (rose) method in figure 1 , which shifts the intrinsic selection objective from minimizing validation cross entropy loss to maximize the reward for the target task. inspired by direct preference optimization (dpo) @xcite , our approach utilizes a few-shot set of pairwise samples as the task-specific preference validation set, which we assume reflects the desired llm's performance on the target task, and we use the dpo loss function to approximate the expected reward value of the trained llm on the preference validation data. by leveraging the gradient-based influence estimation techniques, rose is able to select the instruction tuning samples that leads a downstream llm with optimized performance on the target task. we conduct comprehensive experiments across various datasets and model architectures. experimental results show that our method outperforms existing similarity-based techniques, including token-wise, embedding-based, and gradient-based methods. these results suggest that focusing on reward maximization, rather than loss minimization, is a promising new direction for improving task-specific fine-tuning outcomes. the main contributions are as follows: ## rose: reward-oriented instruction data selection in this section, we introduce rose, a reward orientated data selection method for task-specific instruction tuning . we start from formulating the optimization objectives of rose, followed by a detailed explanation of implementation strategies. ## conclusion we propose rose, a novel instruction tuning data selection method based on influence estimation. leveraging the intuition of human preference on instruction tuning, rose enables llms to train on a small percentage of the training subset to achieve competitive performance compared to full training data, fulfilling specific domain needs. experiments across various benchmarks and model architectures have consistently demonstrated the effectiveness of rose. moreover, we provide empirical analysis and insights to solve the non-monotonic relationship between valida-tion loss and test accuracy or win rate in instruction tuning. our findings indicate that even with limited high-quality data selected by using rose, instruction tuning can lead to robust improvements in model performance.