Scale Down to Speed Up: Dynamic Data Selection for Reinforcement Learning (2025)

Abstract: Optimizing data utilization remains a central challenge in applying Reinforcement Learning (RL) to Large Language Models (LLMs), directly impacting sample efficiency, training stability, and final model performance.Current approaches often rely on massive static datasets, leading to computational inefficiency and redundant gradient updates.In this paper, we propose ScalingRL, a data-centric RL framework that dynamically selects the most informative training samples to optimize RL for mathematical reasoning.Specifically, ScalingRL introduces the Data Effectiveness Score (DES) that quantitatively ranks prompts according to three complementary factors: problem difficulty, Chain-of-Thought complexity, and reward adaptability.Then, ScalingRL employs an adaptive curriculum scheduler that progressively adjusts the overall scale and specific mix of training prompts—balancing exploration of new, challenging data with exploitation of previously learned concepts—thereby tailoring the data distribution to the model’s current learning trajectory and performance.Experimental results demonstrate that ScalingRL achieves comparable performance to full-data training methods while requiring only 1.5K samples instead of 220K, reducing training time from 13 days to just 4 hours on 8\times A800 GPUs.

## introduction recently, r1-like reasoning models @xcite have attracted significant attention for their unprecedented performance in mathematical reasoning, demonstrating spontaneous emergence of long chain-of-thought (cot) @xcite @xcite and self-reflection behaviors @xcite . the central technique driving this revolution is large-scale ## methodology in this section, we introduce our proposed scalin-grl framework. as illustrated in figure 2 , scal-ingrl consists of four key components: (1) initial candidate construction, which refines a diverse 220k mathematics dataset to a curated 8k candidate set through quality, diversity, and difficulty filtering; (2) dynamic data effectiveness scoring, a novel metric combining difficulty coefficient, chain-of-thought complexity, and reward adaptability to evaluate sample utility; (3) dynamic sampling strategy that implements temperaturecontrolled sample selection to balance exploration and exploitation; and (4) reinforcement learning with ppo algorithm and structured reward functions for model optimization. through this progressive refinement strategy, we significantly reduce training time from 13 days to 4 hours while maintaining model performance with 8×nvidia a800 gpus. ## conclusion in this paper, we present an efficient reinforcement learning framework for mathematical reasoning that focuses on maximizing training efficiency through dynamic data selection which contains a three-dimensional filtering process based on quality, diversity, and difficulty. we first refine initial dataset from 220k to 1.5k samples. then, we introduce a novel dynamic effectiveness score (des) for intelligent sample evaluation and selection during training. at last, we achieve substantial improvement in training efficiency. we expect our approach will benefit researchers in scenarios where computational resources are limited.