F ast C u RL : Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models (2025)

Abstract: Improving training efficiency continues to be one of the primary challenges in large-scale Reinforcement Learning (RL). In this paper, we investigate how and influence the RL scaling training process of R1-distilled reasoning models, e.g., DeepSeek-R1-Distill-Qwen-1.5B . Our experimental results reveal that: text-green (1) simply controlling the context length and selecting the training data based on the input prompt length can effectively improve the training efficiency of RL scaling, achieving better performance with more concise CoT; text-blue (2) properly scaling the context length helps mitigate entropy collapse; text-red and (3) carefully choosing the context length facilitates achieving efficient LLM training and reasoning . Inspired by these insights, we propose FastCuRL , a curriculum RL framework with stage-wise context scaling to achieve efficient LLM training and reasoning. Extensive experimental results demonstrate that FastCuRL-1.5B-V3 significantly outperforms state-of-the-art reasoning models on five competition-level benchmarks and achieves 49.6% accuracy on AIME 2024. Furthermore, FastCuRL-1.5B-Preview surpasses DeepScaleR-1.5B-Preview on five benchmarks while only using a single node with 8 GPUs and a total of 50% of training steps.

## introduction large language models (llms) have emerged as immensely potent ai instruments, showcasing extraordinary proficiency in comprehending natural language and executing downstream tasks @xcite @xcite . lately, test-time scaling @xcite has demonstrated a robust correlation between extending the generation length of chain-of-thought (cot) @xcite and improving the reasoning capabilities of llms. a primary finding from recent breakthroughs, exemplified by @xcite , have revealed a notable scaling phenomenon in the training process of reinforcement learning (rl). inspired by these findings, scaling rl for training llms has emerged as a promising paradigm for tackling complex reasoning tasks. this has sparked numerous valuable research efforts aimed at exploring and replicating reasoning models similar to deepseek-r1 @xcite @xcite . these initiatives typically begin with r1-distilled or pre-trained models and extend the generation length of cot reasoning to enhance performance. however, excessively long cot responses significantly increase computational overhead during model training and deployment. moreover, recent studies @xcite @xcite have identified an inherent overthinking phenomenon in reasoning models, which includes irrelevant details and repetitive thinking patterns. this kind of information leads to inefficient use of computational resources and undermines reasoning accuracy, which causes models to stray from valid logical pathways, resulting in incorrect answers. to this end, recent studies @xcite 2025; @xcite focus on efficient reasoning for optimizing the model to generate more concise cot outputs. among them, deepscaler @xcite propose to iteratively increase the context length from 8k to 24k to train the deepseek-r1-distill-qwen-1.5b model toward more concise reasoning, outperforming openai's o1-preview @xcite . by observing the training logs foot_1 of deepscaler in figure 2 , we find two issues: the prior work and the aforementioned issues naturally motivate two research questions: to this end, in this paper, we investigate how the model's context length and the complexity of the training dataset influence the training process of r1-like reasoning models. motivated by our observations, we propose fastcurl, a simple yet efficient curriculum reinforcement learning framework with a stage-wise context scaling strategy to improve the rl training efficiency and achieve concise cot. specifically, the proposed method alternates between cot compression (long-to-short) and extension (short-to-long). we first compress cot reasoning outputs, then extend them, and repeat this compress-extend cycle. the main goal of our method is to enhance cot quality progressively. we hypothesize that this dynamic interplay fosters higher-quality reasoning by iteratively refining essential logical dependencies while pruning redundant or spurious steps. meanwhile, during the iterative process, we control the complexity of the training samples, thereby ensuring training efficiency. extensive experimental results demonstrate that our model fastcurl-1.5b-v3 outperforms recent state-of-the-art reasoning baselines across five competition-level benchmarks, aime 2024, amc 2023, math 500, minerva math, and olympiad-bench. furthermore, our model fastcurl-1.5b-preview surpasses deepscaler-1.5b-preview on five competition-level benchmarks and only uses 50% training steps on a single node with 8 gpus. we hope the findings presented in this paper, the models we have released, and the open-sourced code will benefit future research. ## methodology our method for achieving efficient llm reasoning integrates three main components: (1) a resourceefficient rl algorithm, (2) a complexity-aware, mathematics-focused dataset, and (3) a stage-wise context scaling approach. this section provides a detailed description of the first two components. ## experiments we designed a set of experiments, constrained by available computational resources, to investigate our central research question (section 1): how do model context length and training data complexity affect the rl training of r1-like reasoning models? by analyzing the training behavior of small llms, we aim to extract practical insights. the objective is to move beyond simply providing empirical evidence of performance gains and to offer clear, actionable guidance for both academic and industrial implementation. ## related work advancements in rl methodologies have considerably enhanced the reasoning capabilities of llms. a pivotal development in this domain is openai's o1 (openai, 2024), which employs rl training to promote the development of long cot reasoning in llms. this approach has significantly enhanced performance on complex mathematical and programming benchmarks. building upon this foundation, deepseek-r1 (deepseek-ai, 2025) demonstrates that pure rl post-training via group reinforcement policy optimization (grpo), without needing supervised pre-training, can directly perform robust cot reasoning capabilities. notably, this method not only achieves performance competitive with o1 but also exhibits emergent behaviors such as self-verification and multi-step planning. building on these advancements, the research community has been collectively working to study and apply the methodology of deepseek-r1 to enhance the cot reasoning capabilities of various sizes of language models, yielding remarkable progress, such as face (2025 @xcite . different from the existing approaches, we employ an iterative stage-wise scaling method that alternates between cot compression (long-to-short) and cot extension (short-to-long) training stages. we initially compress the cot outputs, subsequently perform extension, followed by re-compression and re-extension in a cyclical manner. specifically, we hypothesize that this dynamic interplay between compression and extension facilitates the emergence of higher-quality reasoning responses by iteratively refining the essential logical dependencies while eliminating redundant or spurious cot reasoning outputs. ## conclusion we investigate how the model's context length and the complexity of the training dataset influence the training process of r1-like reasoning models. motivated by our findings, we propose fastcurl, a simple yet effective curriculum reinforcement learning framework incorporating a stage-wise context scaling strategy. this method is designed to accelerate the training efficiency and improve the model's long cot reasoning capabilities. experimental results demonstrate that fastcurl-1.5b-preview achieves better performance and reduces computational resource consumption by more than 50%, with all training phases efficiently executed using a single node with 8 gpus.