Towards A Better Initial Policy Model For Scalable Long- C o T Reinforcement Learning (2025)

Abstract: Long-CoT reasoning combined with reinforcement learning for large language models demonstrates remarkable performance and scalability. However, we observe that the initial policy model could significantly influence the final performance as well as the token efficiency. Additionally, there is a lack of systematic guidelines for obtaining a better initial policy model. To bridge this gap, we initiate a comprehensive investigation by activating the initial model using a variety of datasets with different data volumes and reasoning patterns. Then, we conduct a thorough analysis and comparison of the RL process for different initial models from the perspectives of upper bounds, diversity, and token efficiency, providing a deeper understanding and insight into the long-CoT RL. Based on our empirical results, we propose a systematic guideline and a novel Re-RFT method for constructing a better RL start point. Our experiment results based on the 14B model surpass the DeepSeek-R1-Distill-Qwen-14B by an average of 4.6%, demonstrating our approach’s effectiveness and superiority.

## introduction with the rapid advancement of artificial intelligence, the complex reasoning abilities of large language models @xcite @xcite , such as mathematical reasoning @xcite and code generation @xcite , have garnered significant attention. to enhance these capabilities, a powerful technique, testtime scaling has been developed, where the model using more tokens during inference has a significant improvement in performance. there are currently two lines of research to implement test-time scaling. one method, based on † project lead. ‡ corresponding author. ## delving into scalable rl in this section, we conduct a comprehensive and in-depth analysis of the influence of the initial policy model activated with different datasets during the reinforcement learning process. we argue that beyond the final accuracy, it is essential to consider additional aspects such as pass@k (section 3.1), diversity (section 3.2), and token efficiency (section 3.3). these metrics can provide more valuable insights of the long-context reinforcement learning process. experimental setup our objective is to investigate the impact of varying quantities and patterns of data on the model's reinforcement learning process, thus we avoid using excessively large datasets, as they could hinder the ease of our exploration. to enhance the model's reasoning capabilities, we have compiled a activation dataset and rl prompt set comprising level-4 and level-5 problems from math @xcite , aime 1983-2023, and a portion of omni-math @xcite . for sft activation, we conduct our study using three different initial reasoning patterns. the first is the traditional cot generated by qwen-2.5-math @xcite . additionally, we introduce two long-cot variants derived from o1mini and r1. since o1 does not publicly provide its hidden cot, we extract its reasoning pattern using prompt engineering, inspired by the think claude * . for the vanilla model, we have selected qwen-14b-instruct @xcite as well as deepseek-r1-distill-qwen-14b (deepseek-ai et al., 2025) activated with 800k r1-generated data for further validation. in constructing the test set, we have chosen four complex mathematical reasoning benchmarks: math-500 @xcite , omni-math-500 @xcite , aime 2024, and 10 problems from aimo 2024 @xcite . given the limited data available for aime 2024 and aimo 2024, we let the model repeat sampling 10 times and take the average result to ensure a robust evaluation. we have performed string-level matching detection to prevent the data leakage. the detailed data statistics are shown in ## towards a better initial policy model in the previous section, we analyzed the impact of different pattern-activated initial policy models from various perspectives. we also explored what is a good initial policy model for reinforcement learning. specifically, the one with higher pass@k, greater diversity at the early training stage, and better token efficiency during the rl training. based on the observations and conclusions, we try to explore the optimal activation strategy by two key aspects: the quantity of activation data and the reasoning pattern for the policy model. ## related work olympiad-level reasoning for llm with the release of openai o1 series models (openai, 2024), which demonstrated the ability of large language models to handle olympiad-level reasoning tasks, there has been a significant interest in olympiad-level reasoning in recent studies. currently, there are two main technical approaches. the first involves verifier and search @xcite @xcite , where the model repeatedly samples during inference and relies on a verifier to select the highest-quality fi-nal result or reasoning steps, thereby enabling testtime scaling for better performance. the second approach utilizes long-cot combined with rl. previous studies have focused on short-cot @xcite , but with the emergence of long-cot models such as kimi k1.5 @xcite and r1 @xcite , increasing research has highlighted the advantages of long-cot in tackling complex reasoning tasks @xcite . moreover, by incorporating rl, the number of completion tokens of long-cot can be further expanded, enhancing the model's overall capabilities. however, current research has not fully explored why long-cot outperforms short-cot. in section 3, we compare the initial policy models activated with different cot patterns and analyze them from different perspectives, which provide more valuable insight into this question for future studies. reinforcement learning for llm since the introduction of rlhf @xcite , the combination of llms and reinforcement learning has proven to be highly effective, particularly in tasks such as aligning with human preferences, mathematical reasoning, and code generation. recently, long-cot combined with rl @xcite @xcite @xcite has demonstrated powerful capabilities in solving complex problems. the primary pipeline involves first activating the initial policy model through long-cot activation, followed by policy optimization methods such as grpo @xcite or reinforce @xcite , combined with rule-based rewards. however, there is currently a lack of systematic research on long-cot rl and the initial policy model. ## conclusion our work presents the first systematic investigation into the critical role of the initial policy model in long-cot reinforcement learning process. we offer an in-depth analysis of long-cot rl using metrics beyond performance, providing insights into what constitutes a good initial policy model. additionally, we investigate a recipe for a better initial policy model in terms of data volume. we also introduce a simple yet effective re-rft method to obtain more efficient reasoning patterns. we hope our work provides valuable insights into scalable reinforcement learning for future work.