We train our initial reward model based on the DeepSeekMath-Base 7B with a learning rate of 2e-5. For GRPO, we set the learning rate of the policy model as 1e-6. The KL coefficient is 0.04. For each question, we sample 64 outputs. The max length is set to 1024, and the training batch size is 1024. The policy model only has a single update following each exploration stage. We evaluate DeepSeekMath-RL 7B on benchmarks following DeepSeekMath-Instruct 7B. For DeepSeekMath-RL 7B, GSM8K and MATH with chain-of-thought reasoning can be regarded as in-domain tasks and all the other benchmarks can be regarded as out-of-domain tasks. The performance of open- and closed-source models with both chain-of-thought and tool-integrated reasoning on English and Chinese benchmarks shows that: 1) DeepSeekMath-RL 7B attains accuracies of 88.2% and 51.7% on GSM8K and MATH, respectively, utilizing chain-of-thought reasoning. This performance surpasses that of all open-source models in the 7B to 70B range, as well as the majority of closed-source models. 2) Crucially, DeepSeekMath-RL 7B is only trained on chain-of-thought-format instruction tuning data of GSM8K and MATH, starting from DeepSeekMath-Instruct 7B. Despite the constrained scope of its training data, it outperforms DeepSeekMath-Instruct 7B across all evaluation metrics, showcasing the effectiveness of reinforcement learning. Discussion In this section, we will share our findings in pre-training and RL experiments. Lessons Learnt in Pre-Training We first share our experience in pre-training. Unless otherwise specified, we will adhere to the training settings outlined in the Math Pre-Training section. It is worth noting that, when referring to the DeepSeekMath Corpus in this section, we use an 89B-token dataset from the second iteration of the data collection process. Code Training Benefits Mathematical Reasoning A popular yet unverified hypothesis suggests that code training improves reasoning. We attempt to offer a partial response to this, particularly within the mathematical domain: code training improves models' ability to do mathematical reasoning both with and without tool use. To study how code training affects mathematical reasoning, we experimented with two-stage training and one-stage training settings. For Two-Stage Training, we experimented with Code Training for 400B Tokens followed by Math Training for 150B Tokens. As a control, we also tried General Training for 400B Tokens followed by Math Training for 150B Tokens. For One-Stage Training, we tried Math Training for 150B Tokens, and Training on a mixture of 400B Code Tokens and 150B Math Tokens. The results demonstrate the downstream performance under different training settings. Code training benefits program-aided mathematical reasoning, both under the two-stage training and one-stage training settings. Under the two-stage training setting, code training alone already significantly enhances the ability to solve GSM8K and MATH problems using Python. Math training in the second stage yields further improvements. Interestingly, under the one-stage training setting, mixing code tokens and math tokens effectively mitigates the issue of catastrophic forgetting that arises from two-stage training, and also synergizes coding and program-aided mathematical reasoning. Code training also improves mathematical reasoning without tool use. Under the two-stage training setting, the initial stage of code training already results in moderate enhancements. It also boosts the efficiency of the subsequent math training, eventually leading to the best performance. However, combining code tokens and math tokens for one-stage training compromises mathematical reasoning without tool use. One conjecture is that DeepSeek-LLM 1.3B, due to its limited scale, lacks the capacity to fully assimilate both code and mathematical data simultaneously. ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning ArXiv papers are commonly included as a component of math pre-training data. However, detailed analysis regarding their impact on mathematical reasoning has not been extensively conducted. Perhaps counter-intuitively, according to our experiments, arXiv papers seem ineffective in improving mathematical reasoning. We experiment with models of different sizes, including DeepSeek-LLM 1.3B and DeepSeek-Coder-Base-v1.5 7B, using arXiv corpora that underwent varied processing pipelines. The corpora included MathPile, an 8.9B-token corpus with over 85% from scientific arXiv papers, and ArXiv-RedPajama, the entirety of arXiv LaTeX files with preambles, comments, macros, and bibliographies removed, totaling 28.0B tokens. In our experiments, we separately train DeepSeek-LLM 1.3B for 150B tokens and DeepSeek-Coder-Base-v1.5 7B for 40B tokens on each arXiv corpus. It seems that arXiv papers are ineffective in improving mathematical reasoning. When trained on a arXiv-only corpus, both models display no notable improvements or even deterioration across various mathematical benchmarks of different complexities employed in this study. These benchmarks include quantitative reasoning datasets like GSM8K and MATH, multiple-choice challenges like MMLU-STEM, and formal mathematics like miniF2F. However, this conclusion has its limitations and should be taken with a grain of salt. We have not yet studied the impact of arXiv tokens on specific math-related tasks not included in this research, such as informalization of theorems which is to convert formal statements or proofs to their informal versions. We also haven't studied the effect of arXiv tokens when combined with other types of data, or whether the benefits of arXiv papers would manifest themselves at a larger model scale. Thus, further exploration is required, which we leave for future studies. Insights of Reinforcement Learning Towards to a Unified Paradigm In this section, we provide a unified paradigm to analyze different training methods, such as SFT, RFT, DPO, PPO, GRPO, and further conduct experiments to explore the factors of the unified paradigm. Generally, the gradient with respect to the parameter of a training method can be written as a function of three key components: 1) Data Source, which determines the training data; 2) Reward Function, which is the source of the training reward signal; and 3) Algorithm, which processes the training data and the reward signal to the gradient coefficient that determines the magnitude of the penalty or reinforcement for the data.