We also provide a unified paradigm to understand different representative training methods. Within this paradigm, all methods are conceptualized as either direct or simplified RL techniques. As summarized, there exist three key components: Data Source, Algorithm, and Reward Function. We provide some potential future directions about the three components. For the Data Source, which is the raw material of all training methods, we think this is a potential reason that our RL pipeline only improves the Maj@K performance. In the future, we will explore our RL pipeline on out-of-distribution question prompts, in conjunction with advanced sampling (decoding) strategies, like those based on tree-search methods. Also, efficient inference techniques, which determine the exploration efficiency of policy models, also play an exceedingly important role. For Algorithms, which process the data and reward signal to the gradient coefficient, to some extent, all methods now fully trust the signal of the reward function to increase or decrease the conditional probability of a certain token. However, it is impossible to ensure the reward signal is always reliable, especially in extremely complex tasks. For example, even the PRM800K datasets, which have been carefully annotated by well-trained annotators, still contain approximately 20% of incorrect annotations. To this end, we will explore the reinforcement learning algorithm that is robust against noisy reward signals. We believe such weak-to-strong alignment methods will bring a fundamental change to the learning algorithms. For the Reward Function, which is the source of the training signal, we think there exist three important directions for reward models: 1) How to enhance the generalization ability of the reward model. The reward model must be effectively generalized to handle out-of-distribution questions and advanced decoding outputs; otherwise, reinforcement learning may merely stabilize the distribution of LLMs rather than improve their fundamental capabilities. 2) How to reflect the uncertainty of the reward model. The uncertainty could potentially act as a linking bridge between the weak reward model and the weak-to-strong learning algorithms. 3) How to efficiently build high-quality process reward models that can provide fine-grained training signals for the reasoning process. Conclusion, Limitation, and Future Work We present DeepSeekMath, which outperforms all open-source models on the competition-level MATH benchmark and approaches the performance of closed models. DeepSeekMath is initialized with DeepSeek-Coder-v1.5 7B and undergoes continual training for 500B tokens, with a significant component of the training data being 120B math tokens sourced from Common Crawl. Our extensive ablation study shows web pages offer significant potential for high-quality mathematical data, while arXiv may not as beneficial as we expected. We introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), which can notably improve mathematical reasoning capabilities with less memory consumption. The experiment results show that GRPO is effective even if DeepSeekMath-Instruct 7B has reached a high score on benchmarks. We also provide a unified paradigm to understand a series of methods and summarize several potential directions for more effective reinforcement learning. Although DeepSeekMath achieves impressive scores on quantitative reasoning benchmarks, its capability on geometry and theorem-proof are relatively weaker than closed models. For instance, in our dry run, the model cannot handle problems related to triangles and ellipses, which may indicate data selection bias in pre-training and fine-tuning. In addition, restricted by the model scale, DeepSeekMath is worse than GPT-4 on few-shot capability. GPT-4 could improve its performance with few-shot inputs, while DeepSeekMath shows similar performance in zero-shot and few-shot evaluation. In the future, we will further improve our engineered data selection pipeline to construct more high-quality pre-trained corpus. In addition, we will explore the potential directions for more effective reinforcement learning of LLMs. Appendix  Analysis of Reinforcement Learning We provide the detailed derivation of the data source and gradient coefficient (algorithm and reward function) across various methods, including SFT, RFT, Online RFT, DPO, PPO, and GRPO. Supervised Fine-tuning The objective of Supervised Fine-tuning is maximizing the log probability of the target sequence. The Data Source is the dataset employed for SFT. The Reward Function can be regarded as human selection. The Gradient Coefficient is always set to 1. Rejection Sampling Fine-tuning Rejection Sampling Fine-tuning first samples multiple outputs from the supervised fine-tuned LLMs for each question, and then trains LLMs on the sampled outputs with the correct answer. The Data Source is a question in the SFT dataset with outputs sampled from the SFT model. The Reward Function is a Rule (whether the answer is correct or not). The Gradient Coefficient is 1 if the answer is correct, and 0 if it is incorrect. Online Rejection Sampling Fine-tuning The only difference between RFT and Online RFT is that the outputs of Online RFT are sampled from the real-time policy model, rather than from the SFT model. Direct Preference Optimization (DPO) The objective of DPO is to optimize based on preferences between pairs of outputs. The Data Source is a question in the SFT dataset with outputs sampled from the SFT model. The Reward Function is human preference in the general domain (or can be a 'Rule' in mathematical tasks). The Gradient Coefficient is a function of the log probabilities of the preferred and rejected responses under the policy and reference models. Proximal Policy Optimization (PPO) The objective of PPO is to maximize a clipped surrogate objective involving the ratio of probabilities between the new and old policies, weighted by the advantage function. The Data Source is a question in the SFT dataset with outputs sampled from the policy model. The Reward Function is a reward model. The Gradient Coefficient is the advantage, which is computed by applying Generalized Advantage Estimation (GAE), based on the rewards and a learned value function. Group Relative Policy Optimization (GRPO) The objective of GRPO is to maximize an objective based on group-relative advantages. The Data Source is a question in the SFT dataset with outputs sampled from the policy model. The Reward Function is a reward model. The Gradient Coefficient is the group-relative advantage plus a KL-divergence term for regularization, where the advantage is computed based on the group reward scores.