A ce M ath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling (2025)

Abstract: In this paper, we introduce AceMath, a suite of frontier math models that excel in solving complex math problems, along with highly effective reward models capable of evaluating generated solutions and reliably identifying the correct ones. To develop the instruction-tuned math models, we propose a supervised fine-tuning (SFT) process that first achieves competitive performance across general domains, followed by targeted fine-tuning for the math domain using a carefully curated set of prompts and synthetically generated responses. The resulting model, AceMath-72B-Instruct greatly outperforms Qwen2.5-Math-72B-Instruct, GPT-4o and Claude-3.5 Sonnet. To develop math-specialized reward model, we first construct AceMath-RewardBench, a comprehensive and robust benchmark for evaluating math reward models across diverse problems and difficulty levels. After that, we present a systematic approach to build our math reward models. The resulting model, AceMath-72B-RM, consistently outperforms state-of-the-art reward models. Furthermore, when combining AceMath-72B-Instruct with AceMath-72B-RM, we achieve the highest average rm@8 score across the math reasoning benchmarks.

## introduction over the past year, the open large language model (llm) community has made remarkable progress in advancing the key capabilities of llms, including multi-turn conversation @xcite , coding @xcite , multimodal functionalities @xcite , retrieval-augmented generation (rag) @xcite , and mathematical reasoning @xcite @xcite . among these capabilities, mathematics is recognized as a fundamental aspect of intelligence. it can serve as a reliable benchmark due to its objective, consistent, verifiable nature. consequently, solving math problems is widely regarded as a critical testbed for evaluating an llm's ability to tackle challenging tasks that require complex, numerical and multi-step logical reasoning (e.g., @xcite . previous studies have convincingly demonstrated that math-specialized llms significantly outperform general-purpose llms on challenging mathematical benchmarks @xcite @xcite . these mathspecialized models, including the corresponding reward models (a.k.a. verifiers), are not only valuable to the mathematics and science communities (e.g., @xcite ), but they also provide valuable insights into data collection and serve as synthetic data generation tools, contributing to the advancement of future iterations of general-purpose llms. in this work, we push the limits of math reasoning with post-training and reward modeling based on open weights base llms and math base llms. we establish state-of-the-art supervised fine-tuning (sft) and reward modeling (rm) processes for building math-specialized models, while also sharing key insights gained from our comprehensive studies. our contributions are as follows: ## reward model training we train a math reward model for acemath-instruct, aiming to select more accurate solutions and better reasoning paths. to ensure broad applicability across a variety of language models, we curate a diverse training dataset. the following sections detail our training methodology, evaluation protocols, and empirical results. we then annotate the model solutions as correct or incorrect by comparing them against the referenced labels using the qwen-math evaluation toolkit. 1 this process initializes a pool of correct and incorrect candidate responses for each problem, which we treat as positive and negative samples that can be further sampled to create paired responses for training. ## conclusion in this work, we present acemath, a series of frontier-class math instruct and reward models. we demonstrate that our acemath-7b-instruct significantly surpasses the previous best-in-class qwen2.5-math-7b-instruct across comprehensive math reasoning benchmarks, and it performs slightly worse than a 10× larger qwen2.5-math-72-instruct (67.2 vs. 68.2). remarkably, our acemath-72b-instruct greatly outperforms qwen2.5-math-72-instruct, gpt-4o and claude-3.5 sonnet. additionally, we construct acemath-rewardbench, a comprehensive benchmark designed to evaluate math reward models across a diverse range of datasets and difficulty levels. we show that our acemath-72b-rm consistently outperforms state-of-the-art reward models, including qwen2.5-math-rm-72b on various math reward benchmarks. furthermore, when combining acemath-72b-instruct with acemath-72b-rm, we achieve the highest average rm@8 score across the math reasoning benchmarks. to advance open research in the field, we will open source both acemath-instruct and acemath-rm, along with the complete training data used throughout their development.