D ebate4 MATH : Multi-Agent Debate for Fine-Grained Reasoning in Math (2025)

Abstract: Large language models (LLMs) have demonstrated impressive performance in reasoning. However, existing data annotation methods usually suffer from high annotation cost and the lack of effective automatic validation. To address these issues, we propose a Fine-grained Multi-Agent Debate framework (FMAD) and MMATH-Data, a dataset created by FMAD, which consists of 46K reasoning steps. By prompting multiple agents to debate, FMAD assesses the contribution of each reasoning step to the final solution, with labels based on the judge’s confidence score and the winner’s position. To facilitate reasoning in math and examine FMAD and MMATH-Data, we further propose two key components: a Multi-Agent Debate Reward Model (MRM) trained on MMATH-Data, which serves as a reward model to provide robust feedback during the optimization process, and MMATH-LLM, a model designed specifically for mathematical reasoning. MMATH-LLM is fine-tuned using reinforcement learning with supervised feedback from MRM, aiming at improving its mathematical reasoning capabilities. Extensive experiments demonstrate that our model achieves 83.4% accuracy on the GSM8K dataset and 45.1% on the MATH dataset, outperforming the state-of-the-art methods by 1.2% and 3.5%, respectively. All data and code will be available soon at GitHub.

## introduction large language models, exemplified by @xcite and gpt-o1, 2 have demonstrated exceptional performance across a wide range of tasks. however, even the most advanced llms could generate incorrect reasoning paths when solving complex multi-step mathematical problems @xcite @xcite . to improve advanced reasoning, a variety of methods have been proposed from two perspectives: creating step-wise reasoning data and developing step-wise reasoning verifiers. however, these methods usually suffer from two critical challenges: constructing high-quality training data and designing reward models capable of guiding policy refinement and improving data quality. although existing approaches, such as automated annotation techniques and reward methods @xcite @xcite , have made progress, significant gaps persist. creating of high-quality data is fundamental for advancing the reasoning capabilities of llms @xcite @xcite , yet it presents a critical dilemma. on one hand, human-annotated data remains the gold standard for complex reasoning tasks requiring specialized skills @xcite , but its resourceintensive nature severely limits scalability. on the other hand, automated synthesis methods, such as tree/graph-based expansions @xcite , offer promising alternatives but introduce new challenges: they necessitate extensive human validation to maintain quality, and their reliability degrades with increasing complexity. this fundamental tension between quality, scalability, and annotation cost underscores the need for more efficient data construction methodologies. complementary to data creation, reward models play a pivotal role in language model training by providing essential feedback for reinforcement learning (rl) policies and synthetic data quality supervision @xcite @xcite @xcite . however, conventional reward models, such as those in used in proximal policy optimization (ppo) @xcite , exhibit significant limitations. they require additional supervision to ensure accurate and consistent feedback, while inconsistent feedback could degrade model per-formance or reduce the quality of synthetic data. these challenges highlight the need for more robust and scalable reward models. to alleviate the aforementioned issues with reasoning data construction and reasoning-tailored reward models, we propose fmad, a fine-grained multi-agent debate framework that leverages a multi-round debate process to create high-quality reasoning data, and mrm, a multi-agent debate reward model, designed to enable effective selfregulation through agent debates. specifically, in fmad, two debaters engage in a multi-round debate on a given problem, while a judge evaluates their arguments. fmad is structured into three steps. 1) role definition: two debaters and one judge are defined. the debaters argue the correctness of each reasoning step, while the judge evaluates the debate by reviewing their debate history and assigns a score to the winner. 2) debate initiation: the debate begins with one debater presenting supporting evidence for the correctness of a reasoning step, followed by multiple rounds of argumentation. 3) decision making: the judge reviews the debate history, selects a winner, and assigns a confidence score (0% to 100%) to indicate the likelihood of the chosen reasoning step being correct. for the reward model, mrm employs two debaters to evaluate the validity of reasoning steps through structured debates, with a judge generating reward signals based on the debate outcomes. it enhances the training process by providing multi-faceted feedback for both reasoning assessment and policy optimization. with mrmgenerated rewards, we are able to mitigate annotation bias and improve feedback reliability, thereby advancing model performance. in order to validate mrm, we further propose mmath-llm, a language model specialized in mathematical reasoning, where mrm delivers per-step feedback during policy training to refine reasoning capabilities. in a nutshell, our contributions are as follows: improving reasoning ability of llms with high-quality data. mathematical reasoning remains a significant challenge for large language models, where high-quality data is crucial for pretraining and fine-tuning. existing data construction approaches fall into three categories: humanannotated supervision, rule-based synthetic generation, and stochastic search-based methods. humanannotated datasets like prm800k (lightman et al., 2023) provide precise feedback but are costly and short of scalability. rule-based methods, such as backmath (zhang and xiong, 2025), automate data generation using predefined rules (e.g., backward reasoning chain) but suffer from producing biased outputs due to insufficient validation. stochastic search-based techniques, such as monte carlo tree search (mcts) (kocsis and szepesvári, 2006; wang et al., 2024), explore reasoning paths probabilistically but require high sampling density for accurate probability estimation, leading to computational inefficiency and biased results. these methods collectively face trade-offs between cost, scalability, and reliability. to address these limitations, we propose an automated data construction framework that integrates cost-effective data generation with self-verification, eliminating human intervention while ensuring data quality. reasoning verification for llms. beyond enhancing reasoning through data or prompting, verifying training processes or outputs has emerged as a key approach to improving llms' reasoning capabilities. existing verification methods can be categorized into three paradigms: outcome-based, process-based, and hybrid supervision. outcomebased supervision evaluates final outcomes, as seen in the outcome reward model (orm) (ouyang et al., 2022). while simple, orm lacks granularity for intermediate steps, resulting in reward sparsity. process-based supervision addresses this by providing step-by-step feedback. process-supervision reward models (prms) (lightman et al., 2023; uesato et al., 2022) evaluates each reasoning step, enabling precise error correction. however, prms problem: the equation 𝑥𝑥 𝑥𝑥+1 + 𝑥𝑥 𝑥𝑥+2 ground-truth answer: 0, 3 2 , -2𝑖𝑖, 2𝑖𝑖 𝑺𝑺 𝟏𝟏 , … , 𝑺𝑺 𝒌𝒌-𝟏𝟏 : multiplying both sides by (𝑘𝑘 + 1)(𝑘𝑘 + 2), we get [𝑘𝑘(𝑘𝑘 + 2) + 𝑘𝑘(𝑘𝑘 + 1) = 𝑘𝑘𝑘𝑘(𝑘𝑘 + 1)(𝑘𝑘 + 2)]. this rearranges to the equation [0 = 𝑘𝑘𝑘𝑘^3 + (3𝑘𝑘 -2)𝑘𝑘^2 + (2𝑘𝑘 -3)𝑘𝑘, ]or [0 = 𝑘𝑘(𝑘𝑘𝑘𝑘^2 + (3𝑘𝑘 -2)𝑘𝑘 + (2𝑘𝑘 -3)). ] clearly 𝑘𝑘 = 0 is a root of this equation. -this is a math debate task… -there are three players:… -you are given a math problem and some solutions of it… -the debaters' goal is to win the debate -there are a fixed number of rounds. in each round… rely on costly human-annotated data, which limits their scalability. hybrid supervision integrates prm's step-level quality control with auxiliary rewards (e.g., instruction reward model (irm), which supervises quality of generated instruction. @xcite ) that supervise data quality from orthogonal angles (e.g., instruction quality). although this complementary approach mitigates single-source limitations, it introduces optimization conflicts requiring careful calibration. ## reasoning data creation our key interest is to generate high-quality reasoning steps and automatically verify the quality of labels through a systematic process. we hence propose fmad, an automated annotation framework designed to construct well-established reasoning steps. as illustrated in figure 1 , our framework eliminates the need for human annotation while maintaining the validity of each annotation. inspired by multi-agent debate (mad) @xcite , we define the mad process as involving three participants: debater a, debater b, and judge. debater a represents the position where the reasoning step is correct or beneficial to subsequent steps, while debater b represents the position where the reasoning step is incorrect or unhelpful to the overall reasoning. based on the debate history between the two debaters, the judge decides who the winner is. the winner's position serves as the label for the debated reasoning step, and the confidence score provided by the judge indicates the probability that the step is considered correct or incorrect. unlike @xcite , whose goal is to make agents reach a common final answer, our work focuses on each reasoning step, so the debate process should have a clear outcome. during the process of data creation, reasoning steps are directly extracted from the official solutions provided in the gsm8k and math datasets upon their release. in our fmad framework, two debaters engage in a structured debate over the correctness of reasoning steps, moderated by a judge. the process consists of three main stages: mmath -llm base model step 1: collect debate data from the fmad framework. step 2: training multi-agent reward model (mrm) with mmath-data. step 3: training mmath-llm with mrm using ppo. 1. we define the quality of a reasoning step as the confidence score c score assigned by the judge based on the debate history between debater a and debater b. to be specific, given a problem p , let @xmath0, s 2 , . . . , s n } be the sequence of reasoning steps in its solution and a be the final answer. before the automatic annotation process starts, each participant is prompted with a set of debate rules r and assigned with their respective role rules. during the annotation process, as shown in figure 1 , for a reasoning step s @xmath1, we input the step s k , the problem p and the previous reasoning steps s ′ = {s 1 , s 2 , . . . , s k-1 } into the debate framework. in each debate round, debater a presents its argument first, followed by debater b, who rebuts debater a's argument and then presents its own debate. the winner's position, l, serves as the label indicating whether the corresponding step is correct, while the confidence score c score denotes its probability. following @xcite , which demonstrates that pairing strong debaters with relatively weaker judges yields better performance than the converse scenario, we employ qwen2.5-14b-instruct @xcite as the debater and llama-3-8b-instruct as the judge. the debate process is conducted for @xmath2, as fewer rounds may limit discussion depth while more rounds increase computational overhead and risk of information redundancy. to mitigate potential architectural bias, we utilize the same model for both debaters, preventing the judge from favoring debaters with specific output patterns. the detailed ## methodology with the created dataset, we train the proposed mrm and mmath-llm. the training process consists of two stages: (1) the mrm training stage where the judge and debaters are trained using mmath-data, and (2) the rl stage where mmath-llm is fine-tuned on the math and gsm8k training datasets under mrm supervision. ## experiment we conducted extensive experiments and in-depth analyses to evaluate the proposed entire framework, including the automatic reasoning data creation framework fmad, the created dataset mmath-data, the reward model mrm and math-reasoningoriented mmath-llm. ## conclusion in this paper, we have presented fmad, mrm, mmath-data, and mmath-llm. fmad evaluates the contribution of each reasoning step in a problem using the multi-agent debate framework, while mrm, trained with mmath-data, supervises the rl process by providing step-wise feedback. mmath-data, constructed by fmad, supports mrm in guiding mmath-llm, a model designed specifically for mathematical reasoning, during the rl process. through extensive experiments on the gsm8k and math benchmarks, we demonstrate that mrm significantly enhances the rl process. mmath-llm, supervised by mrm, outperforms existing methods, achieving an accuracy of 83.4% on gsm8k and 45.1% on math.