LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback (2025)

Abstract: In recent progress, mathematical verifiers have achieved success in mathematical reasoning tasks by validating the correctness of solutions generated by policy models. However, existing verifiers are trained with binary classification labels, which are not informative enough for the model to accurately assess the solutions. To mitigate the aforementioned insufficiency of binary labels, we introduce step-wise natural language feedback as rationale labels, that is, the correctness of each step and the detailed explanations. In this paper, we propose Math-Minos, a natural language feedback-enhanced verifier by constructing automatically generated training data and a two-stage training paradigm for effective training and efficient inference. Our experiments reveal that a small set of natural language feedback can significantly boost the performance of the verifier in both verification and reinforcement learning and also significantly alleviates the data-demanding problems of the reward model with an over 700% data efficiency improvement.

## introduction large language models have demonstrated strong capabilities in summarization @xcite , coding @xcite , tool using @xcite and dialogue @xcite . however, mathematical reasoning remains a challenge for llms @xcite . to tackle this problem, recent research has focused on using verifiers to validate the correctness of response generated by models @xcite @xcite ). an effective verifier can serve as 1) response reranker in the decoding @xcite @xcite ; 2) reward model in further post-training @xcite ; 3) data purifier that filters erroneous responses in the sft @xcite . however, existing verifiers @xcite @xcite are trained as classifiers by using binary labels as feedback. we argue that the binary feedback are not informative in training as they do not contain explanations for the underlying reasons for the errors, which hindered the performance of the verifier and also cause the data demanding problem during the verifier training process. in this work, we aim to enhance the verifier's evaluation ability for mathematical solutions by introducing step-level natural language feedback(i.e., the correctness of the current step and the further explanations). we propose math-minos, a natural language feedback enhanced verifier as shown in figure 1 . we first create high-quality step-level natural language feedback with a novel label-aware natural language feedback curation process as section 3.2 with human checking. this approach simplifies the task by incorporating step-level binary classification labels, enhancing the critique generation process for gpt-4. natural language feedback can provide in-depth reasons behind classification feedback, which enhances the training of the verifier. by employing supervised fine-tuning on only 30k natural language feedback before binary classification training, we can effectively enhance the model's evaluation capabilities. in the second stage, we introduce a two-stage training for math-minos as section 3.3: firstly, we adopt the supervised fine-tuning on curated natural language feedback to effectively help improve the model's evaluation capabilities, followed by standard orm & prm training on score feedbacks to achieve efficient inference with a single forward step. the experiments in section 4 demonstrate that infusing the model with evaluation capabilities via natural language feedback is more efficient and effective than traditional score feedback. we show that only 30k training data with natural language feedback can significantly boost the performance of mathematical verifiers in both orm and prm setting across verification and reinforcement learning tasks. moreover, our analytical experiments achieved the same level of performance with a 700% reduction in data size, demonstrating that incorporating natural language feedback can significantly enhance the training efficiency of llms. in summary, our contributions are threefold: 1. we are the first study to conduct in-depth analyses of the reasons behind the evaluations generated by verifiers, revealing the shortcomings of the current verifier's training paradigm and inspiring future research. ## methodology in this section, we introduce the background of our proposed method ( §3.1), then delve into our proposed math-minos, which contains label-aware natural language feedback curation ( §3.2) and the two-stage model training ( §3.3). ## conclusion we investigate the effectiveness of the current training paradigm of verifiers, demonstrating that score feedback from binary classification labels is suboptimal for teaching llms to accurately evaluate mathematical solutions. by introducing highquality natural language feedback with a two-stage training paradigm, we significantly enhance the verifier's evaluation ability and training efficiency. the experimental results show that models trained on a small dataset with natural language feedback (30k instances) significantly outperform the baselines that rely solely on classification labels. this highlights the critical role of rich and informative labels in training data in crafting more nuanced and effective training strategies for the development of llms that are capable of complex reasoning tasks. finally, the findings of this work pave the way for the potential integration of natural language feedback with classification verifiers.