LL a MA -Berry: Pairwise Optimization for Olympiad-level Mathematical Reasoning via O1-like M onte C arlo Tree Search (2025)

Abstract: This paper presents LLaMA-Berry, an advanced mathematical reasoning framework to enhance the problem-solving ability of large language models (LLMs). The framework combines Monte Carlo Tree Search with Self-Refine (SR-MCTS) to optimize the reasoning paths and utilizes a pairwise reward model to evaluate different paths globally. By leveraging the self-critique and rewriting capabilities of LLMs, our SR-MCTS overcomes the inefficiencies and limitations of conventional step-wise and greedy search algorithms, enabling a more efficient exploration of solution spaces. To guide the search process, we propose the Pairwise Preference Reward Model (PPRM), which predicts pairwise preferences between solutions through instruction-following capabilities trained by Reinforcement Learning from Human Feedback (RLHF). Finally, the Enhanced Borda Count (EBC) method is adopted to synthesize pairwise preferences into global quantile scores for evaluations. This approach mitigates the challenges of scoring variability and non-independent distributions in mathematical reasoning tasks. The framework has been tested on general and advanced benchmarks, showing superior search efficiency and performance compared to existing open-source and closed-source methods, particularly in complex Olympiad-level benchmarks, including AIME24 and AMC23.

## introduction mathematical reasoning represents a great challenge in artificial intelligence, with broad applications across automated theorem proving, mathematical problem solving, and scientific discovery @xcite . recently, significant strides have been made by large language models (llms) like gpt-4 @xcite in general mathematical tasks involving arithmetic and geometric problemsolving @xcite @xcite . however, complex mathematical reasoning remains challenging, especially at the olympiad-level benchmarks such as aime @xcite . an intuitive approach to improving problemsolving is to break solutions into step-by-step reasoning paths @xcite , as demonstrated in chain-of-thought (cot @xcite . while prompt-based methods can effectively facilitate the construction of such reasoning paths, they may still encounter challenges due to the lack of comprehensive feedback during the generation process, which can affect efficiency @xcite . in contrast to stepwise generation methods, another promising line of research treats the entire solution as an independent state, employing rewriting capabilities to refine the solutions, such as in self-refine @xcite and reflexion @xcite . however, these approaches, while innovative, may occasionally face challenges like being susceptible to local optima or potentially drifting towards suboptimal solutions due to flawed feedback, which could impact their maximum potential performance. in addition to generating reasoning paths, effective solution evaluation is crucial, with models like the outcome reward model (orm) and process reward model (prm) @xcite serving as valuable examples. the orm focuses on the correctness of the final answer in a reasoning path, while the prm emphasizes the correctness of each step in the process. while both methods enable reward models to assign scalar scores, obtaining reliable labeled data for training these reward models remains a significant challenge. moreover, the scoring standards for mathematical reasoning tasks can vary significantly, as each problem presents unique characteristics. this variation complicates the scaling of reward models and hinders their ability to capture local preference relations between solutions. although trained using language models, these reward models have yet to fully leverage instruction-following capabilities, which may limit their effectiveness in handling more complex reasoning tasks @xcite . to improve the efficiency of solution search in mathematical problems, we treat a complete solution as an independent state and apply self-refine to optimize previous solutions in order to obtain better ones. in the self-refine process, feedback from critiques is utilized to make the search more efficient compared to stepwise reasoning path generation. furthermore, we incorporate monte carlo tree search (mcts @xcite to replace the iterative manner in naive self-refine, enhancing the solution search. mcts leverages signals from the evaluation process to assess solutions and uses the upper confidence bound applied to trees (uct) method to balance exploration and exploitation. this approach enables the search process to effectively exploit higher-quality solutions and explore those with greater potential for improvement, while avoiding getting trapped in suboptimal local minima. in the evaluation process, utilizing the instruction-following capabilities trained by reinforcement learning from human feedback @xcite , pairwise preference reward model (pprm) transforms the absolute rewards calculation into preferences prediction between solutions to calculate rewards. the approach reduces the variability with scoring characteristics and thus leads to a more robust and consistent evaluation of different solutions. to overcome the locality limitations inherent in pairwise comparisons, we employ the enhanced borda count (ebc) method to aggregate local preference evaluations into global quantile scores, leading to more informed decision-making and, ultimately, better solutions. combining the pprm and ebc method not only enables the reward model to learn a more robust reward signal but also captures the global characteristics of the solution space, ensuring more reliable comparisons. our contributions are summarized as follows: @xcite we propose sr-mcts, a novel markov decision process (mdp) framework that treats entire solutions as states and self-refine as optimization action to perform advanced solution search with mcts. (2) pprm is developed to leverage the preference relationship between solutions to evaluate their quality, which avoids the volatility of absolute scores while providing a more guided exploration of optimal paths. we adopt the ebc method to convert the local preferences into global evaluations. (3) we verify the effectiveness of llama-berry on multiple benchmarks, which outperforms baseline approaches like tot @xcite and rstar @xcite in both search efficiency and accuracy. notably, llama-berry enhances the performance of llama-3.1-8b, making it comparable to proprietary models, including gpt-4 turbo on olympiad-level mathematical reasoning without additional training. ## related works reward models for reasoning. reliable reward models @xcite @xcite @xcite can effectively distinguish desirable responses from undesirable ones, which is especially important in complex reasoning. the outcome reward model (orm) are trained with the final results of the reasoning paths. as the rewards are determined by the final answers, orm may suffer from coarse supervision and misalignment issues. in contrast, process reward model (prm) provides step-wise reward signals that are easier to interpret and guide the models to follow the cots. therefore, prm is generally considered to be more effective @xcite . however, the success of prm relies on extensive manually annotated data @xcite , which is time-consuming and still faces the challenge of the volatility of absolute reward scores.