Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning (2025)

Abstract: Despite recent advances in large language models, open-source models often struggle to consistently perform well on complex reasoning tasks. Existing ensemble methods, whether applied at the token or output levels, fail to address these challenges. In response, we present Language model Ensemble with Monte Carlo Tree Search (LE-MCTS), a novel framework for process-level ensembling of language models. LE-MCTS formulates step-by-step reasoning with an ensemble of language models as a Markov decision process. In this framework, states represent intermediate reasoning paths, while actions consist of generating the next reasoning step using one of the language models selected from a predefined pool. Guided by a process-based reward model, LE-MCTS performs a tree search over the reasoning steps generated by different language models, identifying the most accurate reasoning chain. Experimental results on five mathematical reasoning benchmarks demonstrate that our approach outperforms both single language model decoding algorithms and language model ensemble methods. Notably, LE-MCTS improves performance by 3.6% and 4.3% on the MATH and MQA datasets, respectively, highlighting its effectiveness in solving complex reasoning problems.

## introduction large language models (llms) have demonstrated superior performance across a range of tasks, primarily due to their large capacity and high-quality training data. however, unlike prominent closedsource llms such as @xcite and gemini-1.5 @xcite , open-source models like mistral @xcite , llama-3 (llama, 2024), and gemma-2 @xcite are constrained by factors such as data availability, architecture, and hyperparameter choices. as a result, they exhibit different strengths and weaknesses @xcite . language model ensembling is a well-known approach for creating a versatile model by combining weaker language models. previous studies on lm ensembling have focused on merging language models at the token and output levels. token-level approaches have merged the output logits or probabilities of language models, using perplexity-based weighted averaging @xcite and vocabulary projection @xcite . although token-level ensemble methods offer fine-grained fusion of language models, they face several constraints related to tokenizer vocabularies and model architectures, requiring the training of additional projection matrices @xcite to mitigate these. outputlevel approaches have involved ensembling fully generated outputs, either by ranking multiple outputs to select the best one @xcite or by using an additional fusion model to fuse them @xcite . although output-level ensembling can be applied to any language model, it cannot produce a correct answer if all candidate outputs are flawed @xcite . our experiments in to address the limitations of token-and outputlevel ensembling, we propose a process-level language model ensemble framework tailored to complex reasoning tasks. many complex reasoning problems can be solved step by step, and evaluating each reasoning step individually allows the model to correct errors early, thus guiding the decoding process toward more accurate solutions @xcite @xcite . this approach is further supported by recent advances in process-based reward models (prms) @xcite @xcite , which make it possible to measure the correctness of each intermediate reasoning step as a scalar value. based on the advantages of process-level reasoning, process-level ensembling offers finer and more efficient control over generation compared to output-level ensembling, as it allows for the correction of intermediate reasoning steps without regenerating entire solutions. additionally, process-level ensembling is less constrained than token-level ensemble approaches, as it eliminates the need to match vocabulary and architecture. we present language model ensemble with monte carlo tree search (le-mcts), a pioneering framework for process-level ensembling of language models. we formulate the step-by-step reasoning involving multiple llms as a markov decision process (mdp) @xcite . specifically, the state is defined as the intermediate reasoning steps, and the action is defined as generating the next reasoning step using one of the language mod-els selected from a predefined pool of llms. our mcts algorithm, inspired by alphazero @xcite , performs a tree search over the unified space of reasoning steps generated by different llms. following the guidance from prm, we can obtain the reasoning chain that is likely to be the most accurate among the possible combinations of reasoning steps. we evaluate le-mcts on five math reasoning benchmarks: gsm8k @xcite , math @xcite , svamp @xcite , asdiv @xcite , and mqa @xcite . our results show that le-mcts consistently outperforms existing lm ensemble approaches across all tasks. we also evaluate a straightforward integration of process reward-guided decoding algorithms with the lm ensemble. although these models perform well on grade school math problems, they fail on more complex datasets, such as math and mqa. in contrast, le-mcts improves performance by 3.6% and 4.3% on math and mqa, respectively, compared to the second-best models. these experimental results highlight the effectiveness of le-mcts in tackling complex reasoning problems. ## methods first, we introduce the problem setup and key notation. we then describe our le-mcts algorithm. ## conclusion in this paper, we introduced a novel framework for process-level ensembling of language models, addressing key limitations of traditional language model ensemble methods. we formulated the stepby-step reasoning with an ensemble of language models as a markov decision process. by leveraging an existing process-based reward model and monte carlo tree search, our approach effectively navigates the unified space of reasoning steps generated by different language models, discovering more accurate solutions. extensive experiments on mathematical reasoning benchmarks demonstrate the efficacy of the proposed method, especially in solving complex reasoning problems. we believe this approach paves the way for broader processlevel ensemble of language models, as it can be applied to any step-by-step reasoning problem where an appropriate process-reward model exists.