RASP berry: Retrieval-Augmented M onte C arlo Tree Self-Play with Reasoning Consistency for Multi-Hop Question Answering (2025)

Abstract: Complex multi-hop question answering requires large language models (LLMs) not only to retrieve external knowledge but also to reason over the retrieved information in order to arrive at the final solution. This involves two key challenges: (i) how to effectively explore the solution space and generate more potentially correct solution candidates, and (ii) how to select the optimal solution from multiple solution candidates, both of which require a training-free approach without introducing a more powerful teacher model. To address these challenges, we propose Retrieval-Augmented Monte Carlo Tree Self-Play with Reasoning Consistency (RASPberry), which introduces a more flexible action-level sampling granularity compared to existing methods, leverages Monte Carlo Tree Search for efficient solution space exploration, and utilizes an enhanced version of reasoning consistency to guide the selection of the optimal solution. Experimental results demonstrate that our proposed RASPberry effectively tackles the two challenges outlined above, achieving more efficient RAG inference-time scaling. Our code is available at https://github.com/BaixuanLi/RASPberry.

## introduction retrieval-augmented generation (rag) @xcite enables large language models (llms) @xcite to incorporate external document knowledge during the question-answering (qa) process, significantly enhancing the performance of llms in single-hop qa. however, more complex multi-hop qa @xcite presents greater challenges for llms. it requires llms to not only identify relevant knowledge from multiple documents but also to perform reasoning based on it to derive the correct response. although a more powerful reasoner can currently be trained through reinforcement learning (rl) to extend reasoning capabilities with rag (guo et al., our raspberry consistently achieves significant performance improvements across all datasets and models. although recent work has introduced monte carlo tree search (mcts) @xcite to guide the exploration of the solution space @xcite , these methods do not consider the actions required to integrate external knowledge, and thus lack a suitable action set for search tree expansion in rag scenarios. second, accurately determining the correct solution from the collection of generated candidate solutions. without additional training or stronger teacher model supervision, smaller llms' self-scoring tends to be nearly random, leading to failures of self-reward methods such as selfverification @xcite . moreover, since smaller llms are more likely to produce incorrect solutions during multiple sampling attempts, this also results in the failure of self-consistency methods. although the reasoning consistency @xcite alleviates this issue by considering the consistency of the reasoning process rather than just the answer, its design is tailored for scenarios that do not rely on external document support. as such, it is not fully applicable in rag scenarios, as it does not take into account the need to ensure that the external documents supporting reasoning remain consistent when computing consistency. to address the two challenges, we propose retrieval-augmented monte carlo tree self-play with reasoning consistency (raspberry). specifically, raspberry consists of two components, each targeting one of the challenges: (i) retrieval-augmented monte carlo tree self-play enables sufficient exploration of the solution space in rag scenarios by integrating the mcts algorithm, which generates a broader set of potentially correct candidate solutions; (ii) retrieval-retained rea-soning consistency is used to filter the candidate solution paths obtained in (i), selecting the optimal solution path as the final answer. as shown in figure 1 , without the need for additional training costs or a stronger teacher model, raspberry achieves stable and significant improvements across three mainstream small llms @xcite @xcite and two complex multi-hop question answering datasets @xcite , enabling effective inference-time scaling in rag scenarios that require external document knowledge. existing methods facilitate reasoning over retrieved documents by iteratively combining rag with the chain-of-thought (cot) process @xcite . additionally, some works focus on query rewriting and query decomposition prior to retrieval to expand the range of retrieved documents @xcite . meanwhile, @xcite proposed a self-reflection mechanism for post-reasoning self-correction. furthermore, recent research has explored the combination of treelike reasoning structures with rag @xcite to further enhance the flexibility of solu-tion space exploration. however, unguided path exploration becomes highly challenging when the solution path is too complex. moreover, the limited set of executable actions increases the risk of the solution path getting trapped in local optima. as shown in inference-time scaling for reasoning. recent works primarily guide the optimization and expansion of reasoning paths in llms by combining a reward model aligned with human preferences @xcite @xcite . however, this introduces additional training costs. alternatively, some approaches employ mcts methods for selfrefinement @xcite or self-play @xcite , enabling inference-time scaling in an inference-only manner. however, the designs of these works are focused on scenarios where only the model's internal parameterized knowledge is used for solving, such as mathematical reasoning or commonsense reasoning. since these models do not have the ability to access external document knowledge, they fail to effectively adapt to the complex multi-hop qa scenarios in rag. in this work, we propose raspberry, which introduces a rich and comprehensive action set tailored for rag scenarios, used for tree expansion during the mcts process. this enables an integration of the rag mechanism with the mcts reasoning structure, allowing the llm to not only leverage its internal parameterized knowledge for reasoning but also flexibly utilize external retrieved documents to provide supportive information. ## preliminary we introduce the mechanism of monte carlo tree search (mcts) @xcite , which is essential for understanding our proposed raspberry. mcts is a decision-making algorithm widely ap-plied in games and complex decision-making processes, which builds a search tree based on a predefined set of actions and simulates possible outcomes to estimate the value of each action. typically, the mcts comprises four key phases: selection: starting from the root, the algorithm navigates through promising child nodes based on specific strategies (e.g., upper confidence bound applied to trees, uct), continuing until it reaches a leaf node. the uct is calculated as follows: expansion: at the leaf node, if it does not represent a terminal state (e.g., reaching the maximum depth or arriving at the final solution), feasible child nodes are added based on the current node's action set to represent potential future moves. simulation: from the newly added node, the algorithm performs random simulations (often termed rollouts), arbitrarily selecting moves until the game reaches its terminal state, thereby evaluating the node's potential (estimated value q). backpropagation: after the simulation, the value of the terminal node (calculated based on a custom reward function) is propagated back to the root, updating the statistical data (visit counts n , estimated values q) of all visited nodes during the simulation to guide future decisions. through iterative execution of these stages, mcts incrementally builds a decision tree, optimizing strategies in scenarios where the vast state space makes direct computation infeasible. ## raspberry as shown in figure 2 , our proposed raspberry consists of two main components: retrieval-augmented monte carlo tree self-play (ra-mct retrieval-augmented monte carlo tree self-play retrieval-retained reasoning consistency selection expansion simulation backpropagation repeated n times question: what type of sport does sergio casal and manuel orantes have in common? candidate solution: masked solution: record candidate solution paths (b) (a) mcts with @xmath0, through document analysis, we derive: ... both sergio casal and manuel orantes have experience in tennis. <@xmath1, through document analysis, we derive: ... both sergio casal and manuel orantes have experience in tennis. <@xmath2: <rephrased @xmath3, through document analysis, we derive: ... both sergio casal and manuel orantes have baseball in common. <@xmath4: <rephrased @xmath5, through document analysis, we derive: ... both sergio casal and manuel orantes have tennis in common. <@xmath6... ... ... ... self-play in §4.1) for solution space exploration and generation of candidate reasoning paths, and retrieval-retained reasoning consistency (rr-rc in §4.2) for optimal final path selection. ## experiments in this work, we select three mainstream small llms, namely llama-3.1-8b-instruct @xcite , qwen-2.5-7b-instruct @xcite , and qwen-2-7b-instruct @xcite . additionally, we choose two commonly used knowledge-intensive multi-hop qa datasets, namely hotpotqa @xcite and 2wiki-multihopqa (2wikiqa) @xcite , both of which are based on wiki documents. we divide the supporting documents in the dataset by their respective topics, embed them into text vectors using a mainstream dense retriever (bge-m3 @xcite ), and then use faiss @xcite to maintain a local vector database for retrieval. furthermore, we include a wide range of baseline methods for comparison, categorized into single-round and multi-round rag baselines: (a) single-round rag baselines. (i) rag-cot, which is rag combined with naive cot @xcite . (ii) rq-rag @xcite , which adds query rewriting and subquery decomposition. (iii) self-rag @xcite , which incorporates self-reflection for reasoning results. it is important to note that for all methods in our experiments, we adopt a unified dense retrieval mechanism and set the number of returned documents per retrieval to 4 by default. design differences and further implementation details are provided in ## conclusion for complex multi-hop question answering, existing methods fail to ensure (i) effective exploration of the solution space and (ii) correct selection of the final solution. to address these challenges, we propose raspberry. compared to existing methods, raspberry enables more effective solution space exploration. additionally, we adopts an enhanced version of reasoning consistency tailored to the rag scenario, offering a more comprehensive final solution selection. overall, our proposed raspberry achieves more effective and efficient rag inference-time scaling in a training-free manner, without the need for a stronger teacher model.