Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths (2024)

Abstract: Advanced models such as OpenAI o1 exhibit impressive problem-solving capabilities through step-by-step reasoning. However, they may still falter on more complex problems, making errors that disrupt their reasoning paths. We attribute this to the expansive solution space, where each step has the risk of diverging into mistakes. To enhance language model reasoning, we introduce a specialized training framework called Reasoning Paths Optimization (RPO), which enables learning to reason and explore from diverse paths. Our approach encourages favorable branches at each reasoning step while penalizing unfavorable ones, enhancing the model’s overall problem-solving performance. Reasoning Paths Optimization does not rely on large-scale human-annotated rationales or outputs from closed-source models, making it scalable and data-efficient. We focus on multi-step reasoning tasks, such as math word problems and science-based exam questions. The experiments demonstrate that our framework significantly enhances the reasoning performance of large language models, with up to 3.1% and 4.3% improvement on GSM8K and MMLU (STEM) respectively. Our data and code can be found at https://reasoning-paths.github.io.

## introduction large language models (llms) have shown remarkable proficiency in following instructions and reasoning @xcite @xcite . analogous to human cognitive processes, chain-ofthought prompting guides models to reason stepby-step before producing the final answer @xcite , significantly boosting their reasoning capabilities and demonstrating exceptional performance across a wide array of tasks @xcite . despite these advancements, llms still exhibit limitations in scenarios that require more complex reasoning @xcite . as shown in figure 1 , the step-by-step reasoning path of the model is at risk of diverging to unfavorable branches that contain mistakes, thus reducing the chance of reaching the correct solution. while such mistakes may not immediately lead to the wrong answer, they can compound and derail the reasoning process @xcite . furthermore, this challenge is amplified for more complex problems such as competition-level math questions @xcite that require long reasoning paths to solve. hence, there is a need to address this challenge by encouraging the models to generate the correct reasoning path while avoiding the unfavorable branches. to ensure a trustworthy answer derivation process, prior studies have explored a range of methods, encompassing both prompting and fine-tuning techniques. prompting methods repeatedly sample from llms for the same question and employ a voting mechanism to select the most accurate reasoning step among several alternatives. such voting mechanisms can be applied at the final stage of the process, as demonstrated in self-consistency @xcite , or at every intermediate step, as illustrated in tree-of-thought @xcite . @xcite shows that leveraging external environmental feedback could remind llms of some potential errors within their reasoning process, which potentially prevents these errors from affecting subsequent steps. however, the prompting methods generally demand extensive token usage to explore multiple reasoning paths from llms and integrate feedback from the environment. this causes a significant computational cost and huge execution latency. roger has 5 tennis balls. he buys 2 more cans of tennis balls. each can has 3 tennis balls. how many tennis balls does he have now? roger started with 5 balls. roger started with 5 balls. ## related work alignment and preference-based optimization reinforcement learning from human feedback (rlhf) @xcite is a popular technique that aligns large language models with human values and preferences. during rlhf, a separate reward model is trained to provide scalar value feedback, which is passed to fine-tune llms with ppo algorithm @xcite . however, ppo is known to be complex and unstable @xcite , and the multi-stage training of a reward model and a policy model is also challenging @xcite . recently, several techniques, including dpo @xcite , @xcite , simpo @xcite , and orpo @xcite , have been proposed to eliminate the need for a reward model, which significantly stabilize and simplify the training process. they make pairwise comparisons between two responses generated by the models and push the model to assign a higher likelihood to the favorable response over the unfavorable one. however, these preference optimization methods indiscriminately compare the two responses in their entirety, overlooking the fact that errors in multi-step reasoning tasks arise only at specific steps and their subsequent branches. in this work, we propose reasoning paths optimizationto make pairwise comparisons emanating from each intermediate step. ## conclusion in this paper, we introduced a novel training framework called reasoning paths optimization (rpo) to enhance the step-by-step reasoning capabilities of llms. our approach addresses the challenge of complex problem-solving tasks, where each reasoning step carries the risk of diverging into errors. rpo considers diverse reasoning branch pairs and encourages favorable branches at each reasoning step while penalizing unfavorable ones. our framework is scalable, as it does not rely on large-scale human-annotated rationales. instead, it leverages the model's own generated reasoning paths, making it adaptable to multi-step reasoning tasks such as math word problems. through extensive experiments on datasets of varying difficulties, our framework provides an effective approach to enhance reasoning, paving the way for more reliable and accurate problem-solving in complex scenarios.