Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing (2024)

Abstract: Large Language Models (LLMs) have demonstrated significant potential in handling complex reasoning tasks through step-by-step rationale generation. However, recent studies have raised concerns regarding the hallucination and flaws in their reasoning process. Substantial efforts are being made to improve the reliability and faithfulness of the generated rationales. Some approaches model reasoning as planning, while others focus on annotating for process supervision. Nevertheless, the planning-based search process often results in high latency due to the frequent assessment of intermediate reasoning states and the extensive exploration space. Additionally, supervising the reasoning process with human annotation is costly and challenging to scale for LLM training. To address these issues, in this paper, we propose a framework to learn planning-based reasoning through Direct Preference Optimization (DPO) on collected trajectories, which are ranked according to synthesized process rewards. Our results on challenging logical reasoning benchmarks demonstrate the effectiveness of our learning framework, showing that our 7B model can surpass the strong counterparts like GPT-3.5-Turbo.

## introduction natural language reasoning has been a fundamental element in the advancement of artificial intelligence (ai), with its significant impact on a variety of applications including planning and decision making @xcite . the goal of building ai systems capable of replicating human-like reasoning remains a primary focus within the research community. recent advancements in large language models (llms) have showcased their ability to perform complex reasoning tasks, creating sequences of reasoning steps akin to human † correspondence to: nancy f. chen and shafiq joty. 1 code and trajectory data are released at sparkjiao/dpotrajectory-reasoning. @xcite for a logical reasoning problem in logiqa-v2 dataset @xcite . it follow the react @xcite format, where each step is marked with a dotted rectangle. the content highlighted in green summarizes some opinions in the context, and is omitted. the central reasoning steps pivotal to arriving at the solution are emphasized in pink. the complete reasoning process can be found in figure 8 . tremendous efforts have been dedicated to improve the reliability and faithfulness of generated rationales, including knowledge distillation @xcite @xcite and self-correction @xcite . yet, these approaches predominantly rely on llms for identifying errors or providing superior reasoning processes, which could be limited by their capacity. an alternative is to consider human process supervision @xcite . for instance, @xcite propose to train a process reward model (prm) using step-level feedbacks on model-generated solutions, which are annotated by human experts. this enables llms to refine their rationales based on the prm's feedback. while human process supervision has proven effective, it often incurs higher costs compared to mere final outcome annotation as well as the automatic process annotation from a teacher llm. in addition to the attempts on process supervision, some research efforts have explored searchaugmented reasoning for better reasoning trace by assessing the quality of future states. @xcite introduce a general framework of reasoningas-planning (rap), where the reasoning process is defined as a markov decision process (mdp). each step in the mdp comprises a state-action pair, whose particular implementation can vary with different application scenarios. figure 1 illustrates this process in the context of logical reasoning using the react @xcite format. at each step the agent can think (optionally) and act, which involves selecting a group of facts and rules to deduce a new conclusion. 2 it can optionally make an observation to get an "updated view" of the state. during inference, each state-action pair is assigned a reward, either by an llm or external verifier. the planning process is then steered by monte carlo tree search (mcts) @xcite to maximize the expected total cumulative reward (or utility) obtained along the chosen path while effectively narrowing the search space (figure 2(a) ). existing rap frameworks often assume llms as the world model being able to assess the quality of each reasoning step. as a result, the online planning may introduce huge latency and cost due to frequent assessments of intermediate states and the large search space. nevertheless, we find that the core idea behind planning-based reasoning is to employ online simulation by taking few forward steps to find the optimal path, and the evaluation becomes more accurate when it has access to real outcome feedback. in this paper, we explore offline simulation to 2 the notion of action subsumes both thinking and acting. the goal of inference is find an optimal reasoning path with maximum expected utility. in our method, the policy model will first explore multiple reasoning paths, with the process rewards calibrated by outcome supervision. and we then optimize it using dpo @xcite to maximize the probability of the paths with higher cumulative reward. ## conclusion in this paper, we propose a novel idea to transform reasoning-as-planning as a learning problem to avoid the latency induced by online search. inspired by mcts, we developed a offline simulation approach to estimate the expected value of intermediate reasoning steps. after that, we use the collected expected value dataset to fit a process reward model and annotate the full trajectories with sequence-level rewards. finally, the policy model is optimized using direct preference optimization. the experimental results on logical and mathematical reasoning demonstrate the effectiveness of our proposed method. towards the future work, we hope to explore the synthesised process reward estimated by weak-supervision from different aspects to further alleviate the reliance on human annotations and enable consistent self-improvement.