A uto PRM : Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition (2024)

Abstract: Recent advancements in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet their reliance on extensive manual labeling to provide procedural feedback remains a significant impediment. To address this challenge, in this paper, we propose a novel self-supervised framework **AutoPRM** that efficiently enhances the fine-tuning of LLMs for intricate reasoning challenges. Specifically, **AutoPRM** first decomposes complex problems into more manageable subquestions with a controllable granularity switch, then sequentially apply reinforcement learning to iteratively improve the subquestion solver. Additionally, we propose context-guided decoding to avoid reward tampering and guide the subquestion solver towards the solution of the holistic problem. Extensive experiments show that **AutoPRM** significantly improves performance on mathematical and commonsense reasoning tasks over SOTA. More encouragingly, **AutoPRM** can be easily integrated with other orthogonal reasoning pipelines.

## introduction the landscape of natural language processing has been profoundly reshaped by the evolution of large language models (llms), which have demonstrated remarkable capabilities in a variety of complex tasks @xcite @xcite . among these, multi-step reasoning has emerged as a particularly challenging area and has drawn significant research attention @xcite @xcite . to enhance complex reasoning capabilities for llms, recent prompting-based approaches, including chain-of-thought (cot) @xcite and self-evaluation decoding @xcite @xcite , have proven to be successful. however, while being effective on large-sized models (e.g., @xcite , palm-2 @xcite ), they are less effective for smaller-sized nonfinetuned models (e.g., gpt-3 @xcite , llama-2-7b @xcite ) which are poor reasoners by nature @xcite . on the other hand, fine-tuning methods are also known to be effective for enhancing complex reasoning capabilities, especially for smaller-sized models @xcite @xcite . therein, procedural supervisionbased fine-tuning @xcite has proved to be particularly effective @xcite , which emulates human problemsolving process and provides step-by-step feedback, as opposed to outcome-based supervision which simply optimizes for the final-answers @xcite . despite its advancements, the reliance on step-wise human annotations for these process-supervised reward models (prm) presents a significant bottleneck. more specifically, such annotation is not only resource-intensive in terms of both time and domain expertise @xcite , but also introduces human bias due to subjective judgements, which could potentially undermine the fine-tuning performance @xcite . these challenges highlight a critical need for a more efficient and scalable approach to fine-tune smaller-sized llms for complex reasoning tasks. to address these challenges, we propose a novel self-supervised procedural reward model termed autoprm to boost the efficiency and accuracy in fine-tuning and inference for long-chained reasoning problems. concretely, autoprm first breaks down the problem into a sequence of sub-questions with a trained question-decomposition (qd) model, then solves each subquestion with a reinforcement learning (rl)-optimized question-answering (qa) model @xcite . inspired by human cognitive process where questioning and answering reciprocally enhances each other @xcite , we train one unified model to handle both qd and qa. foot_0 notably, instead of training a prm that requires extensive manual annotations, autoprm adopts a more natural approach by directly training an intermediate outcome verifier to optimize the subquestion solver. the main contributions of this paper are: (1) autoprm, a novel fine-tuning framework that enhances llms reasoning abilities. this framework reduces the need for extensive human annotations by employing automatic question decomposition and an rl-optimized subquestion solver. ## autoprm in this section, we detail our proposed framework autoprm. our key insight is that automating stepwise question decomposition provides a natural perspective to reduce problem dimensions, through which model inference and optimization can be more precise and efficient. with decomposition, the fine-grained feedback can be obtained with a reliable step-wise verifier trained to predict intermediate results, which lead to a more powerful and bias-free reasoning model. preliminaries and notations. our problem formulation involves a dataset @xmath0, a i )} n @xmath1, where each problem p i is associated with a final answer a i that can be reached through reasoning. autoprm breaks the problem into multiple steps and models the reasoning process as a markov decision process (mdp) ⟨s, a, q, r, p, γ⟩ @xcite , where each mdp episode starts with a sampled problem input p i and ends either when a final answer is generated or the model abstains. au-toprm manages each individual reasoning step as a subquestion-solution pair {(q t , s t ), q @xmath2, s @xmath3, where s is the state (subsolution) space and q is the subquestion space. 2 the transition function appends a subsolution s t to the end of the cumulative state (p i , s 0 , s 1 , . . . , s t-1 ) at each step. a reward function (verifier) r : q × @xmath4, which provides a sparse feedback at the end of generation, or process-based @xcite , which evaluates each step individually and assigns a finegrained score for each intermediate step (q t , s t ). ## experiments in this section, we assess autoprm, our proposed framework, by exploring three key questions: (1) to what extent does autoprm enhance the reasoning capabilities of llms? (2) how do individual sub-modules contribute to autoprm's overall performance improvement? and (3) what are the limitations and opportunities in enhancing multi-step reasoning for smaller-scaled models? ## related work llms struggle with complex reasoning tasks @xcite . to mitigate this limitation, promptbased methods including chain-of-thought (cot) @xcite and its variants such as automatic cot @xcite , complex cot @xcite , tree-of-thought @xcite , graph-of-thought @xcite and exchange-of-thought @xcite are developed. although effective for larger llms, the performance of these methods is limited in smaller models @xcite @xcite , which leads to the exploration of problem decomposition into subquestions and sequential handling @xcite @xcite , coupled with step-wise feedback using self-verification @xcite , external llms @xcite , heuristics @xcite , and human-annotated rewards @xcite . however, these methods often either depend on external large models, or require intensive human effort or specific designs @xcite , which severely limit their applicability. besides these prompt-based methods, finetuning has been another main line of research showing promise in enhancing llms reasoning capabilities for both large and smaller models @xcite @xcite , especially when pairing with data augmentation techniques such as multi-view question bootstrapping @xcite and instruction evaluation @xcite . among numerous fine-tuning approaches, existing studies suggest procedural supervision yields better accuracy than outcome-only methods @xcite @xcite . however, procedural supervision approaches often require extensive, unbiased manual labeling, which greatly limits their generalizability @xcite . inspired by the step-wise approaches as seen in the prompt-based methods, as well as procedural supervision used in fine-tuning approaches, in this paper, we introduce a novel self-supervised finetuning approach that significantly enhances llms reasoning capabilities while not requiring either external large models, or additional human efforts. ## conclusions and future work in this paper, we introduce autoprm, a novel framework that automates procedural supervision for multi-step reasoning in llms. the core of au-toprm consists of two reciprocal components: a qd model that systematically breaks down complex problems into manageable subquestions, and a qa model that accurately answers these subquestions. autoprm employs a robust training methodology, incorporating supervised fine-tuning, feedback-based step-wise verifier, and a final rl fine-tuning for the best performance. through extensive experiments, we demonstrate that auto-prm significantly outperforms sota methods in terms of efficiency and accuracy on three arithmetic and commonsense reasoning tasks. results show that our automated qd process, coupled with a rl-optimized qa model, leads to a substantial improvement in handling complex reasoning tasks. future developments of autoprm could focus on expanding its application to a wider range of complex problem domains to test its versatility and identify domain-specific challenges. additionally, extending its capabilities for long-term reasoning and exploring interdisciplinary applications could also be another promising direction.