Advancing Process Verification for Large Language Models via Tree-Based Preference Learning (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable potential in handling complex reasoning tasks by generating step-by-step rationales. Some methods have proven effective in boosting accuracy by introducing extra verifiers to assess these paths. However, existing verifiers, typically trained on binary-labeled reasoning paths, fail to fully utilize the relative merits of intermediate steps, thereby limiting the effectiveness of the feedback provided. To overcome this limitation, we propose Tree-based Preference Learning Verifier (Tree-PLV), a novel approach that constructs reasoning trees via a best-first search algorithm and collects step-level paired data for preference training. Compared to traditional binary classification, step-level preferences more finely capture the nuances between reasoning steps, allowing for a more precise evaluation of the complete reasoning path. We empirically evaluate Tree-PLV across a range of arithmetic and commonsense reasoning tasks, where it significantly outperforms existing benchmarks. For instance, Tree-PLV achieved substantial performance gains over the Mistral-7B self-consistency baseline on GSM8K (67.55% → 82.79%), MATH (17.00% → 26.80%), CSQA (68.14% → 72.97%), and StrategyQA (82.86% → 83.25%). Additionally, our study explores the appropriate granularity for applying preference learning, revealing that step-level guidance provides feedback that better aligns with the evaluation of the reasoning process.

## introduction large language models (llms) have demonstrated the ability to decompose complex questions into step-by-step problem-solving processes @xcite @xcite , achieving strong reasoning performance across a variety of tasks. ## related work recent studies @xcite @xcite have demonstrated that incorporating a verifier can significantly enhance performance in reasoning tasks. primarily, there are two methods for training verifiers: outcome supervision and process supervision. unlike these approaches, which rely on supervised learning with binary labels, our methodology employs preference learning to achieve step-level alignment. we provide a detailed discussion of additional related work in appendix a. ## conclusion in this paper, we propose tree-plv, an innovative verifier developed through step-level preference learning. utilizing a best-first tree search framework to construct reasoning trees, this method creates a diverse and robust dataset, ensuring a rich pool of paired data for preference training. by leveraging step-level preferences instead of conventional binary labels, our methodology provides more granular and detailed feedback, enhancing the verification of reasoning paths. our empirical findings highlight the efficacy of tree-plv, showing significant performance improvements over existing benchmarks across a variety of arithmetic and commonsense reasoning tasks. in future work, we aim to delve deeper into integrating our verifier within the reasoning process by combining it with various search algorithms, thereby refining the verification of intermediate reasoning steps.