Probability-Consistent Preference Optimization for Enhanced LLM Reasoning (2025)

Abstract: Recent advances in preference optimization have demonstrated significant potential for improving mathematical reasoning capabilities in large language models (LLMs). While current approaches leverage high-quality pairwise preference data through outcome-based criteria like answer correctness or consistency, they fundamentally neglect the internal logical coherence of responses. To overcome this, we propose Probability-Consistent Preference Optimization (PCPO), a novel framework that establishes dual quantitative metrics for preference selection: (1) surface-level answer correctness and (2) intrinsic token-level probability consistency across responses. Extensive experiments show that our PCPO consistently outperforms existing outcome-only criterion approaches across a diverse range of LLMs and benchmarks. Our code is publicly available at https://github.com/YunqiaoYang/PCPO.

## introduction in recent years, enhancing the mathematical reasoning ability of large language models @xcite @xcite @xcite @xcite (llms) has emerged as an important research direction @xcite . among various approaches, direct optimization preference (dpo) @xcite is widely used due to its simplicity and efficiency. since its introduction, numerous extensions of dpo have been proposed to further improve mathematical reasoning in diverse ways. for instance, methods such as @xcite and iterative @xcite demonstrate the effectiveness of iterative training strategies. additionally, constructing high-quality pairwise preference data * equal contribution. to construct high-quality pairwise preference data, previous methods, such as irpo @xcite and scpo @xcite , select preference training pairs from generated responses that include a chain-of-thought (cot) @xcite process followed by a final answer, have proven particularly effective in advancing mathematical reasoning performance. @xcite employs gold labels (correct answers) to distinguish between chosen and rejected responses. specifically, if a response's answer matches the gold label, it is designated as a chosen response; otherwise, it is classified as rejected. on the other hand, scpo @xcite utilizes a voting function to evaluate the self-consistency @xcite of responses. responses whose answers appear most frequently are selected as chosen, while those with the least frequent answers are marked as rejected. however, both methods focus solely on the correctness or frequency of the final answer while overlooking the internal logical connections or nuanced differences between responses. this limitation restricts the creation of more refined and informative preference training data @xcite . consequently, models may have difficulty recognizing subtle yet critical distinctions between chosen and rejected responses during the iterative dpo training process @xcite . in this paper, we propose a novel method called probability-consistent preference optimization (pcpo), which leverages both the final answer and the internal logical connections of responses when selecting preference pairs. our method is grounded in the principle that the token generation process in llms fundamentally involves predicting new tokens based on the highest conditional probability given all existing tokens @xcite ; {(𝑥𝑥 𝑖𝑖 , 𝑦𝑦 𝑖𝑖 * )} 𝑀𝑀 𝑡𝑡 𝑦𝑦 𝑤𝑤 𝑖𝑖 𝑗𝑗 , 𝑦𝑦 𝑙𝑙 𝑖𝑖 𝑦𝑦 𝑤𝑤 𝑖𝑖 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 , 𝑦𝑦 𝑙𝑙 𝑖𝑖 𝑀𝑀 𝑡𝑡+1 𝑦𝑦 𝑖𝑖 1 = (𝑐𝑐 𝑖𝑖 1 , 𝑎𝑎 𝑖𝑖 1 ) ⋮ 𝑦𝑦 𝑖𝑖 𝑁𝑁 = (𝑐𝑐 𝑖𝑖 𝑁𝑁 , 𝑎𝑎 𝑖𝑖 𝑁𝑁 ) 𝒔𝒔 𝑤𝑤 (𝑦𝑦 𝑤𝑤 𝑖𝑖 𝑗𝑗 , 𝑦𝑦 𝑙𝑙 𝑖𝑖 ) pcpo training prompts with answer generated responses candidate pairs weighted scores preference pairs next iteration select calculate token probability consistency radford, 2018 ). specifically, pcpo calculates a weighted score between preferred and dispreferred answers by evaluating the conditional probability of each token in the responses @xcite . preference pairs are then selected based on the highest weighted scores. this approach provides a quantitative framework for selecting preference pairs by considering not only the correctness of the final answer but also the internal coherence of the responses. by incorporating these factors, pcpo ensures a more robust and principled selection of preference pairs. in each iteration, we first use the seed model to generate multiple responses for each math problem, and we construct a candidate preference pair set based on the correctness of the final answer. then, we calculate the token-level consistency score for all the preference pairs from the candidate pair set. afterward, we select the preference pairs with the highest token-level weighted score for each problem to construct preference training pairs eventually. finally, we use the preference pairs selected to train the next iteration model with a modified dpo loss. to validate our method, we apply it to widely used math datasets, including gsm8k @xcite , math-500 @xcite olympiadbench @xcite and amc23 (mathematical association of america, 2023).to comprehensively demonstrate the effectiveness of our approach, we conduct experiments across a diverse range of seed models, such as llama-3-8b-instruct @xcite , mathstral-7b-v0.1 @xcite , qwen-2.5-7b-instruct @xcite and qwen-2.5-math-7b-instruct @xcite . consistent results across these models showcase the effective-ness of our method. in summary, our contributions are as follows: ## method as depicted in figure 1 , our method starts with a pre-trained seed language model and a fixed prompt set of math problems with final answers. the pcpo pipeline mainly consists of three steps. (1) utilize m t (m 0 as the seed model) to generate responses y n i with reasoning c n i and answer a n i , and construct candidate pairs on correctness §2.1. (2) utilize m t to compute weighted scores s w for each pair based on the token probability, and select preference pairs based on it §2.2. (3) train the next iteration model m t+1 with the selected preference pairs and pcpo loss §2.3. the model will be trained and updated at each iteration, resulting in a series of models m 1 , . . . , m t . ## conclusion in this paper, we introduce probability-consistent preference optimization (pcpo), which provides a quantitative framework for selecting preference pairs by considering both the correctness of the final answer and the internal coherence of the responses. we introduced the concept of token probability consistency and the pair-weighted score to help select resulting preference training pairs. extensive experiments demonstrate that our method consistently outperforms existing outcome-only criterion approaches (e.g., irpo, scpo) across a diverse range of llms and benchmarks. this work paves the way for future research aimed at improving reasoning capabilities through more sophisticated preference pair selection methods.