Self-Training Large Language Models with Confident Reasoning (2025)

Abstract: Large language models (LLMs) have shown impressive performance by generating reasoning paths before final answers, but learning such a reasoning path requires costly human supervision. To address this issue, recent studies have explored self-training methods that improve reasoning capabilities using pseudo-labels generated by the LLMs themselves. Among these, confidence-based self-training fine-tunes LLMs to prefer reasoning paths with high-confidence answers, where confidence is estimated via majority voting. However, such methods exclusively focus on the quality of the final answer and may ignore the quality of the reasoning paths, as even an incorrect reasoning path leads to a correct answer by chance. Instead, we advocate the use of reasoning-level confidence to identify high-quality reasoning paths for self-training, supported by our empirical observations. We then propose a new self-training method, **CORE-PO**, that fine-tunes LLMs to prefer high-**CO**nfidence **RE**asoning paths through **P**olicy **O**ptimization. Our experiments show that CORE-PO improves the accuracy of outputs on four in-distribution and two out-of-distribution benchmarks, compared to existing self-training methods.

## introduction large language models (llms) have shown impressive performance across various tasks by generating reasoning paths before yielding the final answer @xcite @xcite . however, the potential for improving reasoning through supervision is limited by the scarcity of high-quality data with human-annotated or ground-truth labels. to address this issue, recent studies have proposed self-training methods for llms, which leverage pseudo-labels generated by the llms themselves and require only the input questions @xcite @xcite @xcite . 1 at a high level, these methods fine-tune the base llms to prefer high-quality outputs identified through self-assessment strategies in inference-time scaling techniques, e.g., self-consistency @xcite , tree-of-thoughts @xcite , and selfrefinement @xcite . as a representative approach, confidence-based methods have improved the reasoning capabilities by training llms to prefer reasoning paths associated with high-confidence answers @xcite @xcite . these methods are motivated by the observation that the answers with high confidence scores tend to yield high accuracy @xcite , and thus assume that reasoning paths leading to such answers are reliable. specifically, @xcite ; @xcite and @xcite estimate the confidence in answers using self-consistency scores measured via majority voting, and then fine-tune llms to prefer reasoning paths associated with high-confidence answers. in this work, we argue that existing confidencebased self-training methods exclusively focus on answer-level confidence and may ignore the quality of the reasoning path. in practice, as illustrated in figure 1 and observed in observation 1, llms often generate incorrect reasoning paths that lead to high-confidence answers, even when those answers are correct @xcite . consequently, llms may learn to prefer incorrect reasoning paths associated with highconfidence answers, which degrade their reasoning capabilities. this pitfall highlights the necessity of incorporating reasoning-aware confidence measures into the self-training of llms to better identify high-quality reasoning paths. the grid lattice means frozen state, so the temperature is 0°c. figure 1 : limitations in existing confidence-based self-training methods. existing self-training methods prefer reasoning paths associated with a high-confidence answer (a), estimated via voting. however, they fail to capture the errors in reasoning paths (e.g. third reasoning path above), even though the answer is correct. as a result, they can degrade the reasoning capabilities of the llm, e.g., preferring "(b)-(d) are boiling points" can lead to "32 • c is the boiling point", as shown in below. in contrast, our method measures reasoning-level confidence (as depicted by the dashed line) and fine-tunes llms to prefer high-confidence reasoning paths that yield correct statements. to this end, we propose incorporating reasoninglevel confidence into a confidence-based selftraining method. as illustrated in figure 1 , our method evaluates the correctness of reasoning by estimating confidence in the reasoning paths rather than relying solely on answer-level confidence. this is motivated by observation 2, which shows that outputs with higher reasoning-level confidence exhibit fewer errors, aligned with prior findings that such confidence is useful for identifying highquality outputs @xcite @xcite . we then propose core-po, a method that finetunes llms to prefer high-confidence reasoning paths using policy optimization. to be specific, we estimate the reasoning-level confidence using p(true) @xcite , measuring the probability that llm returns "true" to the prompt asking whether the reasoning is correct. we consider the two ways to measure p(true) of reasoning: a monolithic way that assesses the entire reasoning path, and a statement-wise way that computes the average confidence across each step in the reasoning path. then, we combine reasoning-level confidence with answer-level confidence, and fine-tune llms using direct preference optimization @xcite to prefer high-confidence outputs. in our experiments, we apply our self-training method to four arithmetic or scientific reasoning benchmarks: gsm8k @xcite , arc-challenge @xcite , gpqa @xcite , and math @xcite . we also consider two external benchmarks, cruxeval @xcite and game-of-24 @xcite , to evaluate the generalization capabilities on outof-distribution tasks. our method improves the accuracy of outputs on both in-distribution and outof-distribution tasks by enhancing reasoning quality, compared to existing self-training approaches. to conclude, our contributions can be summarized as follows: in this section, we describe the preliminaries of existing self-training methods for large language models (llms). we describe additional related works in appendix a. notation. let x denote a question provided to an llm m θ . for a given question x, we assume that the llm outputs a sequence @xmath0, a], where r represents the reasoning path and a is the final 𝑀 ! <reasoning, @xmath1, @xmath2 we introduce our confidence-based self-training method for large language models (llms) to improve their reasoning capabilities. first, we show (1) how existing confidence-based self-training methods can prefer incorrect reasoning paths, and (2) how incorporating reasoning-level confidence mitigates this issue (section 3.1). next, we describe a method that fine-tunes llms to prefer high-confidence reasoning paths using policy optimization, coined core-po (section 3.2). ## experiments in this section, we conduct experiments to validate core-po across various reasoning tasks. ## conclusion in this paper, we propose a new confidence-based self-training method that addresses a key limitation of existing approaches: the exclusive reliance on answer-level confidence, which does not capture the overall quality of the reasoning. by incorporating reasoning-level confidence, our method finetunes llms to prefer high-confidence reasoning paths with fewer errors, thereby improving their reasoning capabilities. empirical results on six benchmarks show that our method improves the reasoning capabilities of llms on both in-distribution and out-of-distribution tasks, outperforming existing self-training methods.