Step- KTO : Optimizing Mathematical Reasoning through Stepwise Binary Feedback (2025)

Abstract: Large language models (LLMs) have recently demonstrated remarkable success in mathematical reasoning. Despite progress in methods like chain-of-thought prompting and self-consistency sampling, these advances often focus on final correctness without ensuring that the underlying reasoning process is coherent and reliable. This paper introduces Step-KTO, a training framework that combines process-level and outcome-level binary feedback to guide LLMs toward more trustworthy reasoning trajectories. By providing binary evaluations for both the intermediate reasoning steps and the final answer, Step-KTO encourages the model to adhere to logical progressions rather than relying on superficial shortcuts. Our experiments on challenging mathematical benchmarks show that Step-KTO significantly improves both final answer accuracy and the quality of intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO achieves a notable improvement in Pass@1 accuracy over strong baselines. These results highlight the promise of integrating stepwise process feedback into LLM training, paving the way toward more interpretable and dependable reasoning capabilities.

## introduction large language models (llms) have recently shown remarkable capabilities in reasoningintensive tasks such as coding @xcite @xcite and solving complex mathematical problems @xcite . prompting strategies like chain-of-thought prompting @xcite @xcite and self-consistency sampling @xcite enhance these models' final-answer accuracy by encouraging them to articulate intermediate reasoning steps. however, a significant issue remains: even when these methods boost final-answer correctness, the internal reasoning steps are often unreliable or logically inconsistent @xcite . this discrepancy between correct final answers and flawed intermediate reasoning limits our ability to trust llms in scenarios where transparency and correctness of each reasoning stage are crucial @xcite . for example, in mathematical problem-solving, a model might produce the right answer for the wrong reasons @xcite , confounding our understanding of its true capabilities @xcite . to address this, researchers are increasingly emphasizing the importance of guiding models to produce not just correct final answers, but also verifiable and faithful step-by-step solution paths @xcite @xcite . prior work in finetuning has largely focused on outcome-level correctness, using outcome reward models to improve the probability of final-answer accuracy @xcite @xcite . while effective, such an approach does not ensure that the intermediate reasoning steps are valid. conversely, while processlevel supervision through process reward models (prms) @xcite @xcite can guide models to follow correct reasoning trajectories, prior work has mainly used prms as a ranking method rather than a way to provide stepwise feedback. as a result, relying solely on process-level supervision may lead models to prioritize step-by-step correctness without guaranteeing a correct final outcome. in this paper, we introduce stepwise kahneman-tversky-inspired optimization (step-kto), a training framework that integrates both processlevel and outcome-level binary feedback to produce coherent and correct reasoning steps alongside high-quality final answers. our approach evaluates each intermediate reasoning step against known correct patterns using a prm, while simultaneously leveraging a rule-based reward signal for the final answer. to fuse these signals, we employ a kahneman-tversky-inspired value function @xcite that emphasizes human-like risk and loss aversion, encouraging models to gradually correct their reasoning and avoid errors. the result is a training objective that aligns the entire reasoning trajectory with verified solutions while ensuring that final correctness remains a top priority. • we propose step-kto, a novel finetuning framework that combines process-level and outcome-level feedback, encouraging both correct final answers and faithful step-by-step reasoning. ## related work outcome-oriented methods. many efforts refine llms using only final outputs. instruction tuning @xcite and outcome-level feedback via reinforcement learning from human feedback (rlhf) (e.g., instruct-gpt @xcite ) or direct preference optimization (dpo @xcite , @xcite , simpo @xcite , @xcite ) improve final answer accuracy using human or synthetic labels. ai-generated feedback (rlaif @xcite ) or predefined rules (constitutional ai @xcite ) aim to reduce human annotation. while refinements like cgpo @xcite offer richer signals, they primarily evaluate entire outputs. a key limitation is that correct final answers do not guarantee sound intermediate reasoning @xcite , potentially yielding untrustworthy solution paths @xcite . process-level feedback and verification. process reward models (prms) @xcite @xcite focus on stepwise correctness, assigning local feedback to guide models toward logically consistent solutions. this is prevalent in math reasoning, supported by datasets like prm800k @xcite , criticbench @xcite , and processbench @xcite that facilitate step-level evaluations. prm-based techniques influence decoding @xcite @xcite , re-ranking @xcite , filtering @xcite , and iterative loops like star @xcite and @xcite . synthetic feedback helps scale annotations @xcite @xcite ). yet, focusing solely on process may not yield correct final answers, as local rewards can be exploited or chains may fail to converge @xcite . integrating outcome-and process-level signals. recognizing the limitations of supervising only outcomes or processes, recent studies combine both signals. facttune @xcite and factalgin @xcite integrate prms with factuality evaluators for alignment, enhancing factual accuracy. in math reasoning, @xcite and @xcite also leveraged combined step and outcome feedback. while the principle of multi-granularity supervision is broadly applicable, especially to math reasoning, these combined approaches can still face challenges in scaling, balancing feedback types, and avoiding premature performance plateaus @xcite @xcite . ## conclusion this work proposes step-kto, a training framework that leverages both outcome-level and process-level binary feedback to guide large language models toward more coherent, interpretable, and dependable reasoning. by integrating stepwise correctness signals into the alignment process, step-kto improves the quality of intermediate reasoning steps while maintaining or even enhancing final answer accuracy. our experiments on challenging mathematical reasoning benchmarks demonstrate consistent gains in performance, particularly under iterative training and for complex reasoning tasks. these findings underscore the value of aligning not only final outcomes but also the entire reasoning trajectory. we envision step-kto as a stepping stone toward more reliable reasoning in llms.