PORT : Preference Optimization on Reasoning Traces (2025)

Abstract: Preference optimization methods have been successfully applied to improve not only the alignment of large language models (LLMs) with human values, but also specific natural language tasks such as summarization and stylistic continuations. This paper proposes using preference optimization methods on Chain-of-Thought steps in order to improve the mathematical reasoning performances of language models. While the chosen answers are obtained from datasets that include reasoning traces, we propose two complementary schemes for generating rejected answers: weak LLM prompting, and digit corruption. Our approach leads to increased accuracy on the GSM8K and AQuA-RAT mathematical reasoning benchmarks for Falcon2-11B and Mistral-7B. Additionally, the improved abilities transfer to non-mathematical tasks, including the ARC benchmark and symbolic reasoning challenges. For example, our method can lead to up to relative 8.47 and 18.73 increases in accuracy on the GSM8K and AQuA benchmarks respectively, without any extra annotations. This work suggests that the path towards better language reasoning abilities goes through spending resources on creating high-quality datasets of reasoning traces.

## introduction in recent years, large language models (llms) have been pivotal in democratizing artificial intelligence (ai), given their ease of use and impressive abilities in a broad spectrum of tasks. while they have significantly contributed to the striking progress of ai, their success has heavily relied on scaling-up to ever-larger models and datasets. nonetheless, scaling has not proved sufficient for achieving satisfying results on tasks involving reasoning. reasoning has been a central theme in the history of ai, defining goal posts that push * denotes equal contribution † work initiated while at technology innovation institute the limits of intelligence. the term "reasoning" is often used to refer to informal reasoning, that "relies on intuition, experience, and common sense to draw conclusions and solve problems" @xcite . the limits of scale in eliciting reasoning abilities has been confirmed by analyses in @xcite ; @xcite ; @xcite , amongst others. one reason multistep reasoning still poses a challenge to llms is that the next-word prediction objective used to train them does not explicitly encourage step-by-step reasoning. chain-of-thought prompting (cot; @xcite , an augmented prompting strategy, has been shown to improve llm performances on reasoning tasks, by guiding them to generate sequences of intermediate steps. it should be unsurprising however that solely prompting a language model to "think step by step", whether alongside a handful of correct rationales @xcite or not @xcite , does not necessarily elicit actual system-2-like @xcite reasoning abilities, but at best only mimics humans' thought processes. despite claims of the type "llms are decent zero-shot reasoners" @xcite , the emergent ability of reasoning appears consistently for very large models (> 100b parameters) only @xcite . a major limitation of cot prompting is its reliance on large models @xcite . @xcite propose to bypass this limitation by generating rationales from very large teacher models and using them to fine-tune smaller student models. in the same line of work, @xcite perform a comprehensive comparison between outcome-based supeversied finetuning (sft), which supervises the final result, and process-based sft, which supervises the reasoning process, and find that process-based supervision significantly helps language models in mathematical reasoning tasks. however, solely relying on high-quality rationales is costly as it requires hu- ## experiments the goal of the experiments presented in this section are threefold. first, we empirically investi-gate the proposed approach, and show that unlike sft, it is less prone to task overfitting. then, we compare the two schemes above for constructing rejected answers, along with different combinations thereof. next, we investigate the effect of the preference optimization method by comparing dpo to some of its variants. finally, we validate the robustness of our method to both the base model and the training dataset. evaluation: to assess the approach, we need to evaluate the models on informal reasoning tasks, for which chains of thoughts can help reach the valid answers. our main evaluation task is the gsm8k test dataset @xcite , that contains 1319 high quality grade school math word problems. to assess the transfer abilities of our approach, we also consider the three following evaluation datasets: base model: as a base model, we use the newly released pre-trained falcon2-11b @xcite for all our experiments, except in section 3.6, where we confirm that our method is agnostic to the base model by using mistral-7b @xcite . training data: as previously mentioned, our method requires using a cot dataset. for all our experiments, except in section 3.7, we use the gsm8k train dataset @xcite , that consists of 7473 examples, given that it contains solution steps, in order to construct the sft and preference datasets d sft train and d pref train , as explained in section 2.2. example tuples (x i , z 1 i , . . . , z n i i ) from this dataset are provided in appendix a. it is important to note that throughout the training and evaluation process, we only use the training set, without any extra data or human annotation. ## related work what is reasoning? reasoning can be thought of as the process of logically and systematically an-alyzing information, drawing on evidence and past experiences to form conclusions or make decisions @xcite . using the taxonomy of @xcite , reasoning can be either deductive (a conclusion is drawn based on the truth of the premises), inductive (a conclusion is drawn based on observations or evidence), or abductive (a conclusion is drawn based on the best explanation for a given set of observations). @xcite also make the distinction between formal reasoning, akin to what is used in mathematics, in which a fixed set of rules is followed, and informal reasoning that is less structured, and is akin to what is used in everyday life. reasoning in llms: mathematics, science, and code benchmarks @xcite @xcite are becoming increasingly popular to study the emergent reasoning abilities of language models trained on next token prediction. chain-of-thought prompting @xcite and related techniques such as tree-of-thought @xcite and graph-of-thought @xcite have shown to improve language model performances on reasoning tasks, simply by prompting them to generate intermediate computations required for solving the problems. it is not clear however whether the improved performances brought about by chain-of-thought prompting are due specifically to human-like task decomposition, or more generally to the increased computation that additional tokens allow @xcite ). an orthogonal direction for boosting language model performances on reasoning tasks is reasoning-enhanced training. for example, @xcite ; @xcite ; @xcite show that training or fine-tuning llms on datasets containing scientific, math, or code data helps improve downstream performances on reasoning tasks. another line of work @xcite @xcite @xcite consists of using llms to self-improve their reasoning abilities via bootstrapping, where rationales generated by the model that lead to the correct answer are further used to fine-tune the model itself. aligned with this direction, and more closely related to our work, ni et al. (2023) propose to use intermediate steps as supervision signal. lightman et al. (2023) conduct a systematic comparison between process supervision (feedback on intermediate steps) and outcome supervision (feedback on final results) for training models on mathematical reasoning, finding that process supervision leads to significantly better performance on the math dataset. another popular set of approaches make use of verifiers, that classify or score reasoning traces @xcite . as an example of such approaches, grace @xcite trains a discriminator with a contrastive loss over correct and incorrect steps, and uses it when generating answers to questions requiring reasoning, to score next-step candidates based on their correctness. preference optimization: to make the most out of a preference dataset, reinforcement learning with human feedback commonly applies the bradley-terry model @xcite to train a reward model that scores instances, and use it to fine-tune the language model to maximize the score of the reward model for the prefered responses using algorithms such as ppo @xcite . more recently, advances in offline methods such as dpo @xcite and its variants @xcite @xcite that directly align the language models without the need for an explicit reward function, have proven successful in practice. these methods however require an sft phase to achieve convergence to desired results @xcite . orpo @xcite on the other hand, bypasses the need for the multi-stage process, and uses a loss that combines both supervised finetuning and preference optimization. in our work, we use these methods as part of the pipeline and compare them thoroughly. concurrent works @xcite have also applied preference optimization techniques on reasoning data. our work differs from these concurrent works in both methodology and scope. while pang et al. @xcite focuses on iteratively optimizing between competing cot candidates and @xcite proposes step-level preference optimization requiring fine-grained process supervision, our work introduces two novel and complementary schemes for generating rejected answers (weak llm prompting and digit corruption) that require no additional annotations or external data. furthermore, we provide a comprehensive empirical study comparing different preference optimization variants (dpo, ipo, kto, orpo) for reasoning tasks. unlike these works which focus solely on mathematical reason-ing, we demonstrate that our approach transfers to non-mathematical tasks, including commonsense and symbolic reasoning, suggesting broader implications for improving general reasoning abilities in language models. ## conclusion we considered the question of using preference optimization to boost reasoning abilities of language models. more specifically, we proposed two different schemes for constructing preference datasets of reasoning steps from datasets that include valid reasonign traces. we showed that by using dpo on these datasets, we are able to improve the reasoning abilities of falcon2-11b and mistral-7b, even on tasks unseen during training. we also compared dpo to several of its variants. our work suggests that constructing high-quality reasoning traces datasets can boost general informal reasoning abilities.