Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning (2024)

Abstract: Teaching small-scale language models to perform math reasoning is a valuable yet challenging task. Besides obtaining labeled data from human experts, one of the most common ways to collect high-quality data is by sampling from a larger and more powerful language model. Although previous works have demonstrated the effectiveness of this method, such a knowledge distillation paradigm can be costly and unstable, especially considering that many large language models, such as GPT-4, are closed-sourced, proprietary, and their behaviors are unpredictable. In this work, to avoid relying on outputs from large models, we demonstrate that the reasoning abilities of small-scale language models can be enhanced through self-training, which involves training models with their own outputs. We also show that the vanilla self-training can be further augmented by an alignment algorithm, direct preference optimization (DPO). We empirically found that models trained with the DPO objective are capable of making better generations that largely benefit multi-turn self-training. The experiments show our models outperform the state-of-the-art models with comparable sizes on a series of downstream math reasoning tasks with minimal resource requirements.

## introduction making language models (lms) perform mathematical reasoning is a valuable, yet challenging research objective @xcite . recent works focus on enhancing large-scale lms' reasoning abilities, e.g., chainof-thought prompting @xcite , continual pretraining @xcite , or adding external verifiers @xcite . however, the research question of how to our approach outperforms baseline models with comparable sizes while minimizing the required compute. comparisons include two knowledge distillation techniques, i.e., codex distillation @xcite , and palm distillation @xcite , and a data augmentation method, calcformer @xcite . recent studies @xcite @xcite demonstrate that the reasoning capabilities of smaller lms can be significantly enhanced through learning from the outputs of larger and more advanced lms, such as codex @xcite and palm @xcite . despite the relative ease of implementation, the associated costs can be substantial. the computational demand, measured in floatingpoint operations (flops), is considerably higher for large lms. additionally, the use of proprietary and closed-source large lms for data annotation can incur significant economic costs. @xcite demonstrated that annotating multiple reasoning chains for a single question with large lms can markedly enhance the performance of smaller models. hence, a trade-off between cost and performance exists. another line of work focuses on making improvements via self-training @xcite @xcite . instead of using data generated by larger models, the essence of self-training methods is to make small models learn from their own generations. while these methods demonstrate the effectiveness of utilizing self-generated data, their success largely depends upon the pre-existing abilities of the models. for example, @xcite started by few-shot prompting a large model, i.e., gpt-j @xcite , to self-generate rationales, which is an emergent ability that only comes with sufficiently large models @xcite . however, it is still unclear whether small-scale lms can benefit from self-training. recently, we have witnessed that reinforcement learning from human feedback (rlhf) has emerged as a prominent method to precisely modify lms' behavior towards human preference @xcite . in this work, we propose to augment the self-training algorithm with an rlhf training process, i.e., direct preference optimization @xcite , for its better performance and stability. our experimental results demonstrate that the proposed method can significantly improve lms' reasoning capabilities while minimizing the computational costs. we visualize the relationship between the downstream task performance and computational cost over a series of specialized models in figure 1 . the computational costs for each model are estimated following previous practice @xcite . it can be observed that our method not only achieves the highest accuracy, but also minimizes the computational demand by learning from its own generations. overall, the main contribution of this work can be summarized as follows: ## method in this section, we first describe the proposed approach. then, we demonstrate how we integrate an external calculator into the model's decoding process which significantly improves lms' performance on the downstream tasks. ## experiments in this section, we first describe our experimental setup. next, we present the performance of our models across a series of math word problem solving tasks, comparing them against a selection of competitive baselines. finally, we empirically analyze what makes the proposed method effective. ## conclusion we present an effective and resource-efficient method called dpo-augmented self-training (dpo-st), which augments the original self-training algorithm with direct preference optimization @xcite . unlike previous studies that improve small-scale language models' reasoning abilities by distilling a larger and more powerful model, we argue that small models that are trained merely on the limited human-labeled data can improve themselves significantly. we also empirically find that models trained with dpo loss can generate pseudo-labeled data with higher quality and diversity. our experiments demonstrate that the proposed method not only outperforms existing methods with comparable model sizes on the gsm8k benchmark, but also achieves remarkable resource efficiency in terms of both computational cost and the requirements of human-labeled data.