Self-training Language Models for Arithmetic Reasoning (2024)

Abstract: Recent language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection of more annotated data.In this work, we explore the potential of improving models’ reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training).In systematic experimentation across six different arithmetic reasoning datasets, we find that models can substantially improve in both single-round (offline) and online self-training, reaching a correct result in +13.9% and +25.9% more cases, respectively, underlining the importance of actuality of self-training feedback. We further find that in the single-round, offline self-training, traditional supervised training can deliver gains comparable to preference optimization, but in online self-training, preference optimization methods largely outperform supervised training thanks to their superior stability and robustness on unseen types of problems.

## introduction despite recent improvements in the practical usability of language models (lms) attributed to preference alignment methods @xcite , these models often struggle with tasks requiring reasoning, i.e., a process of inferring a conclusion or decision logically and systematically @xcite . previous work improves the reasoning capabilities of language models by scaling training data to more diverse @xcite or complex @xcite collections, but reaching further improvements in this direction becomes exceedingly expensive. sample a problem, generate 16 predictions training data variant -preferences: • prediction 2 > prediction 1 • prediction 2 > prediction 3 • ... • prediction 16 > prediction 15 variant -sft: • prediction 2 • prediction 5 • ... • prediction 16 feedback: is result correct? ✘ ✔ ✘ ✘ ... ✘ ✔ self-training loop figure 1 : schema of self-training that we apply to provide the model with training feedback to its predictions. in the offline variant, the model generates all predictions in a single round. in the online variant, the training data is continuously generated. in this work, we evaluate the potential of improving models' capabilities by training from implicit, automated feedback to models' responses. arithmetic reasoning tasks present a challenge that reflects heavily on the model's reasoning capabilities, while the quality of the model's responses can be automatically assessed against the annotated correct results rather than expensive and possibly subjective judgments of model outputs @xcite . thus, we choose the arithmetic reasoning to address our two main research questions: rq1: to what extent can we improve the reasoning abilities of language models without additional data? rq2: can the preference optimization bring further improvements to models' capabilities over traditional supervised fine-tuning? we address these questions by implementing two variants of self-training: (1) an offline variant, where the training feedback to the model responses is constructed in a single iteration ( §3.1), and (2) an online variant, where the model obtains and trains on the feedback to its current predictions ( §3.2). our experiments reveal that both self-training variants present an efficient method for improving lms' capabilities with implicit training signal; both variants allow to significantly improve the initial model without any new data. in the offline variant, similar improvements can be achieved by both supervised and preference optimization methods. however, the online variant reveals crucial issues in scaling the supervised training to autonomous settings. on the contrary, preference optimization methods can robustly persist the original capabilities even in autonomous self-training while reaching further improvements. finally, the difference in average improvement between our best-performing offline (+13.9%) and online method (+25.9%) indicates that the actuality of self-training feedback is a crucial factor of self-training effectivity. our results motivate future research towards exploring new sources of implicit feedback able to provide language models with immediate feedback to their current predictions. ## related work we build upon a line of previous work that experiments with providing feedback to language models in arithmetical reasoning. notably, @xcite train models with ppo @xcite against feedback on individual steps given by chatgpt 3.5. uesato et al. (2022) our work is closest to @xcite and @xcite . @xcite apply self-training with a traditional supervised objective: they train the model on a small set of seed data and continuously use the trained model to generate solutions for a larger set, from which correct solutions are used in another training epoch. they show that three such subsequent epochs can improve the accuracy with diminishing returns. @xcite experiment with self-training with supervised fine-tuning on commonsense and math reasoning. they report positive results of self-training on the model's reasoning capabilities under specific conditions: (1) the initial model must be capable enough to be able to achieve improvements, and (2) training tasks must hold a negligible chance of random success (unlike, e.g., binary classification). our work builds upon these findings but differs from previous work in our objectives and data setting; we provide a systematic comparison of different training objectives in both online and offline settings, including the most recent preference optimization methods and show that training objective indeed plays a crucial role, especially in the online setting. our data setting is more ambitious than of previous work: we show that self-training can deliver substantial improvements also by using only problems already seen in previous training. finally, contrary to previous self-training work, we make our code foot_0 and models foot_1 freely available to accelerate future work in self-training. ## experiments our experiments build upon the 3-billion-parameter flan models fine-tuned specifically for arithmetic reasoning in previous work of @xcite . these relatively compact calculator-assisted models called calcformers were shown to perform noticeably well on multi-step reasoning, while even on single-step and two-step problems perform compared to 70b llama-2 @xcite ). another desiderata of these models is the transparency of their training data. in our experiments, this allows us to opt for a more challenging yet realistic self-training setting where we do not train the models on any new data, but only on the problems that calcformers have already seen in the training, merely with a complementary training signal. specifically, we self-train these models with the prompts from ape210k @xcite , to our knowledge the largest available arithmetical reasoning dataset of over 200,000 math problems. in addi- in both self-training variants, we use the trained model to generate training data (see fig. 1 ). the generated data consists of the original input prompt (x i ) and associated model predictions (y i ) in the form of a chain-of-thought sequence containing the model's final result at the end. for each prompt, we generate 16 predictions using sampled generation. annotations of correct results then allow us to automatically annotate each prediction for either being correct (y ok i ), or incorrect (y nok i ), assigning a set of both correct and incorrect predictions to each input prompt. for the supervised fine-tuning (sft) objective, we construct the training dataset from pairs of (x i , y ok i ). sft uses a standard next-token prediction with cross-entropy loss and teacher forcing @xcite . all preference optimization (po) objectives then train on triples (x i , y ok i , y nok i ), with the y ok i marked as being preferred over y nok i . we experiment with three recent preference opti-mization methods: direct preference optimization; dpo @xcite , kahneman-tversky optimization; kto @xcite and identity preference optimization; @xcite . these methods differ in a variety of aspects in the formulation of training loss. for brevity, we direct the reader to the referenced work for further details on preference optimisation methods. further details of our general training setup can be found in appendix a. ## conclusions this work explores the potential of autonomously improving language models for arithmetic reasoning: a task allowing automated, immediate, and objective feedback based on the correct results. we experiment with two settings: (i) offline selftraining, collecting the feedback in a single iteration, and (ii) online self-training, where the model trains continuously from feedback to its up-todate predictions. in both settings, we apply and compare recent preference optimization methods (dpo, kto, ipo) with standard supervised training (sft). we find that self-training provides an opportunity to improve models' capabilities without any new data, using exclusively models' own predictions and automated feedback. in addition to the offline variant, online self-training provides further opportunities for data-free improvements attributed to the enhanced robustness of preference optimization methods. our work motivates future work towards seeking other sources of implicit training feedback beyond arithmetic reasoning, exemplified in previous work in a reasoning coherence @xcite or consistency @xcite . presenting language models with novel sources of implicit feedback via self-training can fill the gap of the traditional, largely simplified training objectives and empower models to capture more complex structural dependencies necessary in many real-world applications.