Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models (2024)

Abstract: The alignment of reasoning abilities between smaller and larger Language Models are largely conducted via supervised fine-tuning using demonstrations generated from robust Large Language Models (LLMs). Although these approaches deliver more performant models, they do not show sufficiently strong generalization ability as the training only relies on the provided demonstrations.In this paper, we propose the Self-refine Instruction-tuning method that elicits Smaller Language Models to self-improve their abilities.Our approach is based on a two-stage process, where reasoning abilities are first transferred between LLMs and Small Language Models (SLMs) via Instruction-tuning on synthetic demonstrations provided by LLMs, and then the instructed models self-improve their abilities through preference optimization strategies.In particular, the second phase operates refinement heuristics based on Direct Preference Optimization, where the SLMs are elicited to deliver a series of reasoning paths by automatically sampling the generated responses and providing rewards using ground truths from the LLMs.Results obtained on commonsense and math reasoning tasks show that this approach consistently outperforms Instruction-tuning in both in-domain and out-domain scenarios, aligning the reasoning abilities of Smaller and Larger language models.

## introduction previous works have demonstrated that chain-of-thought (cot) prompting can improve the large language models (llms) capacity to perform complex reasoning tasks by decomposing a reasoning task into a sequence of intermediate steps @xcite , where the generation of multistep controlled reasoning can improve results in commonsense @xcite , symbolic and mathematical @xcite reasoning datasets. since the size of llms represents an adoption barrier for many use cases, and smaller models do not seem to have the same emergent reasoning abilities as llms, several state-of-the-art alignment approaches for solving mathematical problems have emerged, where supervised fine-tuning (sft) has been used to train small language models (slms) using cot annotations. however, these annotations outline the intermediate reasoning steps for solving a given problem, which consists of a reasoning pathway generated by the llm for a specific case. this phenomenon can lead to a relatively weaker capacity of tuned models that have a few and limited number of samples. indeed, there are often multiple valid cot annotations for the same question @xcite , which underlines the need for a more general cot-based fine-tuning approach. in this paper, we propose self-refine instructiontuning, which is a method to enable cot reasoning over slms. our approach starts by performing instruction-tuning on slms via demonstrations delivered by llms and then applies preference optimization based on reinforcement learning (rl) heuristics to let the slms refine their abilities to solve a task in a step-wise manner. hence, proposing a teacher-student alignment method, we investigate the impact of transferring chain-of-thought reasoning abilities through the support of demonstrations "taught" by llms to slms as a warm-up to the self-refine process. therefore, to reinforce the instruction-tuning phase, we analyze whether preference optimization methods could strengthen student models' step-wise reasoning abilities. complementing the foundation work of @xcite , we introduce self-refinement based on reinforcement learning, and in contrast to @xcite @xcite , we use an instructiontuning via synthetic demonstrations approach as proposed by @xcite (i.e., a task-figure 1 : in self-refine instruction-tuning, the demonstrations delivered by teacher models are used to align reasoning abilities in a teacher-student setting. following the transference of step-wise reasoning knowledge via instruction tuning, the students self-refine their abilities with the support of direct preference optimization methods. this leads to the target research questions, which are the focus of this paper: rq1: how does instruction-tuning via demonstrations initialize the slms' reasoning abilities? rq2: what is the effect of the preference optimization algorithm on the alignment between teacher and student models? rq3: how much does the ability to solve tasks in a multi-step manner improve across different scenarios? we answer these questions by selecting three different slms: llama2-7b, llama2-13b (touvron et al., 2023), mistral-7b @xcite ; and three llms llama2-70b, mixtral @xcite and gpt-3.5 @xcite . in the teacher-student alignment phase, we operate via llms (teachers) to deliver synthetic demonstrations at the core of the instruction-tuning process (see left side of figure 1 ) used to instruct slms (students). in the self-refine phase, the students improve their step-wise reasoning abilities via direct preference optimization (dpo) @xcite . this phase allows the students to sample different reasoning paths and cot demonstrations and learn from them (see right side of figure 1 ). moreover, differently from previous works (based on teacher-student alignments), preferences are self-generated, and there is no need for a separately trained reward model as in the previous approaches @xcite . we demonstrate the effectiveness of the proposed refinement technique in aligning teacherstudent models, overcoming the differences highlighted by @xcite , from the same family, and in maximizing efficiency in indomain and out-domain tasks. our contributions can be summarized as follows: ## method to transfer the step-wise reasoning properties from large language models (llms) to small language models (slms), we propose self-refine instruction-tuning, a two-step approach as shown in figure 1 . in the first phase, there is a transfer of step-wise (cot) reasoning via instruction-tuning, where llms systematically generate demonstrations which are used by slms to initialize their step-wise (cot) alignment (section 2.1). in the second phase, the instructed slms self-refine their internal cot model via the preference optimization technique presented in section 2.2. ## experimental setup to evaluate the proposed method, we use commonsense and mathematical reasoning tasks (section 3.1) that are used to assess the step-wise inference properties of large language models(llms). regarding the self-refine instruction-tuning on the small language models (slms), we use the approach presented in section 3.2. ## results & discussion the self-refine instruction-tuning improves the alignment between large language models (llms) and small (slms) in both in-family and out-family settings. these conclusions can be observed in figure 2 and figure 3 , which reports the downstream accuracies without tuning (see the baselines), with only the instruction-tuning phase on demonstrations and after the self-refine phase. as discussed in section 4.1, the models with only instruction-tuning on synthetic demonstrations (generated by llms) transfers the reason-ing properties in a marginal way (see instructiontuned in figures 2 ). however, although teacher-student alignment via instruction-tuning produces better students, an improved alignment is achieved through the selfrefine phase, as discussed in 4.2. in particular, the 'self-refine instruction-tuning' bars in figure 2 show that the self-refined student models outperformed the student models tuned only with instruction-tuning ('instruction-tuning' bars on figure 2 ). furthermore, the alignment via demonstrations generated by teachers outside the same family (out-family) delivers more robust students (see figure 3 the self-refine instruction-tuning and (in-family) bars). finally, student models behind the self-refine phase outperformed others in both in-domain and out-domain tasks (discussed in section 4.3). hence, the self-refine mechanism effectively aligns teacher-student capabilities in out-domain tasks by enhancing performance even in the presence of fewer demonstrations (section 4.4). ## conclusion this paper proposes a novel approach for aligning multi-step cot reasoning between teacher large language models (llms) and student smaller lms (slms). in particular, our selfrefine instruction-tuning is framed as an instruction tuning via chain-of-thought demonstrations method based on explanations delivered by llms prompted by the cot mechanism, which is then reinforced via the self-refine phase that uses direct preference optimization. we also contrast the impact of in-family and out-family alignment across teacher and student models. the investigation of large language models' (llms) reasoning capabilities is a rapidly evolv-ing domain. multiple studies have been proposing techniques to increase, transfer, and align reasoning capabilities in english (detailed reported in section 5). several works are emerging that aim to transfer these reasoning capabilities beyond english @xcite . in future developments, we would like to extend our contribution to non-english languages to broaden their beneficial impacts and operability by extending previous alignment @xcite and augmentation @xcite approaches.