Aligning Large and Small Language Models via Chain-of-Thought Reasoning (2024)

Abstract: Chain-of-Thought (CoT) prompting empowersthe reasoning abilities of Large Language Models (LLMs), eliciting them to solve complexreasoning tasks in a step-wise manner. However, these capabilities appear only in models with billions of parameters, which represent an entry barrier for many users who are constrained to operate on a smaller model scale, i.e., Small Language Models (SLMs). Although many companies are releasing LLMs of the same family with fewer parameters, these models tend not to preserve all the reasoning capabilities of the original models, including CoT reasoning.In this paper, we propose a method for aligning and transferring reasoning abilities between larger to smaller Language Models. By using an Instruction-tuning-CoT method, that is, an Instruction-tuning designed around CoT-Demonstrations, we enable the SLMs to generate multi-step controlled reasoned answers when they are elicited with the CoT mechanism. Hence, we instruct a smaller Language Model using outputs generated by more robust models belonging to the same family or not, evaluating the impact across different types of models. Results obtained on question-answering and mathematical reasoning benchmarks show that LMs instructed via the Instruction-tuning CoT method produced by LLMs outperform baselines within both in-domain and out-domain scenarios.

## introduction chain-of-thought (cot) prompting elicits large language models (llms) to break down a reasoning task towards a sequence of intermediate steps @xcite . previous works have demonstrated that in llms with at least several billions of parameters, such as within the gpt @xcite or palm @xcite families, cots enable the delivery of multi-step, controlled reasoning, improving results across commonsense we investigate different properties between the teacherstudent models, including the impact of in/out family alignment and the impact of different demonstration styles within the teacher-student alignment. @xcite , symbolic and mathematical reasoning datasets @xcite . the size of llms, however, presents an adoption barrier for certain users and specific use case scenarios. to facilitate accessibility, derived scaled-down models from the same family but with reduced size have been introduced, such as llama-2-7b and -13b as the corresponding 'smaller language models (slms)' associated with llama-2-70b @xcite . although these slms are highly functional across different tasks, the cot prompting mechanism only proved to be consistently operational for models at a certain scale (e.g., with more than 60b parameters @xcite ). these slms produce illogical answers when prompted under the cot framework. in this paper, we propose a method to enable cot reasoning over slms (named student models) by performing instruction-tuning via demonstrations delivered by llms (teacher models). moreover, we introduce the concept of in-family alignment for teacher-student instruction-tuning. hence, we investigate the induction and alignment of chainof-thought reasoning abilities through the support of cot-demonstrations "taught" by llms teachers to slms students (see figure 1 ), contrasting between in-family and out-family settings. complementing the foundation work of @xcite we introduce the instruction-tuning cot approach (i.e., a taskoriented specialization of supervised fine-tuning) through which we instruct student models with cot-demonstrations produced by in-family and out-family teachers. this leads to the target research questions, which are the focus of this paper: rq1. how does instruction-tuning via demonstrations impact the reasoning abilities of students models? rq2. what is the effect of demonstrations delivered with the chain-of-thought reasoning process? rq3. how much do demonstrations produced by an in-family teacher impact the student models' performances? to answer these questions, we selected llama-2-7b and llama-2-13b @xcite as students and llama-2-70b and gpt-3.5 as, respectively, in-family and out-family teachers. then, we conduct an extensive analysis using different types of benchmarks, from arithmetic reasoning to commonsense tasks. experimentally, we contrast llama-2-70 and gpt-3.5 as teacher models to deliver cot-demonstrations and answers (see figure 1 ) which are used to instruct llama-2-7 and -13. we discern the cot-demonstrations between demonstrationsdelivering cot and demonstrations-misleading cot stems from answers-delivering cot (correct associated cot prediction) and misleading cot (wrong cot predictions). furthermore, to have a term of comparison, we produce the base demonstrations formed the same way as the previous ones without cot prompting. figure 5 shows the terminology used in this work. we show that the instruction-tuning approach on demonstrations instructs student models, and they consistently outperform baseline slms in all proposed benchmarks. finally, students in-structed with demonstrations-delivering cot provided by the in-family teachers outperformed those instructed by out-family and achieved the best performances. our findings can be summarized as follows: ## method in order to align the reasoning abilities of smaller language models using the step-wise reasoning knowledge generated by larger language models, we propose a two-phase alignment approach. in the first part, there is an automated 'annotation phase' where the large language models (llms) systematically prompt generate outputs (section 2.1). these demonstration outputs will be used during the second phase which will perform the instruction-tuning from the smaller language models (section 2.2). ## experimental setup in order to make the experiments comparable with state-of-the-art models, we use two math word problems and four closed-ended question answering benchmarks (introduced in section 3.1) that are generally used to assess the inference abilities of large language models (llms). moreover, to conduct the instruction-tuning phase on the small language models (slms), we use the approach presented in section 3.2. the experimental pipeline and the supporting model is available at github.com/lranaldii/aligning_llms. ## results & discussion language models that were unable to reason can be elicited to do it through the knowledge of teacher models. these conclusions can be observed in figure 2 , which report the downstream accuracies without the instruction-tuning phase (see the baseline) and the instruction-tuning phase on demonstrations. in fact, as discussed in section 4.1, small language models (slms) cot prompted obtained weak results. in contrast, models that are instructed via chain-of-thought (cot) demonstrations, i.e., demonstrations produced by cot-prompted large language models (llms), outperform other models (see the instruction-tuned-cot in figure 2 ). however, although cot-demonstrations produced better students, an improved alignment between students and teachers can be observed via the demonstrations-delivering cot mechanism, as discussed in section 4.2. in particular, the "demonstrations-delivering cot" and "truthful demonstrations" bars in figure 2 show that student models instructed via demonstrations-delivering cot outperformed students instructed via cot-demonstrations, which contained demonstrations misleading cot. finally, students instructed with demonstrationsdelivering cot produced by in-family teachers always outperformed students instructed with demonstrations-delivering cot produced by outfamily teachers. in figure 2 , it is possible to observe the phenomenon of family-alignment between llama-2-70 and llama-2-7 and -13. additional details can be found in section 4.2 and section 4.5. ## conclusion in this paper, we analyzed the alignment of stepwise cot reasoning between teacher large language models (llms) and student small language models (slms). in particular, we propose instruction-tuning-cot, an instruction tuning via chain-of-thought (cot) demonstrations, based on explanations delivered by llms prompted with the cot mechanism. we also contrast the impact of infamily and out-family alignment across teacher and student models. the results highlight the impact of teacher-student instruction-tuning interventions as a mechanism to improve the step-wise reasoning properties of smaller language models.