Large Language Models Are Reasoning Teachers (2023)

Abstract: Recent works have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks, step-by-step. However, prompt-based CoT methods are dependent on very large models such as GPT-3 175B which are prohibitive to deploy at scale. In this paper, we use these large models as reasoning teachers to enable complex reasoning in smaller models and reduce model size requirements by several orders of magnitude. We propose Fine-tune-CoT, a method that generates reasoning samples from very large teacher models to fine-tune smaller models. We evaluate our method on a wide range of public models and complex tasks. We find that Fine-tune-CoT enables substantial reasoning capability in small models, far outperforming prompt-based baselines and even the teacher model in many tasks. Additionally, we extend our method by leveraging the teacher model’s ability to generate multiple distinct rationales for each original sample. Enriching the fine-tuning data with such diverse reasoning results in a substantial performance boost across datasets, even for very small models. We conduct ablations and sample studies to understand the emergence of reasoning capabilities of student models. Our code implementation and data are available at https://github.com/itsnamgyu/reasoning-teacher .

## introduction language models (lms) have demonstrated remarkable performance in a wide range of downstream tasks. recently, large language models (llms) have demonstrated in-context generalization capabilities: performing downstream tasks simply by conditioning on few in-context exemplars or plain natural language task descriptions @xcite . despite these advancements, even the largest llms have been found to struggle with complex tasks which require multiple reasoning steps @xcite . 1 our code implementation and data are available at https://github.com/itsnamgyu/reasoning-teacher . ## chain-of-thought fine-tuning we propose fine-tune-cot, a task-agnostic approach to enable chain-of-thought reasoning in small lms. the core idea is to generate reasoning samples from very large teacher models using cot prompting and subsequently fine-tune small student models using the generated samples. this approach preserves the versatility of prompt-based cot methods while overcoming their reliance on prohibitively large models. to maximize versatility and minimize teacher inference costs, we use the task-agnostic zero-shot-cot prompting method @xcite on teacher models, as it does not require any reasoning examples or long inference context. we discuss our choice of teacher cot prompting method in section 7.3. in the following, we characterize fine-tune-cot in three distinct steps. we also provide a visual overview in figure 2 . ## experiments tasks and datasets we evaluate our method on 12 datasets pertaining to four categories of complex reasoning, following @xcite . these include arithmetic (singleeq, addsub, multiarith, gsm8k, svamp), other (date understanding, tracking shuffled objects), symbolic (last letter concatenation, coin flip), and common sense (commonsenseqa, strategyqa) reasoning. we provide details and references in appendix b. models for teacher models, we use four variants of gpt-3 175b @xcite , provided by the openai api. unless otherwise stated, we use text-davinci-002 based on instructgpt 175b @xcite as the teacher for finetune-cot. for student models, we consider four popular model families. for our main experiments, we use gpt-3 {ada, babbage, curie} as they are readily available for fine-tuning via the ope-nai api. due to the blackbox nature of the api, we also consider various open-source models under controlled settings. we use gpt-2 {small, medium, large} @xcite and t5-{small, base, large} @xcite as representative model families for decoder-only and encoder-decoder architectures, respectively. we also use the instruction-tuned version of t5, flan-t5-{small, base, large} @xcite , to investigate the effects of instruction tuning on student models, prior to applying fine-tune-cot. these student models are 25-2500x smaller than the teacher model, thus considerably more feasible for realworld deployment. we provide details on models and api usage in appendix c. ## conclusion we have proposed fine-tune-cot, a method that uses llms as reasoning teachers to transfer the broad reasoning capabilities previously found @xmath0.3b. we propose diverse reasoning as a novel approach to maximize these teaching effects, exploiting the unique characteristics of this new learning setup to vastly improve performance. our extensive experiments show that fine-tune-cot elicits significant reasoning performance in small models, thus demonstrating the distillation of cot reasoning which has been considered an emergent ability of scale. by leveraging publicly available models with zero-shot prompting, we demonstrate a taskagnostic approach to elicit reasoning performance in small models, making complex reasoning feasible for real-world deployment and accessible to the broader community.