MCC - KD : Multi- C o T Consistent Knowledge Distillation (2023)

Abstract: Large language models (LLMs) have showcased remarkable capabilities in complex reasoning through chain of thought (CoT) prompting. Recently, there has been a growing interest in transferring these reasoning abilities from LLMs to smaller models. However, achieving both the diversity and consistency in rationales presents a challenge. In this paper, we focus on enhancing these two aspects and propose Multi-CoT Consistent Knowledge Distillation (MCC-KD) to efficiently distill the reasoning capabilities. In MCC-KD, we generate multiple rationales for each question and enforce consistency among their predictions by minimizing the bidirectional KL-divergence between the answer distributions. We conduct comprehensive experiments to investigate the effectiveness of MCC-KD with different model architectures (LLaMA/FlanT5) and various model scales (3B/7B/11B/13B) on both mathematical reasoning and commonsense reasoning benchmarks. The empirical results demonstrate that MCC-KD achieves superior performance on in-distribution datasets and exhibits a strong generalization ability on out-of-distribution datasets.

## introduction recently, large language models (llms) such as chatgpt have exhibited impressive emergent capabilities, showcasing their competence in various tasks, including those demanding complex reasoning. while directly providing answers without generating intermediate steps may lead to errors and limited interpretability, chain of thought (cot) @xcite prompting enables llms to break down reasoning tasks into a series of intermediate steps, guiding the model to generate the subsequent steps before arriving at the final answer. the effectiveness of cot prompting has been demonstrated on diverse reasoning tasks @xcite . despite the effectiveness of cot prompting, recent studies @xcite @xcite have shown that these reasoning capabilities only manifest in language models with over 100 billion parameters, such as palm (540b) @xcite and gpt-3 (175b) @xcite . these llms with massive parameter sizes require significant computational resources during both training and inference, which restrict their deployment on resource-limited platforms. while llms could be accessed through api calls, there are still several challenges to overcome, including network instability, difficulty in customizing the models, and privacy concerns. therefore, an alternative approach is to deploy smaller language models such as llama-7b/13b @xcite and flant5-xl/xxl @xcite , which have fewer than 13 billion parameters. through knowledge distillation (kd) @xcite , the reasoning capabilities can be transferred from llms to these smaller models. however, traditional kd techniques require the teacher model to provide output logits or hidden layer features, which cannot be readily applied to llms due to the limited accessibility of their internals. one potential solution is to leverage rationales generated by llms to train smaller mod-els, thereby acquiring their reasoning abilities @xcite @xcite . however, these methods for rationale distillation also face several challenges. firstly, the limited diversity in reasoning paths may lead to a dilemma where the student model simply mimics the superficial style of the teacher model's outputs @xcite or overfits the training data, resulting in limited generalization capabilities. secondly, despite the existence of multiple rationales leading to the same answer for each given question (as depicted in figure 1 ), these methods neglect the consistency among different rationales in reaching the predicted answer when training the student model. such oversights can undermine the stability of student models during training and impair their generalization capabilities. to address these challenges, we propose multi-cot consistent knowledge distillation (mcc-kd), a novel solution that incorporates two pivotal characteristics. firstly, this approach leverages multiple diverse rationales for each given question and aims to improve their consistency in predicting the answer. this improvement is expected to enhance the stability and generalizability of the student models. secondly, we introduce a similaritybased method to facilitate the selection of diverse rationales. mcc-kd draws inspiration from realworld teaching scenarios, where presenting multiple distinct solutions to one problem benefits the student's learning process. with these inherent advantages, mcc-kd enables the smaller models to acquire reasoning capabilities from larger models through effective knowledge distillation. we conduct extensive experiments with llama @xcite and @xcite on both mathematical reasoning and commonsense reasoning benchmarks. the empirical results demonstrate the effectiveness and superiority of mcc-kd over previous cot-based knowledge distillation methods. for example, mcc-kd achieves an accuracy improvement from point 38.01 to 41.58 on the gsm8k @xcite dataset with llama-7b. moreover, the generalization experiments reveal that mcc-kd achieves a substantial accuracy improvement, raising the performance from point 47.69 to 49.52 on the outof-distribution dataset asdiv (miao et al., 2020) using flant5-xxl. these findings provide compelling evidence of the effectiveness and robustness of mcc-kd. ## related work in this section, we briefly review the related work on chain of thought and knowledge distillation. ## method we introduce multi-cot consistent knowledge distillation (mcc-kd), an approach designed to enhance the generalization and robustness of smaller student models during knowledge distillation. in particular, mcc-kd enforces the consistency among diverse chain of thoughts (cots) generated by the teacher llms in three key steps. firstly, given an input question, we utilize zeroshot cot prompting @xcite to obtain multiple cots as rationales from a llm such as gpt-3.5. secondly, we filter out rationales that exhibit limited diversity. lastly, we impose constraints on the outputs from multiple rationales to facilitate their consistency. the overall framework of mcc-kd is illustrated in figure 2 . ## experimental setup in this section, we present the datasets and backbone models utilized in our experiments. ## results and analysis in this section, we present the main results, ablation studies, and additional experiments. ## conclusion in this paper, we propose multi-cot consistent knowledge distillation (mcc-kd) to transfer reasoning capabilities from larger language models (llms) to smaller models. the primary objective is to address the diversity and consistency challenges present in existing knowledge distillation methods for this purpose. our approach leverages multiple rationales for each given question and focuses on improving their consistency in predicting the answer. extensive experiments are conducted using different model architectures, including llama and flant5, and a range of model scales, such as 3b, 7b, 11b, and 13b. the experiments cover both mathematical and commonsense reasoning benchmarks. the results clearly demonstrate the superior performance of mcc-kd on both in-distribution and out-of-distribution tasks. these findings confirm that mcc-kd enhances the stability and generalizability of the student models.