Mind’s Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models (2024)

Abstract: Large language models (LLMs) have achieved remarkable advancements in natural language processing. However, the massive scale and computational demands of these models present formidable challenges when considering their practical deployment in resource-constrained environments. While techniques such as chain-of-thought (CoT) distillation have displayed promise in distilling LLMs into small language models (SLMs), there is a risk that distilled SLMs may still inherit flawed reasoning and hallucinations from LLMs. To address these issues, we propose a twofold methodology: First, we introduce a novel method for distilling the self-evaluation capability from LLMs into SLMs, aiming to mitigate the adverse effects of flawed reasoning and hallucinations inherited from LLMs. Second, we advocate for distilling more comprehensive thinking by incorporating multiple distinct CoTs and self-evaluation outputs, to ensure a more thorough and robust knowledge transfer into SLMs. Experiments on three NLP benchmarks demonstrate that our method significantly improves the performance of distilled SLMs, offering a new perspective for developing more effective and efficient SLMs in resource-constrained environments.

## introduction with the gradual increase in the number of parameters, large language models (llms) have achieved significant successes in the field of natural language processing @xcite @xcite @xcite . however, the tremendous model sizes and computational requirements of llms introduce challenges to their practical application, especially in resource-limited environments @xcite . to address these challenges, various studies have delved into the * corresponding authors. compression of llms into small language models (slms) using knowledge distillation techniques and have led to significant reductions in computational complexity and inference costs @xcite @xcite . this process involves traditional teacher-student learning methods and the more recent chain-of-thought (cot) distillation method @xcite . the cot distillation methods use the cot reasoning process of llms as supervision for training slms, rather than just labels. this allows slms to learn the reasoning process of llms, thereby improving the performance of slms. while these cot distillation methods have proven to be beneficial, they are not without their flaws, particularly: training slms with these flawed cots will result in slms inheriting these flaws and performance degradation @xcite . ## distilling self-evaluation capability and comprehensive thinking we propose a new methodology for distilling the self-evaluation capability and comprehensive thinking of an llm into an slm. our overall framework is illustrated in figure 2 , which operates in 4 steps: ## experiments tasks and datasets to evaluate our distillation method, we conduct comprehensive experiments on three tasks: 1) math word problems (mwps) task with the svmap dataset @xcite ; 2) commonsense question answering (cqa) task with the cqa dataset @xcite ; 3) natural language inference (nli) task with the anli dataset @xcite . for dataset samples, we use either human-annotated labels from the dataset or llm-generated pseudolabels to explore the effect of human annotation availability on our method. setup in distillation, we utilize gpt-3.5-turbo as the llm foot_0 . we utilize 5-shot cot prompting to enhance the quality and standardize the formats of the responses generated by the llm. we follow the cot prompts from @xcite for the cqa dataset and devise similar prompts for other datasets and self-evaluation. to strike a balance between diversity and cost, in the main experiment, we obtain five cots for each training instance and five self-evaluation outputs of each cot from the gpt-3.5-turbo model and choose the t5-base model (220m) @xcite as the slm. we provide more experimental details in appendix a. we also explore the effect of the value of the hyperparameter λ on the results, which are presented in appendix b. therefore, we select λ = 0.5 as the optimal hyperparameter for our main experiments. in all experiments, we report the mean results and standard deviations over 3 random runs. ## conclusion in this study, we have introduced an innovative method to effectively distill the more comprehensive capabilities from llms into slms, emphasizing both the transfer of self-evaluation capability and comprehensive thinking, to mitigate the shortcomings of previous cot distillation methods. comprehensive experiments demonstrate that our method outperforms prior distillation methods consistently in various nlp tasks, significantly improving the performance and reliability of slms. we hope that this study can promote the more effective and efficient utilization of slms, especially in resource-limited environments.