Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning (2025)

Abstract: Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought (CoT) prompting. However, CoT prompting greatly increases computational demands, which has prompted growing interest in distilling CoT capabilities into Small Language Models (SLMs). This study systematically examines the factors influencing CoT distillation, including the choice of granularity, format and teacher model. Through experiments involving four teacher models and seven student models across seven mathematical and commonsense reasoning datasets, we uncover three key findings: (1) Unlike LLMs, SLMs exhibit a *non-monotonic* relationship with granularity, with stronger models benefiting from finer-grained reasoning and weaker models performing better with simpler CoT supervision; (2) CoT format significantly impacts LLMs but has *minimal* effect on SLMs, likely due to their reliance on supervised fine-tuning rather than pretraining preferences; (3) Stronger teacher models do *NOT* always produce better student models, as diversity and complexity in CoT supervision can outweigh accuracy alone. These findings emphasize the need to tailor CoT strategies to specific student model, offering actionable insights for optimizing CoT distillation in SLMs.

## introduction large language models (llms) have demonstrated exceptional capabilities through extensive pretraining on diverse human language data @xcite @xcite @xcite . chain-of-thought (cot) prompting has further enhanced their abilities by guiding llms to generate intermediate reasoning tokens, which emulate human cognitive processes and improve interpretability @xcite . advances in cot prompting have explored techniques like extending reasoning steps @xcite and refining reasoning formats @xcite . however, cot's token-intensive nature significantly increases computational demands @xcite , limiting its practicality in resource-constrained settings. this has spurred interest in distilling cot capabilities into small language models (slms) as a more efficient alternative @xcite . since slms often struggle to independently generate effective cot reasoning solutions @xcite , distilling cot capabilities requires fine-tuning slms on teacherannotated cot datasets, where the teacher can be either human experts or more powerful llms. while previous research has demonstrated successful distillation of cot capabilities into slms @xcite @xcite , the choice of teacher annotators and cot generation methods has often been arbitrary. a critical, yet unexplored, research question remains: what is the most effective cot supervision for training a student model to develop robust reasoning capabilities? analogous to how human teachers instruct students, there are three key factors that influence how effectively a student absorbs knowledge: building on this analogy of how human teaching impacts student performance, we conducted extensive experiments on four mathematical reasoning datasets of varying difficulty and three commonsense reasoning datasets, using four teacher models to distill reasoning skills to seven student models. we adopted a 1-shot prompting approach for generating cot annotations, which we found to be the most effective in maintaining consistency in teaching style while controlling granularity. our key findings are: (1) while llms benefit monotonically from detailed steps, slms exhibit an nonmonotonic relationship. stronger student models benefit from finer granularity, while weaker ones can be overwhelmed by excessive explanations and prefer simpler cot annotations; (2) cot format changes influence llms, likely due to their pretraining preferences for certain structures, but this effect is less pronounced in slms, which adapt more readily to diverse formats during fine-tuning; cot prompting cot prompting @xcite has become a pivotal technique for enhancing reasoning capabilities in llms by introducing intermediate reasoning steps. automated approaches like auto-cot @xcite , treeof-thoughts @xcite and self-play mutual reasoning @xcite explore multiple reasoning paths to expand the search space and improve task accuracy. these methods focus on increasing the reasoning length or expanding the reasoning horizon to handle complex tasks. recent studies have underscored the importance of reasoning granularity and formats in enhancing llm performance. for instance, @xcite identified that longer reasoning steps improve task success for complex problems, while overly concise steps can reduce effectiveness. tailored reasoning formats @xcite @xcite have demonstrated substantial improvements across tasks. however, these reasoning optimization strategies often comes with significant computational costs @xcite , raising concerns about the trade-off between accuracy and efficiency. knowledge distillation while direct prompting enables llms to perform complex reasoning through cot, slms struggle due to limited capacity @xcite . knowledge distillation (kd) provides an effective framework for transferring the reasoning capabilities of teachers to slms @xcite . a simple yet effective approach is using a teacher-student paradigm, which employs teacher-generated cot steps to guide slms, addressing their limitations and enhancing reasoning-intensive task performance @xcite @xcite . despite these advances, a systematic exploration of how to balance reasoning granularity, format, and teaching strategies remains lacking. addressing these gaps is crucial for optimizing cot distillation and enabling efficient reasoning in slms. ## problem formulation let @xmath0, y i )} n 1 denote a reasoning dataset with n (x i , y i ) pairs. chain-of-thought distillation aims to train a student s to generate intermediate reasoning steps c i for each input x i in order to generate the right y i . the optimal c i to train the student model is influenced by three key pedagogical factors: choice of teacher, granularity of teaching, and format of teaching: choice of teacher the teacher model t generates a reasoning chain c t (x i ) for each input x i , which guides the student model in producing the correct answer. the teacher can either be an llm or a human with varying styles and expertise. ## effects of granularity while previous research has shown that increasing reasoning granularity improves llm performance through detailed intermediate steps @xcite , slms differ fundamentally from llms in their ability to process complex reasoning chains. this raises a critical question: does increasing reasoning granularity still yield consistent benefits for slms in the task of cot distillation? in this section, we investigate this question using gpt-4o as the teacher model.