Towards Efficient C o T Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales (2025)

Abstract: CoT distillation is critical for enhancing small language models’ (SLMs) reasoning by transferring multi-step reasoning capability from the larger teacher models. However, existing work underestimates the importance of rationale quality, focusing primarily on data quantity, which may result in transferring noisy or incorrect information to the student model. To address the above issues, we proposed Model-Oriented Rationale Selection Distillation (MoRSD), which can discern and select high quality rationales for distillation. We further propose a Rationale Difficulty (RD) metric to measure the ability of the student model to generate the correct answer under a given rationale. Compared to the baseline, we achieved 4.6% average accuracy improvement on seven datasets over three tasks, using fewer rationales by controlling their accuracy, diversity, and difficulty. Our results reveal that a small portion of the high quality rationales can enhance the reasoning ability of student models than the entire dataset. Our method promises to be a possible solution for efficient CoT distillation. Our code will be released in https://github.com/Leon221220/MoRSD .

## introduction large language models (llms) such as llama, @xcite deepseek-r1, have achieved remarkable performance in various reasoning tasks by instructing them to think stepby-step @xcite @xcite @xcite @xcite . engaging in reasoning through logically coherent steps has substantially enhanced performance in tasks such as mathematical problem solving and question answering. these intermediate reasoning steps are referred to as rationale @xcite . to achieve emergent reasoning abilities, llms require large-scale parameters, making slms inherently limited @xcite @xcite . cot distillation has become a key technique for enhancing slm reasoning by transferring rationales from stronger teachers @xcite , showing strong results on arithmetic and symbolic tasks @xcite @xcite . beyond basic distillation, recent works explore consistency enforcement @xcite , cross-task supervision @xcite , and tailored strategies @xcite . mentor-kd @xcite introduces intermediate models for better supervision, mcc-kd promotes consistent yet diverse reasoning @xcite , while lion @xcite and ta-inthe-loop @xcite use adversarial and auxiliary guidance, respectively. however, these approaches often require additional models, discard useful failures, or introduce iterative overhead-resulting in high computational costs and limited flexibility. and many works still rely on enlarging the rationale set (increasing from 1 to 8 per instance @xcite ) to improve performance, while overlooking rationale quality. such data scaling ignores variance in correctness and diversity, risking the distillation of noisy signals. furthermore, most approaches neglect the specificity of student models, failing to adapt to their strengths or limitations. these limitations motivate our focus: how to select a small set of high-quality, student-aware rationales for efficient and effective distillation. to overcome these limitations, we propose morsd, a simple but effective method that enables student models to customize their distillation data autonomously. as presented in figure 1 , morsd consists of four stages: 1) rationale generation, 2) self-evaluation, 3) rationale selection and 4) distillation. the rationale generation stage prompts the teacher llm to generate the rationale dataset. in the self-evaluation stage, we calculate rationale difficulty (rd) to measure the contribution of a given rationale to distillation. specifically, rd measures the student's ability to generate the correct answer given a question and rationale. those with smaller rd are considered more beneficial to generate the corresponding answer. then, we first apply model-agnostic accuracy selection and diversity selection to the rationale dataset. accuracy selection adjusts the proportion of correct rationales in the dataset to achieve the given accuracy threshold, diversity selection involves pairwise jaccard similarity to eliminate similar rationale in the dataset. finally, we use difficulty selection to select the rationales with smaller rd. since difficulty selection uses perplexitybased rd, a model-specific metric, it enables the student model to customize its distillation data during the difficulty selection. through these stages, we obtain a small amount of high-quality rationale data to improve distillation performance for specific student models. in summary, our contributions are three-fold: ## results in this section, we report the performance of our morsd and baseline methods on 7 benchmarks. we compare our approach with baselines of different model sizes. the performance on the test set demonstrates the effectiveness of our approach, showing that our method achieves better performance with fewer samples. ## conclusion in this work, we propose morsd, an efficient cot distillation method that enhances the performance of small language models using fewer rationales. by introducing a self-guided rationale difficulty metric, morsd enables the autonomous selection of high-quality rationales, effectively addressing challenges related to the rationale quality. experiments across seven datasets demonstrate an average accuracy improvement of 4.6% over the baseline. morsd outperforms full dataset distillation with a small, tailored set of rationales, providing a robust solution for efficient cot distillation and advancing knowledge transfer in a more efficient manner.