C o TD - PO : Chain-of-Thought Distillation with Preference Optimization (2025)

Abstract: Chain-of-Thought (CoT) distillation has emerged as a promising paradigm to enhance the reasoning ability of small language models by imitating the reasoning and outputs of larger teacher models. However, existing approaches suffer from a critical limitation: a distribution mismatch between teacher-generated training trajectories and the student model’s own generative distribution. This mismatch leads to exposure bias during inference and often induces mode collapse or mode averaging, thereby degrading the student model’s generative diversity and robustness. To address these issues, we propose CoTD-PO (Chain-of-Thought Distillation with Preference Optimization), a reinforcement learning framework that shifts the training paradigm from passive imitation to active trajectory exploration. Instead of forcing the student to imitate exact teacher traces, our method enables the student to sample its own answer paths. To support training with non-open-source teacher models, we approximate the teacher’s output distribution through preference-based scoring. Furthermore, we adopt an offline iterative training procedure that enables stable and efficient optimization. Experiments on diverse open-ended generation tasks demonstrate that CoTD-PO significantly outperforms standard CoT distillation baselines, achieving higher output quality while mitigating mode collapse and preserving semantic diversity.

## introduction recently, the capabilities of large language models (llms) have been shown to scale not only with an increase in model size but also through techniques such as chain-of-thought (cot) reasoning @xcite and model distillation @xcite . cot reasoning has gained significant attention for its ability to enhance the reasoning ability of llms by encouraging them to explicitly articulate intermediate reasoning steps before arriving at a final answer @xcite . however, empirical evidence demonstrates a strong scale-dependence in cot effectiveness, with significant performance gains confined to models exceeding 100 billion parameters. smaller models often fail to perform well with cot, as their limited parameter space hinders their ability to generate intermediate reasoning steps effectively. this limitation has motivated research into chainof-thought distillation (cotd), which seeks to transfer multi-step reasoning processes and outputs from large teacher models to compact student architectures @xcite . unlike traditional knowledge distillation (kd) that primarily aligns final answers @xcite , cotd requires the student to imitate both the intermediate rationales and final answers of the teacher. this strict trajectory-level imitation introduces : while recent advances have sought to avoid exposure bias and distribution mismatch by focusing on structured tasks (e.g., closed-form arithmetic problems with deterministic solutions and multiplechoice qa @xcite ), where the output space usually consists of a finite number of classes. however, these approaches often struggle in open-ended generation settings @xcite , which are characterized by high output diversity and semantic ambiguity. meanwhile, they typically assume access to full teacher distributions-an impractical requirement for black-box models, struggling to generalize to open-ended settings with high diversity and ambiguity @xcite . to address these challenges, we introduce chainof-thought distillation with preference optimization (cotd-po). our method shifts the paradigm from strict teacher-forcing imitation to student-led exploration. specifically, we leverage reinforcement learning to empower the student model to discover its own answer trajectories, using the teacher's rationales as high-level guidance rather than a fixed path to be mimicked. this selfexploration directly mitigates exposure bias by aligning the training process with the autoregressive nature of inference. furthermore, to enable distillation from black-box teachers, cotd-po incorporates a preference alignment strategy that replaces explicit probability matching with rewardbased relative quality scoring. this allows effective supervision without requiring access to the full teacher distribution. we evaluate our approach on multiple instruction-following benchmarks, and experimental results demonstrate that cotd-po not only achieves superior performance but also effectively mitigates the aforementioned issues of exposure bias and distribution mismatch. to summarize, our contributions are as follows: ## conclusion we present cotd-po, a novel rl framework that transforms chain-of-thought distillation from passive imitation to active exploration. by aligning student-teacher preferences instead of probabilities and enforcing entropy-regularized policy updates, our method effectively mitigates distribution mismatch while preserving generation diversity. experiments across instruction-following benchmarks demonstrate state-of-the-art performance without mode collapse degradation.