Small Models Struggle to Learn from Strong Reasoners (2025)

Abstract: Large language models (LLMs) excel in complex reasoning tasks, and distilling their reasoning capabilities into smaller models has shown promise. However, we uncover an interesting phenomenon, which we term the Small Model Learnability Gap: small models (3B parameters) do not consistently benefit from long chain-of-thought (CoT) reasoning or distillation from larger models. Instead, they perform better when fine-tuned on shorter, simpler reasoning chains that better align with their intrinsic learning capacity. To address this, we propose Mix Distillation, a simple yet effective strategy that balances reasoning complexity by combining long and short CoT examples or reasoning from both larger and smaller models. Our experiments demonstrate that Mix Distillation significantly improves small model reasoning performance compared to training on either data alone. These findings highlight the limitations of direct strong model distillation and underscore the importance of adapting reasoning complexity for effective reasoning capability transfer.

## small model learnability gap in this section, we fine-tune student models using different cot data. we then reveal the small model learnability gap given the performance of fine-tuned models. ## mix distillation: bridge small model learnability gap this section presents our mix distillation approach to bridge the small model learnability gap. ## conclusion and future work in this paper, we showed that long cot data and large model responses were not uniformly beneficial for small student models. we found that small models may perform better when fine-tuned with short cot and small model cot. we termed this challenge as the small model learnability gap. the reason behind it may be that small student models excel on data that closely match their inherent distribution but struggle with significant distribution shifts. to bridge the gap, we introduced mix distillation, including mix-long, which combined long cot and short cot data in a ratio, and mix-large, which integrated large and small teacher cot. experimental results showed that both mix-long and mix-large outperform baselines across most evaluation metrics, which implied mix distillation outperforms training on a single data distribution. this paper provided practical insights for optimizing post-training strategies to enhance small language model reasoning capability. we will explore several promising directions as future work. first, we will refine mix distillation by optimally combining diverse data sources and proposing more fine-grained mixing algorithms to boost reasoning capabilities. second, we propose to study how strong reasoning teachers can generate data that is better suited for tuning small student models, thereby facilitating more effective knowledge transfer. third, we will conduct further theoretical and model interpolability studies on the small model learnability gap. lastly, we will investigate which sft methods yield the best initial policies for subsequent rl procedure.