T iny T hinker: Distilling Reasoning through Coarse-to-Fine Knowledge Internalization with Self-Reflection (2025)

Abstract: Large Language Models exhibit impressive reasoning capabilities across diverse tasks, motivating efforts to distill these capabilities into smaller models through generated reasoning data. However, direct training on such synthesized reasoning data may lead to superficial imitation of reasoning process, rather than fostering a genuine integration of reasoning capabilities with underlying knowledge. To address this, we propose TinyThinker, a framework introducing two novel approaches. First, we introduce a three-stage process that incrementally guides the student model through the reasoning process, progressively refining knowledge from coarse to fine granularity. Second, we develop a two-phase training framework comprising an initial reasoning acquisition phase followed by a self-reflection phase utilizing self-generated data. Experiments on commonsense reasoning benchmarks demonstrate that TinyThinker achieves superior performance compared to baselines. Ablation studies further validate the effectiveness of each component in our framework. We expect that TinyThinker can be extended to other knowledge-intensive reasoning tasks, offering an alternative strategy for developing effective reasoning capabilities in smaller language models. Codes are available at https://github.com/shengminp/TinyThinker.

## introduction large language models (llms) demonstrate impressive reasoning capabilities across a wide range of tasks @xcite , largely facilitated by in-context learning @xcite and chain-of-thought (cot) prompting @xcite . these methods enable llms to generalize from examples of intermediate reasoning steps during inference, enabling them to effectively address complex multi-step reasoning † corresponding author. tasks, particularly in commonsense and arithmetic reasoning domains @xcite . a notable limitation of cot prompting is its reliance on models with over 100 billion parameters, making it computationally expensive and less feasible for real-world deployment @xcite . to address these limitations, prior works @xcite @xcite have explored leveraging llms to generate reasoning data for knowledge distillation, thereby strengthening the reasoning capabilities of smaller language models (lms) foot_0 . building on these advances, recent studies generally follow two main steps: (1) applying cot prompting or its variants @xcite to generate reasoning data from a teacher model, and (2) designing specialized fine-tuning strategies to transfer reasoning ca-pabilities to a student model foot_1 @xcite @xcite . although these methods have yielded promising results, it remains uncertain whether the student model genuinely develops reasoning capabilities. a recent study attributes the success of cot prompting to its decomposition of compositional functions during in-context learning, enabling the model to focus on relevant data at each step and learn single-step composition functions in context @xcite . this interpretation aligns with the perspective that llms perform multi-step reasoning internally @xcite . however, when the student model lacks such intrinsic capabilities, direct training on synthesized reasoning data raises a critical concern: the student model may imitate the style of the synthesized data rather than its factuality @xcite . given that reasoning typically involves multiple inference steps-each requiring explicit or implicit application of knowledge @xcite -there is a risk that the student model might mimic these step-bystep reasoning processes superficially, without truly internalizing the underlying knowledge. to enable flexible incorporation of internal knowledge for effective reasoning, we propose tinythinker, a novel framework that structures the reasoning process into three stages, progressively refining knowledge from coarse to fine granularity (figure 1 ). to facilitate the acquisition and refinement of this structured reasoning process, we introduce a two-phase training framework, consisting of a reasoning acquisition phase followed by a self-reflection phase. this study focuses on commonsense reasoning, which inherently requires the integration of extensive knowledge with effective reasoning capabilities. given that most commonsense reasoning datasets are structured in a multiple-choice question (mcq) format @xcite @xcite , we designed a threestage process inspired by human problem-solving strategies. in the reasoning acquisition phase, the model first recalls general knowledge relevant to the question and options. it then conducts an analysis of specific knowledge for each option, using the recalled general knowledge as context. finally, the model integrates the acquired knowledge to summarize and identify the correct answer. after acquiring new reasoning capabilities, independent reflection is essential for consolidating these capabilities. in the self-reflection phase, the student revisits previous training data and generates new reasoning data through the same threestage process. these self-generated data, combined with iterative direct preference optimization (dpo) @xcite , further refine the reasoning capabilities, which fosters a deeper internalization of the underlying knowledge. experimental results on three commonsense reasoning datasets (commonsenseqa, openbookqa, and strategyqa) demonstrate that tinythinker significantly enhances the reasoning capabilities of the student model. furthermore, the scalability of our approach is validated by consistent performance improvements as the model size increases. additionally, ablation studies on both the three-stage reasoning process and the self-reflection phase further underscore the effectiveness of the coarse-tofine reasoning approach, along with the model's iterative self-reflection capabilities through the generation of internal knowledge. ## related work knowledge distillation transfers knowledge from a larger teacher model to a smaller student model @xcite , enabling the student to leverage the features learned by the teacher @xcite . as llms advance in generating highquality data, this process has been extended to transfer reasoning capabilities via teacher-generated data. recent research in this area can be categorized into five main approaches: supervised finetuning, decomposer-solver framework, feedbackbased framework, retriever-augmented framework, and self-improvement framework. ## methodology to enable the student to internalize the underlying knowledge for effective reasoning, we propose a two-phase training approach (figure 2 ): reasoning acquisition ( §3.1) and self-reflection ( §3.2). during the reasoning acquisition phase, the student model progresses through three stages-recall, analyze, and summarize-which progressively refine its reasoning from general to specific knowledge. in the subsequent self-reflection phase, the student generates multiple reasoning processes based on its learned capabilities to further refine the recall and analyze stages with dpo. ## conclusion in this study, we introduced tinythinker to enhance reasoning capabilities through effective knowledge internalization. in contrast to prior methods, we developed a structured three-stage reasoning process that progressively refines knowledge from coarse to fine granularity. this process is complemented by a two-phase training approach, consisting of reasoning acquisition and self-refinement phases. experiments on commonsense reasoning benchmarks demonstrate that tiny-thinker achieves superior performance compared to existing baselines. additionally, the ablation study further confirms the contributions of each component, highlighting the overall effectiveness of our approach. we expect that tinythinker provides a flexible framework extendable to other knowledge-intensive reasoning tasks, offering a promising strategy for developing effective reasoning capabilities in smaller lms.