Cognitive Flow: An LLM -Automated Framework for Quantifying Reasoning Distillation (2025)

Abstract: The ability of large language models (LLMs) to reason effectively is crucial for a wide range of applications, from complex decision-making to scientific research. However, it remains unclear how well reasoning capabilities are transferred or preserved when LLMs undergo Knowledge Distillation (KD), a process that typically reduces model size while attempting to retain performance. In this study, we explore the effects of model distillation on the reasoning abilities of various reasoning language models (RLMs). We introduce Cognitive Flow, a novel framework that systematically extracts meaning and map states in Chain-of-Thought (CoT) processes, offering new insights on model reasoning and enabling quantitative comparisons across RLMs. Using this framework, we investigate the impact of KD on CoTs produced by RLMs. We target DeepSeek-R1-671B and its distilled 70B, 32B and 14B versions, as well as QwenQwQ-32B from the Qwen series. We evaluate the models on three subsets of mathematical reasoning tasks with varying complexity from the MMLU benchmark. Our findings demonstrate that while distillation can effectively replicate a similar reasoning style under specific conditions, it struggles with simpler problems, revealing a significant divergence in the observable thought process and a potential limitation in the transfer of a robust and adaptable problem-solving capability.

## introduction the rapid development and constant evolution of large language models (llms) gave rise to a class of highly-capable reasoning language models (rlms). rlms like openai's o1 @xcite and o3 1 , and deepseek-r1 (deepseek-ai et al., 2025), leverage reinforcement learning techniques to develop the ability to generate "thinking traces", or chain-of-thought (cot), in order to solve complex tasks. to make this power accessible, reasoning from these large and heavy models (teacher models) is often distilled into smaller, more efficient llms (student models), leading to the transfer of capabilities that were once exclusive to high scale proprietary systems. this has led to the appearance and proliferation of open-source distilled reasoning models that achieve remarkable performance on reasoning benchmarks. this trend has also raised concerns about model homogenization, where different models tend to exhibit convergent behaviors and responses due to reliance on common distilled data sources @xcite . however, the success achieved by distilled models in task performance remains somewhat uncertain. are we truly distilling the "cognitive" properties of larger models, or merely replicating the superficial appearance of their reasoning? the cot traces used for training reflect the behavior of teacher models, but they may not always accurately capture the underlying reasoning process @xcite @xcite . this raises a critical issue that current evaluation methods fail to address: they rely on task accuracy, which is an inadequate and potentially misleading metric for assessing the quality of the transferred reasoning. this gap is a central and unsolved issue in the field. as stated in a recent comprehensive survey on llm distillation, there is a critical need for "holistic evaluation protocols that measure nuanced llm capabilities", as current metrics "fail to evaluate whether distilled datasets preserve deeper reasoning abilities, such as chain-of-thought logic" @xcite . there is a need for tools to look deeper to assess quality, robustness and trustworthiness of the distilled thinking of these small models. looking to bridge this gap, we introduce a novel framework to extract meaning and identify transitions between the cognitive actions of a model's cot. as a case study, we apply it to quantify the fidelity of distilled reasoning. our approach moves beyond solely structural analysis and comparison of final answers, modeling the observable reasoning process as a "cognitive flow": a sequence of transitions between states of different cognitive activity. by systematically extracting these states (e.g. "problem decomposition", "calculation", "conclusion") from a model's reasoning traces and computing the probabilistic transitions between them, our technique allows for a quantitative comparison of reasoning patterns across different rlms. to demonstrate our framework, we conduct a novel comparative analysis of the deepseek-r1 family of rlms @xcite as a case study. these models are ideal for our analysis as they provide a clear teacher-student hierarchy, consisting of a 671b parameter teacher and its 70b, 32b and 14b student versions. by also including a control rlm from outside this model family, we can effectively isolate the impact of the distillation process itself. our contributions are listed as follows: ## related work the appearance of chain-of-thought (cot) marked a paradigm shift in the ability of llms to tackle complex reasoning tasks @xcite . it began with prompt engineering, but recently started being incorporated directly in the model's supervised fine-tuning (sft) and reinforcement learning (rl) stages of the post-training process @xcite , giving way to modern rlms, which treat the reasoning trace as an intrinsic part of their output. while cot has shown to improve llm performance by taking advantage of test-time scaling @xcite , one of its primary advantages lies in the promise of explainability. by expressing intermediate steps taken to solve a problem, a model reveals its internal thinking process. however, this promise of transparent reasoning is shadowed by the challenge of faithfulness. a growing body of research demonstrates that while a model's reasoning steps may be plausible and appear logical and coherent to a human observer, they are not guaranteed to be faithful to the actual internal process that produced the final answer @xcite @xcite . therefore, a model can arrive at a correct answer for incorrect reasons, meaning that "the end justifies the means", and not the other way round. in knowledge distillation (kd), knowledge is transferred from a larger, more complex "teacher" model to a smaller, more efficient "student" model. in traditional kd, the student is trained on both the logits of the teacher model and a target dataset @xcite . in the modern llm paradigm, it involves performing instruction fine-tuning of smaller llms on a synthetic sft dataset generated by larger ones. an emerging subject in this area is the transferring of reasoning and cot capabilities from teacher rlms to non-reasoning llm students. studies have shown that kd enables smaller llms to properly capture the reasoning capability and enable smaller models to achieve high performance on complex tasks, proving itself more effective than pure rl for such models @xcite . the distillation process introduces another layer of uncertainty to the cot faithfulness problem, as we are distilling a behavioral trace that may not be faithful to the teacher's underlying cognitive process. the question shifts from if the student can learn to solve the task, to how it learns how to solve it. this concern is backed by findings suggesting that the structure of cot demonstrations matter much more for successful learning than the correctness of the content itself @xcite . this implies that we can't assess the broad performance of a distilled model solely based on its accuracy. other research has focused on quantifying the broader effects of distillation @xcite , proposing a framework to measure the degree of "homogenization" across various llms by evaluating inconsistencies in "identity cognition" and similarity in final responses. given the challenges of faithfulness and the uncertainties of distillation, the need for methods that analyze and compare cot reasoning traces is crucial. while the broader field of explainable ai (xai) offers many techniques for assessing llms @xcite , for multi-step reasoning the most direct approach is to analyze the cot itself. some notable current approaches are landscape of thoughts (lot) @xcite and reasongraph @xcite . lot visualizes reasoning steps by mapping them as feature vectors in relation to answer choices. while useful for distinction between correct and incorrect paths, this approach is fundamentally tied to multiple-choice tasks and fails to capture the intrinsic semantic meaning of each reasoning step. reasongraph focuses on visualizing the structure of the reasoning path. it dynamically renders the cot as a directed graph, and is able to represent sequential and tree based logic. however, it only focuses on the structure of the cot, disregarding the semantic meaning of cognitive steps that make up the reasoning trace. furthermore, both these tools are designed for single answer analysis, making it impossible to extract the common cognitive patterns that define a model's characteristic reasoning style across many tasks of a similar type. an evident gap arises in the current methodologies, which lack ways for quantitatively comparing the semantic and structural patterns of reasoning between teacher and student models. existing tools tend to focus solely on the final answer, prioritize structural aspects excessively, or fall short in supporting a broader analysis. this work addresses that, by proposing a framework that enables visualization and quantitative comparison of a model's "cognitive flow", offering a method to evaluate the quality of reasoning distillation, extending beyond accuracy to focus on the reasoning itself. ## methodology this section is divided into two main parts. first, we introduce the framework. then, we present the experimental setup used to apply it to a family of teacher-student models, enabling a systematic comparison. ## results & discussion our analysis, conducted through the cognitive flow framework, provides a quantitative measurement of reasoning style similarity after distillation. results are summarized in to ensure increased confidence on these results, we conducted a human evaluation on 150 randomly sampled reasoning steps (appendix a). the step classifier llm achieved a cohen's kappa of 0.836, a score indicating "almost perfect agreement" with the human annotator @xcite . this confirms that the step classifier llm reliably captures the cognitive states of the observable reasoning process, validating the foundation for the upcoming analysis. we first establish a baseline by evaluating models on tasks of moderate complexity (high school). in this setting, kd proves highly effective at transferring reasoning flows between teacher and student. as detailed in however, when models were tested on tasks of either higher or lower complexity, the high fidelity seen previously degrades significantly, revealing a critical weakness in generalization. on highly complex problems (college), while the similarity in the direction of reasoning paths remains high (@xmath0.92 for all models), kld scores increase. this indicates that, while the distilled models can still follow the teacher's cognitive path, they do so with a different probabilistic confidence, revealing a divergence from the learned reasoning style under high cognitive requirements. more surprisingly, we find a significant "failure" of distillation on the simpler elementary tasks. for all distilled models, cognitive flows exhibit a large gap from the teacher, with kld scores soaring to over 1.3 and cs scores lowering to around 0.8 ( results obtained from the control model, qwenqwq-32b, provide a counterpoint and help explain the findings so far. on the elementary tasks where the distilled models heavily diverged from the teacher model, the control model actually resembled a very similar reasoning, achieving the highest cs (0.956) and the lowest kld (0.268). this result is crucial. firstly, it demonstrates that flow unalignment is not an inherent limitation posed by model size or architecture. moreover, and because our control and 32b student models were both run under identical 4-bit quantization, we can also conclude that divergence is not merely an artifact of model compression. secondly, after isolating the post-training methodology (distillation vs. reinforcement learning) as the key variable, this provides strong evidence that distillation, while effective for transferring specific skills, may be less effective at generating a robust and adaptable reasoning style than an independent learning process such as rl. our analysis of cognitive states distribution in figure 3 provides an explanation for this difference. on elementary tasks, the distilled models neglect the crucial verification step, using about 20% less tokens towards it, compared to the teacher. this suggests a form of overfitting, where distilled models learn to replicate the teacher's sophisticated problem-solving logic but fail to generalize to simpler problems where those advanced techniques are unnecessary, and fundamental steps like verification can play a huge part. furthermore, across all three levels of complexity tested, the control model consistently allocates a larger portion of its computational effort to verification (figure 3 ). findings from this experiment highlight the profound impact of the underlying training objective. distillation is a form of process supervision, implemented as sft on the teachers' outputs, meaning that the students' objective is to replicate the teacher's reasoning path precisely. this effectively forces the student to adopt the teacher's reasoning style but, as we show, can lead to fragile behavior and affect the generalization capability. on the other hand, the control model's rl training represents outcome supervision. its policy is not optimized to follow a specific, defined path, but to maximize a reward signal based on the cor-rectness of the outcome, encouraging emergent reasoning strategies. the model is free to discover that frequent self-verification is a highly-rewarding behavior because it often leads to correct answers, resulting in a robust, generalized reasoning ability developed through exploration and iteration. ## conclusion we introduced cognitive flow, a novel framework for quantifying and comparing the semantic structure of an llm's reasoning. by modeling cot as a sequence of transitions between cognitive states, we moved beyond accuracy-based metrics to assess the fidelity of reasoning distillation. our analysis of the deepseek-r1 family provided a key insight. simply learning to replicate a teacher's reasoning style through sft is a fragile strategy when trying to distill reasoning flows between models. while highly effective for a set of tasks under the right conditions, the approach fails to generalize, leading to an increase of flow unalignment on simpler problems and the neglect of fundamental cognitive actions, like self-verification. in contrast, an independently rl trained model demonstrated a more robust and adaptable reasoning style. these findings challenge the paradigm of pure kd as is, for creating smaller and efficient reasoning llms. they suggest that future work should focus on hybrid training approaches that combine the guidance of distillation with the exploratory benefits of rl to build models that can not only achieve higher performance, but also higher robustness and alignment with their teachers. the code used for our experimentation is available at https://github.com/nlp-cisuc/ cognitive-flow .