Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition (2024)

Abstract: Knowledge distillation allows smaller neural networks to emulate the performance of larger, teacher models with reduced computational demands. Traditional methods for Large Language Models (LLMs) often necessitate extensive fine-tuning, which limits their accessibility. To address this, we introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from large-scale teacher models (over 8 billion parameters) to small-scale student models (up to 8 billion parameters). This approach leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions. Empirical evaluations on the GSM8K and MATH datasets show that student models achieve accuracy gains of up to 113% on GSM8K and 20% on MATH, with significant improvements particularly notable in smaller models like Llama 2 and Zephyr. Our results suggest a promising pathway for open-source, small-scale models to eventually serve as both students and teachers, potentially reducing our reliance on large-scale, proprietary models. Our code, featuring data analytics and testing scripts, is provided here: https://github.com/traceofthought/trace-of-thought-prompting/tree/main.

## introduction knowledge distillation, as initially proposed by @xcite , involves leveraging the outputs of larger neural networks as soft targets to train smaller, more efficient networks. this method, primarily applied to tasks like mnist @xcite in computer vision, uses computationally heavy teacher models to facilitate equivalent reasoning capacities in smaller models, substantially reducing computational demands on the user. as the popularity of large language models (llms) has surged, adaptations of this technique have been explored, particularly through fine-tuning based on the outputs of these large models. however, these adaptations often introduce a significant computational overhead and necessitate a deep understanding of machine learning, limiting their accessibility for average consumers @xcite @xcite . concurrently, the rapid development of llms has been paralleled by innovations in prompt engineering-the strategic design of prompts to enhance reasoning and explore various problemsolving pathways @xcite . methods such as chain-of-thought prompting and self-consistency have demon-strated the potential of llms to engage in complex reasoning and provide novel solutions to challenging problems @xcite @xcite . nevertheless, these approaches typically operate within a single contextual framework and rely heavily on the innate reasoning capabilities of models, often failing when applied to smaller, open-source variants @xcite @xcite . this suggests a critical need for a more adaptable and scalable approach to knowledge distillation that can leverage the advances in prompt engineering for broader accessibility and effectiveness. in response to this need, we explore the intersection of prompt engineering and knowledge distillation through a novel concept we term prompt-based knowledge distillation. this approach utilizes incontext learning (icl) to emulate traditional distillation processes within the accessible framework of llm prompting, mirroring the cognitive process of a student learning from a teacher @xcite . to implement this concept, we introduce trace-of-thought prompting, a technique that decomposes complex arithmetic reasoning problems into manageable steps, facilitating the distillation of critical reasoning skills from large-scale models to their small-scale counterparts (see figure 1 ). this strategy not only improves the performance of small-scale models but also demonstrates their potential to serve as effective teachers themselves. our contributions to this novel extension of knowledge distillation are threefold: tandem and socratic reasoning. the exploration of collaborative problem-solving in model suites, such as socratic chain-of-thought and socratic questioning, introduces novel ways to utilize multiple models in a cohesive manner @xcite @xcite @xcite . however, these approaches encounter difficulties with managing large context sizes and reliance on fine-tuning @xcite . our work contributes to this area by implementing a structured approach that minimizes token bloat and finetuning dependency, offering a more efficient and scalable solution for collaborative reasoning within llm environments. ## prompt-based knowledge distillation traditional knowledge distillation, as originally proposed by @xcite , involves finetuning smaller neural networks on the soft outputs (logits) of larger, teacher networks. this transfer learning method enhances the smaller model's performance to emulate its larger counterpart, albeit with significantly reduced computational overhead. despite its effectiveness, traditional knowledge distillation is resource-intensive, necessitating extensive computational efforts and substantial data, which limits its accessibility for average users. in contrast, we introduce prompt-based knowledge distillation. this novel approach leverages incontext learning (icl) to facilitate knowledge transfer without the extensive fine-tuning traditionally required. it conditions a small-scale student model on carefully crafted prompts derived from the largescale teacher model, significantly reducing computational demands and enabling rapid adaptation to new tasks. consider the general framework for promptbased knowledge distillation, where a teacher model t and a student model s interact. the teacher model processes an input question q to generate an informative prompt p , which encapsulates key insights or directions rather than explicit answers: the student model s then uses this prompt to infer the answer a, leveraging the distilled knowledge without direct output replication: consider an educational scenario where a student model is required to solve geometry problems involving circle areas. for the problem "calculate the area of a circle with a diameter of 10 cm," a large-scale teacher model could generate a prompt that distills essential concepts into several key points: this structured prompt guides the student model to focus on the fundamental mathematical relationships and proper problem-solving practices. by applying these principles, the student model calculates the radius as 5 cm and then uses the formula to determine the area as 25π square cm. this approach not only aids in solving the current problem but also reinforces good mathematical practices for future tasks. ## trace-of-thought prompting many problems in domains such as arithmetic reasoning can be broken down into intermediate steps that mimic the cognitive process of a human evaluator. trace-of-thought prompting, an application of the prompt-based knowledge distillation framework introduced earlier, enhances models' problemsolving capabilities by breaking down such problems into simpler, actionable steps. ## results ## conclusion this paper introduces a structured approach to prompt-based knowledge distillation, building on traditional methods to enhance accessibility and practicality for end-users. our methodology, traceof-thought prompting, serves as a practical implementation of this framework, designed to facilitate problem decomposition and improve problemsolving capabilities in both large-scale and smallscale models. through our experiments with various teacher model sizes, we have demonstrated how trace-of-thought can effectively leverage the knowledge distilled from both large and small models, improving reasoning capabilities in a variety of contexts. our results show significant gains in model performance, especially in scenarios involving small-scale models, highlighting the potential of this approach to make ai more accessible and effective for a broader range of applications.