Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting (2025)

Abstract: Hand-crafting high quality prompts to optimize the performance of language models is a complicated and labor-intensive process. Furthermore, when migrating to newer, smaller, or weaker models (possibly due to latency or cost gains), prompts need to be updated to re-optimize the task performance. We propose Concept Distillation (CD), an automatic prompt optimization technique for enhancing weaker models on complex tasks. CD involves: (1) collecting mistakes made by weak models with a base prompt (initialization), (2) using a strong model to generate reasons for these mistakes and create rules/concepts for weak models (induction), and (3) filtering these rules based on validation set performance and integrating them into the base prompt (deduction/verification). We evaluated CD on NL2Code and mathematical reasoning tasks, observing significant performance boosts for small and weaker language models. Notably, Mistral-7B’s accuracy on Multi-Arith increased by 20%, and Phi-3-mini-3.8B’s accuracy on HumanEval rose by 34%. Compared to other automated methods, CD offers an effective, cost-efficient strategy for improving weak models’ performance on complex tasks and enables seamless workload migration across different language models without compromising performance.

## introduction large language models (llms) have shown remarkable capabilities for various downstream tasks. an inexpensive alternative to training and finetuning, prompt engineering has emerged as a powerful method to control and optimize the outputs from llms. prompt engineering is enabled by the in-context learning (icl) capability of llms @xcite , which allows us to apply llms to new tasks by providing them with a suitable input prompt that contains relevant information and instructions @xcite . as such, crafting high-quality prompts can be a challenging and labor-intensive process. finding the right instructions can require several rounds of trial-and-error experimentation. further, the same prompt may not work for different tasks, models, or domains @xcite . importantly, weak models such as gpt-3.5 or mistral-7b often lack the same reasoning capabilities as strong models such as gpt-4o, and as a result, struggle with complex and high-reasoning tasks @xcite . this leads to significant performance gaps between stronger and weaker models for such tasks. conversely, practical reasons (e.g., lower runtime latency, cost, and memory footprint) may still motivate and impose the use of weak models in practical applications @xcite . while fine-tuning methods such as lora @xcite may close this gap, they involve modifying the model's parameters -thus making it task-specific and limiting its reuse across different contexts. in particular, these approaches require fine tuning infrastructure and know-how, which may not be available or accessible in many practical scenarios. in contrast, our cd approach preserves the model's parameters, allowing the model to remain flexible for various tasks, and requiring only prompt-engineering level of access. another key area that the current work addresses is the efficient adaptation of prompts for various models. a primary challenge is transitioning prompts from an existing model, such as gpt-4, to a newly released variant like gpt-4o. it is essential to recognize that different models may respond uniquely to the same prompts. as such, there is the need for strategies that effectively modify and tailor existing prompts to maintain alignment with new or evolving models. in this paper, we introduce concept distillation (cd), an automated prompt optimization technique. cd improves the performance of weak/small language models on complex tasks by distilling key rules, concepts, or examples from a strong/large model via hypotheses-to-theories prompting. these distilled concepts are then verified and used to guide a weak model, enabling it to produce more accurate responses, all without the need for fine-tuning. the structured approach within the cd framework ensures that these distilled concepts are sufficiently general to be transferable across various other language models. figure 1 shows a high-level illustration of the concept distillation for prompt optimization approach. overall, this paper makes the following contributions: ## related work given the significance and broad-scale effectiveness of prompt engineering, there have been several efforts to perform automated prompt optimization and generation. these methods typically involve an iterative algorithm consisting of several steps -an initially generated prompt, scoring of the prompt, and regeneration of the prompt using the score as an improvement signal, till a stopping criteria is met @xcite @xcite @xcite @xcite . we propose an approach that augments this framework for prompt optimization through the distillation of concepts, and introduces an explicit verification step to demonstrate relative performance improvements for a small model. our method is inspired by several recent works. ape @xcite deduces an initial prompt from training samples, and then uses an llm to refine and generate new semantically similar prompt candidates. however, prompts are simply paraphrased during the refinement process, which is akin to random search in the prompt space. evoke @xcite uses the same llm to review and score the quality of a prompt, as well as to refine the prompt based on the reviewer feedback. @xcite first uses an llm to induce a rule library from a set of training examples, which are later sampled for dynamic prompt construction. this is followed by a deduction phase where these rules are evaluated based on their coverage and confidence. @xcite generates high and low-level concepts from mistakes using an llm, and later uses the same llm for solving tasks. there is no deduction phase to filter out the generated concepts. pe2 @xcite explores meta-prompt variants to guide llms to perform automatic prompt engineering. they introduce 3 meta-prompt components -two-step task description, context specification and step-by-step reasoning template to improve task performance. in contrast to all these works, our method focuses on transferring capability from a large/strong model to a small/weak one by inducing concepts mainly from the mistakes made on a task by the weak model. additionally, our deduction phase filters out the generated concepts in a metric-driven way, which is a crucial additional step in our framework that improves task adaptability and performance of weak models. many other works explore various fundamentally different frameworks for automatic prompt optimization, and are noteworthy to mention here. there are text-based error-propagation techniques such as promptagent @xcite which uses monte carlo tree search, and protegi @xcite which mirrors the steps of gradient descent-like updates for prompts. tran @xcite takes a different approach by accumulating failure-driven rules at inference time, enabling llms to iteratively improve without finetuning. another category of related works employs parametric (non-interpretable) prompt optimization techniques, as opposed to edit-based ones @xcite @xcite . ## background in this section, we explore the foundational concepts and terminologies central to this paper. this technique draws inspiration from human cognitive processes @xcite , particularly in how we acquire, refine, and apply knowledge and concepts across various domains. concept distillation: distinction from knowledge distillation depicted in fig. 2 . the core of our technique is encapsulated in the process of 'concept distillation'. this process involves the transfer of concepts from a stronger lm (referred to as the 'teacher') to a weaker lm (referred to as the 'student'). the differentiation between knowledge and concept distillation is critical. unlike traditional knowledge distillation @xcite , which focuses on the explicit transfer/update of learned weights and biases through intensive training or fine-tuning procedures, concept distillation emphasizes the induction of general concepts, rules, examples, or key ideas from the teacher model, applying them to the student model solely via in-context learning (icl), without necessitating extensive training or fine-tuning. figure 3 depicts the distinction between knowledge and concept distillation. hypotheses, theories, and reasoning: frameworks for conceptual transfer. our approach is deeply rooted in the scientific methodologies of hypothesis generation, experimental validation, and theory @xcite . a hypothesis, in this context, is a proposition based on limited evidence, serving as a foundation for further investigation that could culminate in a theory, i.e., a well-substantiated explanation of a phenomenon. this framework is critical in concept distillation, where hypotheses derived from observations are validated through experimental evidence to form theories that explain the underlying principles or phenomena. the transformation from hypotheses to theories is facilitated by two modes of reasoning: inductive and deductive reasoning. inductive reasoning involves deriving general rules from specific observed facts, whereas deductive reasoning entails deriving new facts from established facts and rules. deductive reasoning allows us to apply general principles to specific cases to derive accurate conclusions. these modes of reasoning allow the extrapolation of concepts from inductive reasoning and the application of these concepts to new, unseen instances. drawing parallels to the human process of scientific discovery @xcite , our technique mirrors the iterative cycle of observation, hypothesis formulation, experimentation, and theory development. this analogy highlights the integration of inductive and deductive reasoning in forming robust concepts that not only explain observed phenomena but also predict outcomes in unseen scenarios. ## concept distillation framework our technique consists of three main phases: initialization, induction, and deduction from verification. initialization phase. phase 1 starts with a base prompt template, which can be either an existing prompt we aim to modify (for a strong, large model we aim to replace), a generated prompt using an off-the-shelf algorithm, or one manually crafted by domain experts, serving as a foundation for subsequent refinement. in this phase, we assess the strengths and weaknesses (mistakes) of the weaker model regarding the intended task. the primary goal here is to pinpoint areas and examples where the weaker model struggles, enabling us to induce concepts that aid in reasoning in these specific areas. it is important to focus on the model's weaknesses, avoiding unnecessary adjustments in areas where the model already performs well. induction phase. phase 2 involves the induction of concepts from a strong model, such as gpt-4, tailored to address the identified weaknesses and mistakes of the weaker model. the aim is to enhance the weaker model's performance by equipping it with these newly induced concepts. during this process, we use the strong model to reason through the facts or questions presented to the weak model, the incorrect responses it generated, and the correct answers, in order to generate general concepts that can overcome the mistakes of the weak model. deduction from verification phase. phase 3 is the deduction-from-verification process. the assumption is that not all induced rules/concepts or examples qualify as useful distilled concepts. this phase uses a deduction process to verify the induced rules and examples. rules that qualify as having broad coverage and high prediction confidence are accepted as distilled concepts. consequently, they are added to the initial prompt template that we started with to form an improved, updated prompt. after adding the induced concepts to the base prompt template, a verification process is applied to filter the concepts. either the strong model can be used to generate test examples similar to the weaknesses identified earlier or similar examples from a validation set can be used for the verification. the weaker model is required to accurately address all validation examples with a level of certainty or probability that meets or exceeds a specific predefined threshold before we accept the induced concepts as distilled concepts and integrate them into its prompt. this ensures that the final prompt effectively addresses the weak model's shortcomings, leading to improved performance. algorithm 1 succinctly captures the proposed cd framework. it details the three key phases-initialization (see fig. 5 ), induction (see fig. 6 ), and deduction/verification (see fig. 7 ). the definitions and descriptions of the notations and processes as well as the prompts used in the algorithm are provided in appendix a. a detailed description of the concept distillation process with a walk-through example is provided in appendix c. we focus on three benchmark datasets: nl2code (humaneval) @xcite , multi-arith @xcite , and gsm8k @xcite . humaneval involves generating code from natural language prompts, while multi-arith and gsm8k evaluate arithmetic and mathematical reasoning, requiring step-by-step solutions. we compare our approach with methods such as automatic prompt engineering (ape) @xcite , iterative ape @xcite , chain of thought (cot) @xcite , prompt engineering a prompt engineer (pe2) @xcite , and automatic prompt optimization (apo) @xcite . we evaluate cd using gpt-3.5 turbo, claude 2.1, phi-3-mini-3.8b, mixtral-8x7b*, and mistral-7b, with gpt-4o for concept distillation. our experiments focus on improving weak models' performance through cd and testing the transferability of optimized prompts across models. we split each dataset into training and test sets for prompt optimization and evaluation, comparing our method with state-of-the-art techniques. ## results and analyses in firstly, we observe a performance increase of 11% for the gpt-3. in for the gpt-3.5 turbo model, a significantly larger gain for the claude 2.1 model with a 29% increase in accuracy from 0.62 to 0.91, and a similarly large 20% accuracy gain for the mistral-7b model. on average, weak models observed a performance lift of 15% on the multi-arith mathematical reasoning task. the results in the results also highlight significant improvements for mixtral-8x7b* and mistral-7b, where cd boosts their accuracies to 0.95 and 0.96, respectively, compared to lower accuracies with ape (0.73 and 0.71) and cot (0.88 and 0.87). notably, phi-3-mini-3.8b's accuracy slightly decreases with cd compared to cot due to its initial weaknesses during training, which resulted in a lower baseline accuracy of 38% on the training set. as a result, the extensive concept distillation required to address these weaknesses introduced slight confusion in some edge cases. despite this, phi-3-mini-3.8b still maintains competitive performance. we also evaluated the transferability of optimized prompts from gpt-3.5 turbo to other models like claude 2.1, phi-3-mini-3.8b, mixtral-8x7b*, and mistral-7b. results show significant performance gains, with smaller models like phi-3-mini-3.8b improving by 34% and claude 2.1 achieving 100% accuracy. detailed results and further analysis are provided in the appendix b.2. these findings highlight the generalizability of the distilled concepts across models. finally, in appendix b.3, we provide a qualitative analysis of the prompt changes generated for the humaneval benchmark. this analysis demonstrates how cd extracts generalizable concepts to improve reasoning and adaptability in weak models, achieving substantial performance gains while addressing the limitations of rigid few-shot demonstrations. ## conclusion in conclusion, our study demonstrates the robustness of concept distillation in significantly enhancing the performance of weaker language models across various tasks, as evidenced by substantial accuracy improvements on the humaneval, multi-arith, and gsm8k datasets. by distilling and transferring essential concepts from stronger models, cd not only boosts the capabilities of smaller models but also ensures the transferability of these improvements across different models. our extensive experiments show that cd consistently outperforms various state-of-the-art prompt optimization methods. this robust framework, therefore, addresses critical challenges in prompt engineering, offering a scalable and resource-efficient solution that advances the state-of-the-art in prompt optimization for language models.