Stronger Models are Not Always Stronger Teachers for Instruction Tuning (2025)

Abstract: Instruction tuning has been widely adopted to ensure large language models (LLMs) follow user instructions and engage with users meaningfully. The resulting instruction-following capabilities of LLMs heavily rely on the instruction datasets used for tuning. Recently, synthetic instruction datasets have emerged as an economically viable solution to provide LLMs diverse and high-quality instructions. However, existing approaches typically assume that larger or stronger models are stronger teachers for instruction tuning, and hence simply adopt larger models as response generators to the synthetic instructions. In this paper, we challenge this commonly-adopted assumption. Our extensive experiments across five base models and twenty response generators reveal that larger and stronger models are not necessarily stronger teachers of smaller models. We refer to this phenomenon as the Larger Models’ Paradox. We observe that existing metrics cannot precisely predict the effectiveness of response generators since they ignore the compatibility between teachers and base models being fine-tuned. We thus develop a novel metric, named as Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response generators. Our experiments across five base models demonstrate that CAR outperforms almost all baselines.

## introduction instruction tuning (figure 1 ) has been widely adopted to tailor the behavior of base large language models (llms) to align with specific tasks and user intents @xcite . this approach leverages instruction datasets, consisting of samples pairing an instruction with a corresponding response. the success of instruction tuning depends on the availability of high-quality instruction datasets. initially, constructing these datasets required large human effort in generating and curating instruction-response pairs @xcite ; zheng @xcite , which is timeconsuming and labor-intensive @xcite . to reduce the reliance on human-curated datasets, synthetic datasets generated by llms have surfaced as a viable solution @xcite . recent works, such as @xcite @xcite @xcite , have shown the strong potential of synthetic datasets in instruction tuning. while current research has primarily focused on using llms to create large, diverse, and highquality instructions @xcite , the selection of appropriate llms for generating corresponding responses remains largely unexplored. the common approach relies on distilling from state-ofthe-art models that excel in benchmark evaluations @xcite to generate responses for instruction tuning. for instance, llama-3.2-3b-instruct uses responses generated by llama-3.1-405b-instruct (i.e., the largest model in llama-3.1 family) for instruction tuning @xcite . additionally, most of the existing open synthetic datasets @xcite @xcite @xcite depend on expensive, closed-source models like gpt-4 @xcite and gemini @xcite to produce responses. is it always better to use the larger or stronger models as teachers? in this paper, we investigate the choice of the teacher model that gener-ate responses during synthetic dataset generation, which we refer to as response generators, influence the instruction-following performance of the instruction-tuned llms. specifically, given a base model and a set of high-quality instructions, we investigate the following research questions: rq1: which models are the most effective response generators for instruction tuning? to answer rq1, we conduct extensive experiments with five base models, and fine-tune them on datasets generated by 20 response generators across seven model families: qwen2, qwen2.5, llama 3, llama 3.1, gemma 2, phi-3, and gpt-4. our findings challenge common assumptions in the field, revealing a surprising result which we term the larger models' paradox: larger response generators (e.g., llama-3.1-405b-instruct) do not always enhance a base model's instruction-following capabilities compared to their smaller counterparts within the same model family (e.g. llama-3.1-70b-instruct). moreover, we find that open-source models (e.g., gemma-2-9b-it and qwen2.5-72b-instruct) outperform gpt-4 as response generators. these findings question established practices and suggest more efficient and accessible approaches to create high-quality instruction datasets. to further explore the larger models' paradox, we investigate statistical metrics to reveal potential factors influencing the effectiveness of response generators. here, we pose our second research question: rq2: how can we determine the most effective response generators for a certain base model without instruction tuning? this question is crucial due to the significant computational costs associated with instruction tuning across multiple datasets generated by diverse response generators. our investigation reveals that existing metrics in alignment data selection, including quality @xcite , difficulty @xcite , and response length @xcite , fail to consider the compatibility between the base model being fine-tuned and the response generator, thus results in their inability to explain the larger models' paradox. to bridge this gap, we formulate the task of finding the most effective response generators as a risk-return problem. we solve this by calculating an compatibility-adjusted reward (car), where compatibility serves as the risk factor. this compatibility is quantified by the average loss of responses on the base model being finetuned, with higher average loss indicating lower compatibility and thus higher risk. our comparison of the proposed car with existing metrics demonstrates that it outperforms all baselines in predicting the effectiveness of response generators. we believe that our findings on the larger models' paradox and the proposed car can effectively guide future instruction tuning of llms. instead of selecting response generators solely based on benchmark performance (e.g., gpt-4), practitioners should prioritize those with higher compatibility to better enhance the instruction-following capabilities of their llms. ## conclusion and future work this paper investigates the impact of response generators in synthetic dataset generation for instruc-4399 tion tuning. we uncovered the larger models' paradox, wherein larger response generators do not necessarily enhance a base model's instructionfollowing capabilities compared to their smaller counterparts within the same model family. to explain this phenomenon, we considered the compatibility between response generators and the base model, and proposed the compatibility-adjusted reward (car). our metric achieved better performance in identifying the effectiveness of different response generators without the need for fine-tuning, outperforming existing baselines in alignment dataset selection. we will explore several promising directions. first, efficiently transforming existing datasets to achieve better compatibility can lead to more effective use of available instruction tuning datasets. second, investigating theoretical foundations of compatibility would enhance our understanding of the underlying mechanisms of instruction tuning. lastly, studying the impact of different response generators for preference tuning may help aligning llms to better reflect human values.