SC oder: Progressive Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLM s (2025)

Abstract: Existing code large language models (LLMs) often rely on large-scale instruction data distilled from proprietary LLMs for fine-tuning, which typically incurs high costs. In this paper, we explore the potential of small-scale open-source LLMs (e.g., 7B) as synthesizers for high-quality code instruction data construction. We first observe that the data synthesis capability of small-scale LLMs can be enhanced by training on a few superior data synthesis samples from proprietary LLMs. Building on this, we propose a novel iterative self-distillation approach to bootstrap small-scale LLMs, transforming them into powerful synthesizers that reduce reliance on proprietary LLMs and minimize costs. Concretely, in each iteration, to obtain diverse and high-quality self-distilled data, we design multi-checkpoint sampling and multi-aspect scoring strategies for initial data selection. Furthermore, to identify the most influential samples, we introduce a gradient-based influence estimation method for final data filtering. Based on the code instruction datasets from the small-scale synthesizers, we develop SCoder, a family of code generation models fine-tuned from DeepSeek-Coder. SCoder models achieve state-of-the-art code generation capabilities, demonstrating the effectiveness of our method.

## introduction code generation has long been a central challenge in computer science and has attracted wide attention from the research community. recent advancements in code large language models (llms) @xcite @xcite @xcite have led to significant breakthroughs. these models can generate code that closely aligns with user intent and are increasingly being widely adopted. typically, instruction tuning on base models (e.g., deepseek-coder-base) is a crucial step in developing high-performance code llms. therefore, extensive research on code llms focuses on constructing high-quality instruction data. a common approach involves distilling knowledge from proprietary llms. for instance, code alpaca @xcite and wizardcoder @xcite are fine-tuned with instruction data distilled from gpt-3.5, using self-instruct @xcite and evol-instruct @xcite , respectively. additionally, magicoders @xcite is fine-tuned on data distilled from both gpt-3.5 and gpt-4, using oss-instruct to generate coding problems and solutions based on the given code snippets. while these methods have proven effective, they all suffer from the cost-intensive issue caused by the distillation of large-scale instruction data from the proprietary llms like gpt-3.5 and gpt-4. in this paper, we explore the potential of relatively small-scale (7b, 8b, and 14b) open-source llms as synthesizers for code instruction data construction. previous works have shown that small llms can assist in pre-training data synthesis for non-code domains @xcite . however, instruction data typically takes a different form from pre-training data and requires higher quality standards @xcite . to validate the feasibility of small llms in synthesizing code instruction data, we conduct a preliminary experiment. first, we use small-scale llms as original synthesizers and further train them on a limited set of proprietary llm-distilled samples as enhanced synthesizers. then, we fine-tune code generation models using data provided by them. the results on the left of figure 1 show that the instruction data provided by the enhanced synthesizer outperforms that of the original, highlighting that a few superior samples can unleash the data synthesis potential of small models. however, distilling more proprietary samples to further improve the synthesis capability of small synthesizers would again trigger the costintensive issue. therefore, a crucial question arises: can we continuously improve the data synthesis capability of small-scale synthesizers without relying on proprietary llms' samples? to address this, we propose a progressive selfdistillation method that iteratively bootstraps the code instruction data synthesis capability of smallscale llms. specifically, starting with an enhanced synthesizer, we employ a two-step approach in each iteration to obtain high-quality self-distilled data synthesis samples for further training. first, we design multi-checkpoint sampling and multiaspect scoring strategies to obtain diverse and reliable self-distilled samples. then, we introduce a gradient-based influence estimation method to further select the influential ones by comparing the gradients induced by self-distilled samples with those induced by superior samples from proprietary llms. we validate our method on small-scale llms like qwen2.5-coder-7b/14b-ins @xcite , improving their data synthesis capabilities as shown in the left of figure 1 , and transforming them into powerful data synthesizers. based on the code instruction datasets provided by our small-scale synthesizers, we develop scoder, a family of code generation models finetuned from deepseek-coder-6.7b-base @xcite . experimental results on humaneval (+) @xcite , mbpp (+) @xcite , livecodebench @xcite show that scoder outperforms or matches state-of-theart code llms that use the instruction data from proprietary llms. overall, our contributions can be summarized as follows: ## conclusion in this paper, we propose an iterative selfdistillation bootstrap method to fully unlock the data synthesis potential of small-scale llms, transforming them into powerful code instruction data synthesizers while reducing reliance on proprietary llms and minimizing costs. we design multi-checkpoint sampling and multi-aspect scoring strategies to ensure the diversity and reliability of self-distilled samples, followed by a gradientbased influence estimation method to select influential ones for training. we validate our method on llama3.1-8b-ins and qwen2.5-coder-7b/14b-ins, demonstrating their effectiveness as data synthesizers. based on the data generated by these small-scale synthesizers, we introduce scoder, a family of code generation models that achieves strong performance on humaneval (+), mbpp (+), livecodebench, and bigcodebench, showcasing the potential of small models in code instruction data synthesis.