Mastering the Craft of Data Synthesis for C ode LLM s (2025)

Abstract: Large language models (LLMs) have shown impressive performance in code understanding and generation, making coding tasks a key focus for researchers due to their practical applications and value as a testbed for LLM evaluation. Data synthesis and filtering techniques have been widely adopted and shown to be highly effective in this context. In this paper, we present a focused survey and taxonomy of these techniques, emphasizing recent advancements. We highlight key challenges, explore future research directions, and offer practical guidance for new researchers entering the field.

## introduction code intelligence leverages machine learning techniques to enhance software development by improving both code quality and programmer productivity @xcite . the rise of llms, such as chat-gpt (openai, 2023), gemini @xcite , claude @xcite , and llama @xcite , has significantly reshaped the automation of code-related tasks, including code completion @xcite , translation @xcite , repair @xcite , and documentation (khan and uddin, 2022). tools like github copilot @xcite , codegeex @xcite , and cursor (cursorai, 2024) hold great promise in substantially increasing human programmer efficiency and revolutionizing the software industry, attracting considerable attention from both academia and industry. recently, specialized llms for code-related tasks (denoted as codellms) have emerged, including code llama @xcite , starcoder @xcite , @xcite , and codeqwen @xcite . recent advancements @xcite in llms have highlighted the critical role of high-quality data in building strong, robust models. similarly, for codellms, diverse, high-quality datasets are essential for improving performance across a wide range of code-related tasks. significant efforts have been devoted to collecting and curating code-related corpora. prominent examples include the pile @xcite , the stack @xcite and bigscience roots @xcite , which draw primarily from open-source and permissively licensed platforms such as github and stack overflow. however, relying solely on human-generated data for code-related tasks poses several challenges. first, collecting large-scale human data is labor-intensive and expensive, particularly for high-quality instruction tuning and preference alignment data. second, human-generated data is prone to biases and errors @xcite , as it reflects the varying skill levels of programmers, and may not be optimal for model training. third, data integrity concerns, such as the risk of sensitive personal/corporate information leakage, complicate data collection. lastly, for low-resource programming languages--either due to limited popularity or proprietary restrictions--data scarcity hinders the effectiveness of codellms in specialized fields and systems programming @xcite . consequently, synthetic data generated by llms has emerged as a valuable alternative to complement natural data. leveraging their vast knowledge and advanced linguistic capabilities, llms can generate high-quality data, providing a valuable foundation for model training in code-related tasks. while generating synthetic datasets for coderelated tasks may appear straightforward, achieving both high accuracy and sufficient diversity is a complex process requiring meticulous design and advanced techniques @xcite . this makes a systematic exploration of llm-driven synthetic data generation both essential and timely. although there are survey papers in the fields of general data engineering @xcite @xcite and code intelligence @xcite @xcite , there is a notable gap in literature focusing specifically on data synthesis and filtering techniques for coderelated tasks. to fill this gap, we present a targeted review of recent advancements in synthetic data generation and filtering for training codellms, covering over 50 recent works across 23 topic categories from the past two years. the techniques discussed are organized into a taxonomy (fig. 2 ) and analyzed in terms of their motivation, methodologies, and key contributions. we also maintain a github repository 1 to collect open-source synthesis datasets for codellms and track recent advancements. our goal is to provide an in-depth overview of the current state of the field, highlight key challenges, and offer insights to guide researchers and practitioners in building efficient and robust codellms through effective data engineering practices. ## key data synthesis techniques this section reviews recent data synthesis techniques for code-related tasks, structured by the taxonomy in figure 2 along three dimensions: building phases, core objectives, and specific tasks. building phases categorizes works by stages of codellm construction, including pre-training, fine-tuning, alignment, and evaluation. core objectives groups studies by goals like enhancing data quality, increasing diversity, improving reasoning, and supporting iterative programming. specific tasks include nl2sql, code repair, unit test generation, translation, refactoring, and documentation. ## key data filtering techniques data filtering is the process of selecting specific subsets of data based on predefined criteria to optimize performance. effective filtering offers key advantages: (1) improving model accuracy by reducing noise and bias, especially in synthesized datasets; ## challenges and future directions we envisage the following important challenges and research directions worthy of investigation. supporting low-resource languages. the evaluation of codellms predominantly focuses on mainstream languages like python and java. however, data synthesis and filtering play an even more important role for low-resource languages @xcite , which include legacy languages such as cobol, fortran, and haskell; domain-specific languages like r and elixir; and commercial languages such as ibm rpg, oracle suitescript, and sap abap. ## practical guidance this section introduces a streamlined pipeline (fig. 1 ) for synthetic data generation in codellms, providing practical guidance for researchers. ## conclusion code-related tasks, showcasing llms' capabilities, have gained significant interest for their practical value and as a robust testbed for llms. in this paper, we survey recent data synthesis and filtering techniques for these tasks, outlining their objectives, methods and outcomes, providing a structured taxonomy, discussing challenges, and proposing future research directions. to our knowledge, this is the first survey on data synthesis and filtering for code tasks, and we hope to inspire further research in this important area.