K od C ode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding (2025)

Abstract: We introduce KodCode, a synthetic dataset that addresses the persistent challenge of acquiring high-quality, verifiable training data across diverse difficulties and domains for training Large Language Models for coding. Existing code-focused resources typically fail to ensure either the breadth of coverage (e.g., spanning simple coding tasks to advanced algorithmic problems) or verifiable correctness (e.g., unit tests). In contrast, KodCode comprises question–solution–test triplets that are systematically validated via a self-verification procedure. Our pipeline begins by synthesizing a broad range of coding questions, then generates solutions and test cases with additional attempts allocated to challenging problems. Finally, post-training data synthesis is done by rewriting questions into diverse formats and generating responses under a test-based reject sampling procedure from a reasoning model (DeepSeek R1). This pipeline yields a large-scale, robust and diverse coding dataset. It is suitable for supervised fine-tuning and the paired unit tests also provide great potential for RL tuning. Fine-tuning experiments on coding benchmarks (HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench) demonstrate that KodCode-tuned models achieve state-of-the-art performance, surpassing models like Qwen2.5-Coder-32B-Instruct and DeepSeek-R1-Distill-Llama-70B.

## introduction recent advances in large language models (llms) for coding such as qwen2.5-coder @xcite , and opencoder @xcite have demonstrated remarkable capabilities in programming tasks. these models excel at function writing @xcite , debugging @xcite , issue resolution @xcite , and agent ideally, training high-performing coding llms requires high-quality data with verified solutions and test cases for post-training stages, including supervised fine-tuning (sft) and reinforcement learning (rl) @xcite @xcite . while human-curated coding datasets like taco @xcite , apps @xcite , and code-contests @xcite offer high quality questions, canonical solutions, and tests, their limited scale constrains model training. synthetic datasets have emerged as an alternative @xcite , but often lack diversity @xcite , sufficient complexity @xcite , and reliable response verification @xcite . in this paper, we bridge this gap by introducing kodcode-v1, hereafter referred to as kodcode, a synthetic dataset consisting of 447k coding questions with verified solutions and unit tests. our approach starts from synthesizing coding questions from 12 sources using five distinct methods to ensure diversity and complexity. in solution & test generation step, we generate unit tests along with the solution and execute the unit test to verify the correctness of the solution. this self-verification mechanism not only ensures each solution is functionally correct but also offers verifiable correctness by providing explicitly curated unit tests. in addition, for challenging questions whose solutions fail the self-verification, we allocate additional attempts rather than discarding them, ensuring challenging questions are not filtered out during reject sampling. we further enhance the dataset for posttraining by rewriting questions into diverse formats and generating chain-of-thought (cot) responses using deepseek-r1 @xcite under a test-based reject sampling procedure. we conduct comprehensive analyses of our coding data generation pipeline. first, we evaluate our self-verification mechanism by testing solutions against human-written unit tests from the mbpp validation dataset. our experiments show that error rate remains below 2.5%, demonstrating the effectiveness of our self-verification mechanism. we then examine the benefits of scaled computation for challenging coding questions. finally, we perform statistical analyses of kodcode's token length, diversity, difficulty distribution, and potential contamination with existing benchmarks. furthermore, to validate kodcode's effectiveness for code llm post-training, we evaluate models using supervised sft and rl across standard benchmarks: humaneval(+), mbpp(+), big-codebench, and livecodebench. our experimental results show that kodcode-fine-tuned models achieve state-of-the-art performance, surpassing other open-source models such as qwen2.5-coder-32b-instruct and deepseek-r1-distill-llama-70b in most of the benchmarks. we hope our open-source dataset and models will help the community develop more capable coding assistants. we believe kodcode will advance current sft and rl post-training pipelines for code generation models, pushing the boundaries of llms in coding tasks. ## kodcode : synthesizing diverse, challenging, and verifiable correct post-training data for code as illustrated in figure 1 , our approach follows a three-step pipeline: coding question synthesis, solution & test generation, and post-training data synthesis. generally, we begin by synthesizing diverse coding questions q through a combination of prompt engineering and llm-based augmentation. next, we leverage a self-verification process to create high-quality solutions and test cases (sol, test) while offering verifiable correctness. finally, to provide high-quality post-training data, we diversify the generated synthetic questions by rewriting them into different formats, and generate responses by prompting a reasoning model (i.e., deepseek-r1) with a test-based reject sampling procedure. ## analysis in what follows, we conduct a comprehensive analysis to demonstrate the effectiveness of kod-code in generating diverse, challenging, and correct question-solution-test triplets. ## performance evaluation to evaluate the performance of the kod-code dataset, we conduct both supervised finetuning using qwen2.5-coder-32b-instruct @xcite from different sources and difficulties. the second version, kodcode-sft-hard-18k, contains all 18k instructions that are labeled as hard after applying the same preprocessing filters. we refer to the models fine-tuned on these two datasets as kod-code-32b-sft-50k, and kodcode-32b-sft-hard-18k. we use a cosine learning rate schedule with a maximum learning rate of 1 × 10 -5 when fine-tuning the qwen-2.5 model. the maximum sequence length is 16384. the detailed training configurations can be found in appendix c.1. rl setup. we conduct rl experiments on both qwen2.5-7b-instruct-1m @xcite and qwen2.5-coder-7b-instruct @xcite using 9.5k randomly selected samples from kod-codeas the training set, and 0.5k as the validation set. we refer to this dataset as kodcode -rl-10k. we perform grpo @xcite for 256 steps with actor learning rate of 5 × 10 -7 , 16 rollouts per question, a batch size of 256, max response length of 4096, and apply kl coefficient of 0.001. we assign binary rewards for rl: 1 if the model's solution passes all unit tests, and 0 otherwise. baselines. we compare our kodcode-tuned model against several strong baselines, including non-reasoning models (llama-3.3-70b-instruct (dubey et al., 2024), llama-3.1-tulu-3-70b (lambert et al., 2025), qwen-2.5-32b/72b-instruct @xcite , qwen-2.5-coder-32b-instruct @xcite ), and reasoning models (qwq-32b-preview @xcite , deepseek-r1-distill-llama-70b @xcite , sky-t1-32b-preview @xcite , and bespoke-stratos-32b @xcite ). benchmarks and evaluation setups. we evaluate models on humaneval(+) @xcite , mbpp(+) @xcite @xcite , and livecodebench (v5) @xcite , each designed to assess different aspects of code generation, including functional correctness, external library usage, and competitive programming challenges. we use evalplus for humaneval(+) and mbpp(+) evaluation, and skythought-evals @xcite for livecodebench evaluation. we evaluate performance on both complete and instruct subsets of bigcodebench. we follow the official setups in each benchmark and evaluate all models using greedy decoding with a maximum generation length of 16,384 tokens. we follow the official chat templates of the model during inference. ## conclusion and future work in this work, we presented kodcode, a largescale synthetic dataset of 447k diverse coding ques-tions paired with verified solutions and unit tests. our three-step synthesis pipeline-comprising coding question generation, solution and test case refinement via self-verification, and post-training data synthesis-ensures both the diversity and quality of training data for high-performing coding language models. through detailed analysis, we validate our pipeline's effectiveness and thoroughly examine the dataset's attributes. comprehensive experiments demonstrate that models finetuned on kodcode not only achieve state-of-theart performance across multiple benchmarks (hu-maneval(+), mbpp(+), bigcodebench, and live-codebench) but also outperform larger models. future work will focus on three directions. first, we plan to scale up the dataset with more challenging problems, as our findings indicate that difficult instances significantly improve model performance. second, we aim to investigate optimal strategies for post-training data selection. finally, we will explore methods for generating repository-level synthetic data to further enhance coding llms.