How Do Your Code LLM s perform? Empowering Code Instruction Tuning with Really Good Data (2024)

Abstract: Recently, there has been a growing interest in studying how to construct better code instruction tuning data. However, we observe Code models trained with these datasets exhibit high performance on HumanEval but perform worse on other benchmarks such as LiveCodeBench. Upon further investigation, we find that many datasets suffer from severe data leakage. After cleaning up most of the leaked data, some well-known high-quality datasets perform poorly. This discovery reveals a new challenge: identifying which dataset genuinely qualify as high-quality code instruction data. To address this, we propose an efficient code data pruning strategy for selecting good samples. Our approach is based on three dimensions: instruction complexity, response quality, and instruction diversity. Based on our selected data, we present XCoder, a family of models finetuned from LLaMA3. Our experiments show Xcoder achieves new state-of-the-art performance using fewer training data, which verify the effectiveness of our data strategy. Moreover, we perform a comprehensive analysis on the data composition and find existing code datasets have different characteristics according to their construction methods, which provide new insights for future code LLMs.

## introduction code pre-trained models have achieved remarkable progress in the era of large language models (llms), such as codex @xcite , al-phacode @xcite , @xcite and starcoder @xcite . training on large code corpora @xcite has been shown to enhance the coding capabilities of current llms @xcite ; rozière @xcite . in addition to costly pretraining, recent research has garnered increased interest in code instruction tuning and obtains promising results on several code benchmarks (chaudhary, 2023; @xcite @xcite @xcite @xcite . differing from the high demand of pre-training for data quantity, instruction tuning aligns existing model abilities towards a desired direction using high-quality but much smaller datasets. to construct code instruction datasets, earlier research predominantly relies on heuristic automation (e.g. distillation from chatgpt) or manual selection. for example, code alpaca @xcite and wiz-ardcoder @xcite ) use distillation signals from chatgpt via self-instruct and evol-instruct. other methods such as octopack @xcite and magicoder @xcite construct code instructions from pre-training code corpora. although these code instruction datasets seem excellent on popular code benchmarks like humaneval foot_0 , we find some of them dramatically drop on another contamination-free benchmark livecodebench @xcite which continuously collects new problems over time from on-line contests. as shown in figure 1 , magicoder evol-instruct and code-feedback @xcite achieve top ranks on humaneval but drop on livecodebench. we perform a further decontamination process and find that several existing code models achieve abnormally high performance on humaneval because of the potential use of the benchmark or benchmark-similar data. thus, it remains unclear what good code instruction data is and how these datasets actually work. besides, all the data come from different pipelines and have no unified principle to ensure good quality. we need to systematically define what constitutes good examples of data for code instruction tuning and establish an effective principle for achieving competitive performance using only highly valuable samples. in this work, we aim to define the characteristics of good data for code instruction tuning based on a diverse range of existing code datasets. our goal is to select the most influential samples through a comprehensive and quantitative data assessment measure. drawing inspiration from @xcite ; @xcite , we propose a paradigm of data-efficient instruction tuning for code capabilities. generally, we assume good code samples are complex, of high quality, and diverse. for the complexity aspect, we adopt the evolved complexity scorer to predict the complexity of a given instruction. the scorer is trained on evolved samples via the complexity prompt @xcite with chatgpt. for the aspect of quality, we train a verified model to generate multiple test cases given an (instruction, response) pair and evaluate its quality via the pass rate of the generated test cases. for the aspect of diversity, we select the sample with a large distance to a data pool via instruction embeddings. combining the three measures, our simple but effective data selection strategy pursues valuable code instruction data and achieves more efficient instruction tuning where fewer training samples yield performance on par with, or even surpassing, models trained on significantly larger datasets. moreover, we also analyze the composition of our selected data mixture and give suggestions for future code instruction tuning research. we present xcoder, a family of models finetuned from llama3 3 using our selected code instruction data mixture. experiments on live-codebench and humaneval demonstrate that 3 https://llama.meta.com/llama3/ xcoder is able to outperform or be on par with state-of-the-art code instruction models such as wizardcoder @xcite , magicoder @xcite , starcoder2-instruct 4 and open-codeinterpreter @xcite while using fewer automatically selected data examples. for example, xcoder-8b based on llama3-8b achieves 43.66 livecodebench-easy and 54.9 humaneval when trained on only 40k data samples. besides, our xcoder-70b based on llama3-70b achieves top-tier results compared to the state-of-the-art open-source models. ## deep dive into existing datasets we present mainstream and open-source code instruction tuning datasets in from the results, we observe that different training datasets lead to significant performance differences on humaneval, but the differences on live-codebench are minimal. this phenomenon leads us to suspect whether the remarkably high performance of some data in humaneval is due to data leakage. therefore, we propose the test leakage index (tli) to detect the degree of data leakage for each dataset in the test set. ## what characteristics do good data have in this section, we first define the characteristics of good data for code instruction tuning and then select the most influential samples via data pruning. inspired by deita @xcite , we select the samples in the data pool from three dimensions: instruction complexity, response quality, and instruction diversity. for a data pool p , we first use the a complexity score c and unit test model u to calculate the complexity score c and quality score q for each data. then, we use linearly combine c ′ and q ′ to obtain a score s representing complexity and quality. finally, we sort the data pool p and apply the diversity-based sampling to iteratively select samples from the data pool into the final training set d, until d reaches the budget size. our data selection approach is illustrated in the figure 2 and algorithm 1. the details of complexity score, unit test model and diversity-based sampling are as follows. ## conclusion and future work code llms have raised great interest in current llm research and plenty of code instruction datasets are proposed over time. however, although many of them claim that good results are achieved on the popular benchmark humaneval, we find several datasets may have data leakage by using benchmark samples as seed data in self-instruct or evolveinstruct. in this paper, we aim to identify which dataset genuinely qualifies as high-quality code instruction data and propose an efficient code data selection strategy for selecting valuable samples. based on three dimensions of assessing data, we present xcoder, a family of models finetuned from llama3 on our selected dataset. xcoder achieves superior performance than the sota baselines using fewer training samples. from the composition of our selected data mixture, we find existing code datasets have different characteristics corresponding to their construction methods, which provide new insights for developing better code llms.