Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024)

Abstract: While large language models have achieved remarkable performance on various code generation benchmarks, there have been growing concerns regarding potential contamination of these benchmarks as they may be leaked into pretraining and finetuning data. While recent work has investigated contamination in natural language generation and understanding tasks, there has been less extensive research into how data contamination impacts the evaluation of code generation, which is critical for understanding the robustness and reliability of LLMs in programming contexts. In this work, we perform a comprehensive study of data contamination of popular code generation benchmarks, and precisely quantify their overlap with pretraining corpus through both surface-level and semantic-level matching. In our experiments, we show that there are substantial overlap between popular code generation benchmarks and open training corpus, and models perform significantly better on the subset of the benchmarks where similar solutions are seen during training. We also conduct extensive analysis on the factors that affect model memorization and generalization, such as model size, problem difficulty, and question length. We release all resulting files from our matching pipeline for future research.

## introduction the compute requirements (encompassing both model size and data volume) for training large language models (llms) has grown significantly over the years, correlating with consistent observed enhancements in model performance in both language @xcite and code @xcite generation tasks. larger models trained on larger training corpora tend to lead to an increased risk of data contamination 1 code and data available at https://github.com/ yale-nlp/code-llm-contamination of the expected output, which we refer to as instances of evaluation benchmark data appearing within the data used during the training of models. llms tend to perform better on evaluation samples that resemble the documents and instances encountered during training @xcite @xcite , and are more likely to emit memorized training data when they have seen it multiple times @xcite . recent papers have also shown evidence that llms are possibly contaminated @xcite , which limits our understanding of their generalization capabilities to unseen inputs. despite significant research into data contamination in natural language (nl) benchmarks @xcite @xcite @xcite ), there's been relatively little exploration into how this issue affects the evaluation of code generation capabilities in llms. we posit that the fundamental disparities between nl and programs warrant a deeper examination. recent studies, such as work by @xcite ; @xcite , suggest that code-based llms may demonstrate patterns of memorization, underscoring the need for scrutiny into their generalization capabilities to unseen cases. key distinctions between code and nl include the critical role of syntax and the variable requirements for naming functions and variables across different programs. these differences lead us to argue that traditional surface-level comparisons might not be adequate for identifying contaminated data points. in this paper, we propose a pipeline to measure the overlap between code generation benchmarks and pretraining corpus of code llms, incorporating both surface-level and semantic-level code matching. as a result of the exhaustive search amoung the training corpus with our pipeline, we provide a precise quantification of the examples whose solutions are seen during training, for popular code generation benchmarks as mbpp @xcite and @xcite . we study two open pretraining corpus which contain code, the pile @xcite and star-coderdata @xcite , as well as three model series trained on either corpora, starcoder-base @xcite , pythia @xcite and codegen-nl @xcite . our results show severe contamination of the widely used mbpp and humaneval benchmarks within the pile and starcoderdata corpora, as shown in fig. 1 , with models performing significantly better on questions that the models have seen the same or similar program solutions to. we perform thorough analysis on factors that may affect model memorization and generalization such as model sizes and difficulty of the questions. we also include a case study on outliers, to provide a more comprehensive understanding of model behavior given different levels of exposure to test data. ## methodology to quantify data contamination for code llms, we first introduce methods used to measure program similarity from surface-and semantic-level in § 2.1. next, in § 2.2 we describe how to combine similarity measurements to identify the overlapping pro-grams in the training data and test benchmarks as well as introduce how to quantify data contamination based on the similarity scores and the number of appearances of similar programs seen during the course of training. ## experimental setup we select two of the most popular public pretraining corpora for general llm and code llms, namely the pile (gao et al., 2020) and star-coderdata @xcite , and three series of popular open-source models, i.e., pythia (biderman et al., 2023), codegen-nl @xcite and starcoderbase 6 (li et al., 2023). for the coding benchmark, we opt to study mbpp @xcite and @xcite due to their popularity. we introduce them in more detail in the following subsection. ## results in this section, we first present our main results in §4.1, then with several analysis on how the length, difficulty and model sizes affects the our findings in §4.2, and finally we present a case study in §4.3. ## discussions measuring contamination on the outputs. for a generation task, both the input and output are a sequence of tokens, allowing for the contamination of either the input sequence, the output sequence, or their coexistence within the training data to be measured. given that language models only use the input as context and do not attempt to generate it, we believe it is unlikely that seeing the input during training would help them generate better code at test time. instead we choose to measure data contamination of the outputs only. we believe that this is a reasonable and arguable the most effective strategy for code generation for the following reasons: 1) it is common to include the function name and signature in the task description for code generation tasks. after the model copies the function signature, having seen the function during training, it is easy for the model to reproduce the function body at test time. 2) since programs are formal languages with strict grammar rules, it is easier to measure the semantic similarity between programs with different surface forms. in our work, we measure the semantic similarity between programs based on their abstract syntax trees. such semantic comparison would be much harder and more prone to false-negatives for natural language (i.e., the input), as illustrated by recent work @xcite . 3) we find during our case study that while the models may struggle to reproduce gold programs that they have seen during training due to a misunderstanding of the natural language fig. 7 , there were only a few instances of this found. 4) while it is indeed the case that the model still needs to associate the natural language description with the code it has seen during training, our results (tab. 1 and fig. 1 ) suggest that there is a strong correlation between output side contamination and model performance, suggesting that having seen the similar code outputs at test time does provide an unfair advantage to the models. suggestions for future work. the issue of contamination poses a significant challenge for the future evaluation of large language models' capabilities. our research suggests that acknowledging the potential for contamination is crucial, especially when utilizing datasets known to be affected, such as humaneval and mbpp. when creating new datasets, while it is advised to follow our pipeline to decontaminate against the popular pretraining corpus (e.g., github), our results also suggest that it is possible to decouple complexity and memorization by cross referencing the results from models pretrained with different corpus (i.e., results in tab. 3 ). beyond mere surface-level matching, we advocate for a more nuanced approach to decontamination that incorporates understanding the semantics of code when creating pretraining data and new llms. furthermore, model developers should ideally include analyses on the effect of contamination on their evaluations. given the increasing scale and complexity of state-of-the-art models and dataset used to train these models, developing evaluation benchmarks that are both relevant and completely unseen by models during training seems critical, yet increasingly challenging. ## limitations false negative and positives. due to the flexibility of programs, there can be multiple correct ways to solve a problem using python. what we are searching for is only one possible solution presented as the gold solution, and as such we present our findings on the minimum number of questions to which models trained on the pile and starcoderdata have been exposed to. another source of false negatives (examples we falsely believe to be uncontaminated) is from the search itself. to reduce compute costs, we had to limit our search to relevant splits of the training corpus, which is the github split for the pile and the python split for starcoderdata. it is possible for the models to see more of similar solutions in other parts of the pretraining corpus (e.g., similar programs in java for starcoderdata). on the other hand, while performing semantic-level comparison helps with the general recall of similar programs, it is possible for it to flag a program as being similar to the gold program despite being quite different, creating false positive example. we present the results for all programs found to be perfect matches to the gold program in § a.4, and some of examples show that false positives do exist. different training stages. as mentioned in § 3.1, while we only study the "base" models and their pretraining data in this work, current models are typically trained in multiple stages, including supervised-finetuning, instruction-tuning and rlhf. however, the same methodology should be applicable to training data from these stages as well. at the same time, the size of such data is often several magnitudes smaller than the pretraining data, thus less likely to contain sources of contamination on the example-level. for example, an inspection on the instruction-tuning data for octocoder @xcite shows that none of the mbpp examples has a similarity score above 80. scarcity of open data. we only include two benchmarks and three models for this study. this is due to the scarcity of commonly used benchmarks that provide a gold program for every question, and models that have an open source training dataset. we hope more open models with open data will become available in the future to fuel further research in this domain. inability to retrain model. ideally, to observe the effect that having seen the answers to questions have on models, we would remove the answers from the training dataset, retrain the model, and compare with the original model. this approach is unfortunately prohibitively expensive given our compute constraints. ## conclusion in this work, we quantify the data contamination issues for two popular code generation benchmarks, namely mbpp and humaneval. we use both surface-level and semantic similarity to exhaustively search among the pretraining data using gold solutions for these benchmarks. by studying three series of models trained on two different corpus, the pile and starcoderdata, we find that significant portions of the benchmarks have solutions leaked into the pretraining data. further analysis shows that models perform much better when similar solutions are seen during training, and such correlation is independent of the difficult and length of the problems. prompt: write a function that takes two lists and returns true if they have at least one common element. 0 100 200 300 400 500 600 700 800 900 length of gold program 40 50 60 70 80 90 100 average aggregated score of top 10 programs correct incorrect figure 12: length of gold programs in the humaneval benchmark plotted against the average aggregated similarity score of the top-10 scores within the training dataset used for the starcoderbase model family.