On Leakage of Code Generation Evaluation Datasets (2024)

Abstract: In this paper, we consider contamination by code generation test sets, in particular in their use in modern large language models.We discuss three possible sources of such contamination and show findings supporting each of them: (i) direct data leakage, (ii) indirect data leakage through the use of synthetic data and (iii) overfitting to evaluation sets during model selection.To address this, we release Less Basic Python Problems (LBPP): an uncontaminated new benchmark of 161 prompts with their associated Python solutions. LBPP is released at https://huggingface.co/datasets/CohereForAI/lbpp

## introduction code generation has emerged as an important skill for large language models to master. measuring recent progress in code generation has relied on few, critical benchmarks to judge performance between model families and checkpoints. while many recent sophisticated evaluation datasets have been proposed @xcite , the community largely relies on humaneval @xcite and mbpp @xcite to judge a new model's code capability. in fact, all major announcements in 2023-2024 claiming advanced code capabilities-from either academic or industry labs-boast one or both of these datasets. practically, reporting humaneval and mbpp is mandatory for a model to report competitive code generation. however, the importance of these benchmarks has led to a conflict between popularity and utility. obtaining competitive numbers comes with significant scientific and economic reward-made increasingly easy with the proliferation of public replicas and evaluation harnesses featuring these datasets. however, this prevalence has led to data leakage beyond the original evaluation scope. when this evaluation data contaminates model training, the validity of the metrics as a measure of generalization capability becomes unreliable. if a model is trained on data used for out-of-distribution generalization (or selected for its performance on that data), we break an implicit tenet of how model capability should be measured. we argue that understanding the effect of contamination is critical to accurately interpreting scores on these benchmarks. in this paper, we review the evidence that most contemporary llms are contaminated with data sourced from these two benchmarks. we define contamination here as any procedure leaking datasets during model training such that these datasets are no longer unseen at inference. the most obvious method of contamination is the presence inside training data. section 3.1 reviews evidence that these benchmarks are widespread in training corpora in original and paraphrased forms. unfortunately, it is not feasible to manually remove all the corresponding examples from the training corpora and the most common automatic decontamination methods have low recall. section 3.2 proposes that contamination also occurs indirectly through the use of synthetic data-a widespread paradigm used to increase coding capabilities by generating additional code training tokens for pretraining or fine-tuning. finally, section 3.3 argues that checkpoint selection may be overly influenced by these datasets, overfitting to these benchmarks over general-purpose code-oriented generalization. in this paper, we propose a more challenging python code generation benchmark: less basic python problems. lbpp is similar in size to humaneval and mbpp, but designed to be more complex using model in the loop filtering. lbpp is also designed to share no inspiration or sources with existing training and evaluation datasets, presenting a novel generalization challenge to contemporary llms. in section 4, we observe that sota models on humaneval perform up to 43% worse on lbpp. we contribute lbpp as a genuinely held-out test to measure current code generation capability, and potential overfitting to humaneval and mbpp. ## related work humaneval @xcite and mbpp @xcite remain the most reported results on public leaderboards, but similar datasets exist @xcite . they consist of short and mostly simple (not programming competition level) instructions with completions in python. these datasets have also been translated into more programming languages @xcite , as well as versions with additional tests @xcite . @xcite proposes a continuously updated set of interview questions to improve dataset challenge by including harder and novel (unseen) prompts. @xcite aims for challenging software engineering problems requiring understanding of entire repositories. in a similar vein, repoqa @xcite and bug in the code stack @xcite focus on understanding long contexts within code tasks. @xcite proposes using hidden evaluation sets, however, this approach does not allow inspection of failure cases and requires trusting the quality and correctness of an opaque 'black-box' evaluation setup. recently, @xcite analyzed data contamination in popular pretraining datasets: reporting that 12.2% of humaneval samples are present in the pile @xcite , and 18.9% in the stack @xcite . while @xcite reports "we do not find the performance rankings of the models to change with decontaminated results", we identify the ranking between models to vary between contaminated and uncontaminated evaluation datasets (see ## possible sources of contamination we provide three hypotheses, with respective evidence, on why existing evaluation datasets are leaked and models may already be over-optimized towards existing leaked benchmarks. ## lbpp: less basic python problems as mentioned above, we have created less basic python problems (lbpp) for a less biased measure of modern models' code capabilities. lbpp is a dataset of 161 code completion problems in the style of humaneval. all prompts include evaluation unit tests, with a median of 4 tests per example. dataset annotation: human annotators were asked to create brand new problems that were not solvable by a strong internal model in the loop. all annotators had competitive programming experience. they were instructed to come up with unique unsolved problems in lbpp given a list of unique words each of size k and an n sized word, w, where n is a multiple of k, write a program in python to determine the number of unique combinations of words in the list that can be concatenated to form an anagram of the word. write a function in python that takes as input a recursive function and some parameters. the function should return the number of times the recursive function ran itself when starting it with the provided parameters. write a monte carlo function in python to compute the median number of cards you'd need to draw from a deck such that the sum equals or exceeds the value v . all annotators were paid above minimum wage in their respective countries, and all final promptcompletion pairs were manually reviewed by the authors. this adversarial collection resulted in more difficult problems, with most models solving less than 50% of the dataset. initial results: ## conclusion we study the cause and effect of data contamination via two popular code generation benchmarks. mbpp. lbpp performance correlates with both prior datasets, but is designed to be genuinely unseen by contemporary llms. our analysis highlights that contamination is likely unavoidable at the llm scale given the difficulty of filtering every potential permutation of a benchmark dataset. this insight motivates our contribution of lbpp: a novel code generation benchmark to evaluate contemporary llms in a contaminationfree setting. we are well aware that our decision to release this dataset will make future leakage impossible to control. however, with the context of the fast-paced model development cycles that llms are currently undergoing we believe that releasing this increases the trustworthiness and usefulness of this dataset. it is conveniently designed to serve as a drop-in replacement (or addition) of current evaluation sets. on top of its novelty, the more challenging nature of this dataset also provides a cleaner signal for model comparison.