Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models (2024)

Abstract: Recent statements about the impressive capabilities of large language models (LLMs) are usually supported by evaluating on open-access benchmarks. Considering the vast size and wide-ranging sources of LLMs’ training data, it could explicitly or implicitly include test data, leading to LLMs being more susceptible to data contamination. However, due to the opacity of training data, the black-box access of models, and the rapid growth of synthetic training data, detecting and mitigating data contamination for LLMs faces significant challenges. In this paper, we propose CDD, which stands for Contamination Detection via output Distribution for LLMs. CDD necessitates only the sampled texts to detect data contamination, by identifying the peakedness of LLM’s output distribution. To mitigate the impact of data contamination in evaluation, we also present TED: Trustworthy Evaluation via output Distribution, based on the correction of LLM’s output distribution. To facilitate this study, we introduce two benchmarks, i.e., DETCON and COMIEVAL, for data contamination detection and contamination mitigation evaluation tasks. Extensive experimental results show that CDD achieves the average relative improvements of 21.8%-30.2% over other contamination detection approaches in terms of Accuracy, F1 Score, and AUC metrics, and can effectively detect implicit contamination. TED substantially mitigates performance improvements up to 66.9% attributed to data contamination across various contamination setups. In real-world applications, we reveal that ChatGPT exhibits a high potential to suffer from data contamination on HumanEval benchmark.

## introduction in recent years, llms have revolutionized the fields of natural language processing (nlp), artificial intelligence, and software engineering. to evaluate llms' capabilities in various downstream tasks, such as automatic question answering, natural language reasoning, and code generation, people conduct extensive tests for llms based on enormous benchmark datasets @xcite . the results indicate that llms exhibit superior performance on these tasks. while marveling at the powerful capabilities of llms, people usually want to determine whether an llm's excellent performance is due to the genuine understanding of tasks to achieve generalization, or merely because it has seen the test data to form memorization, i.e., suffering from data contamination. data contamination, also known as data leakage, refers to the scenario where the test data has been included in the model's training data @xcite , leading to the model performing exceptionally well on these leaked test data. owing to the vast size and wide-ranging sources of the pre-trained datasets for llms, they are more susceptible to data contamination, which can be primarily categorized into two situations: data contamination exerts a profound and deleterious impact on llms @xcite @xcite . as shown in figure 1 , with llms continuing to learn on contaminated data (i.e., both leaked data and other training data), their performance keeps improving on leaked data but stagnates and even degrades on similar data. this example reflect that data contamination can lead to a substantial overestimation of models' performance, thus affecting the trustworthiness and effectiveness of llms in practical applications. furthermore, data contamination may also conceal the potential flaws of models, presenting major obstacles for people to identify and improve upon llms' shortcomings. therefore, it is crucial for llms to detect data contamination and ensure trustworthy evaluation. although acknowledged the significance, data contamination detection and trustworthy evaluation for llms still persist as open and challenging issues @xcite . the difficulties of data contamination detection can be essentially attributed to three factors: 1) opaque training data. it is usually non-public and comprehensive, while continuously evolving for new llms. 2) black box models. the parameters and output probabilities of llms may not be available, such as chatgpt and @xcite . 3) proliferation of synthetic data. it could implicitly introduce the variants 2 of test data to training data. further, the evaluation to mitigate the impact of data contamination has hardly been studied. in this paper, we overcome the preceding challenges by proposing cdd: contamination detection via output distribution for llms. cdd uses the sampled texts to identify the peakedness of llm's output distribution for data contamination detection. we follow a hypothesis that training is likely to alter the model's output distribution, resulting in a more peaked output distribution for 2 these variants may include, but are not limited to, translations into other languages, additions of explanations or intermediate processes, and provisions of alternate solutions. ## motivation example a powerful llm that transcends memorization has the capability to generate diverse outputs in response to a given input. considering the huge vocabulary size of llms, which encompasses a good number of tokens with analogous semantics, the output distribution sampled from llms ought to not exhibit peakedness. however, when llms solely form memorization via training, llms are prone to generate outputs that abnormally resemble their training data, as shown in figure 2 . from a statistical perspective, assuming that the average probability of llm's output tokens is 0.95, the likelihood of sampling two outputs that contain the same 100 consecutive tokens is about 0.005 < 0.01, which is an extremely improbable event. therefore, if an llm consistently outputs some identical or highly similar texts through sampling, it is most 2 12040 likely caused by memorization. figure 3 displays an example of how the llm's output distribution changes as the degree of data contamination varies. we model the llm's output distribution by computing the edit distances of sampled texts, referred to as edit distance distribution ( § 3.1). as shown in figure 3 , in the absence of data contamination (i.e., occurrence 0), the density of zero edit distance stands at 0.0035, where zero edit distance means that sampled texts exactly match. however, upon the llm being exposed to the leaked data even once (i.e., occurrence 1) during training, the density of zero edit distance escalates sharply to more than 20 times larger than the original, showing the peakedness. therefore, the impact of data contamination on the llm's output distribution is substantial. in this paper, to the best of our knowledge, we are the first to consider from the standpoint of llms' output distribution to address the challenges associated with data contamination detection and contamination mitigation evaluation, employing only the sampled texts without access to the output probability and training data. ## methodology in this section, we first establish the edit distance distribution ( § 3.1), and then on this basis, we design cdd for data contamination detection ( § 3.2) and ted for contamination mitigation evaluation ( § 3.3). ## experiment in this section, we first introduce two datasets, det-con and comieval, tailored for the tasks of data contamination detection and contamination mitigation evaluation, respectively ( § 4.1). we then evaluate the efficacy of cdd on detcon dataset ( § 4.2). following this, we assess the performance of ted on comieval dataset ( § 4.3). finally, we demonstrate the application results of both cdd and ted in real-world scenarios ( § 4.4). ## conclusion in this paper, we have proposed two novel approaches, namely cdd and ted, to deal with data contamination detection and contamination mitigation evaluation for llms, considering the llm's output distribution. we construct two corre-8 sponding datasets, i.e., detcon and comieval, for these two tasks. extensive experimental results indicate the superiority and versatility of cdd and ted. moreover, we also discover that chatgpt is likely to suffer from data contamination on hu-maneval dataset. we hope to shed light on this direction and call more attention to data contamination issues.