An Open-Source Data Contamination Report for Large Language Models (2024)

Abstract: Data contamination in model evaluation has become increasingly prevalent with the growing popularity of large language models. It allows models to “cheat” via memorisation instead of displaying true capabilities. Therefore, contamination analysis has become an crucial part of reliable model evaluation to validate results. However, existing contamination analysis is usually conducted internally by large language model developers and often lacks transparency and completeness. This paper presents an extensive data contamination report for over 15 popular large language models across six popular multiple-choice QA benchmarks. We also introduce an open-source pipeline that enables the community to perform contamination analysis on customised data and models. Our experiments reveal varying contamination levels ranging from 1% to 45% across benchmarks, with the contamination degree increasing rapidly over time. Performance analysis of large language models indicates that data contamination does not necessarily lead to increased model metrics: while significant accuracy boosts of up to 14% and 7% are observed on contaminated C-Eval and Hellaswag benchmarks, only a minimal increase is noted on contaminated MMLU. We also find larger models seem able to gain more advantages than smaller models on contaminated test sets.

## introduction recent years have seen remarkable progress in large language models (llms) pre-trained on massive text corpora scraped from the web. however, many widely used evaluation benchmarks are also constructed from similar web sources, leading to a concerning issue of data contamination where examples from test sets are unintentionally included in training data. contamination enables models to "cheat" via memorisation of test data rather than displaying true generalisation @xcite , which creates an illusion of progress, distorts model com-parisons, and undermines the utility of benchmarks @xcite . contamination analysis therefore becomes a crucial part of reliable llm evaluation to validate the results. however, as the training data of llms is often not openly accessible, existing contamination analysis is mostly conducted internally by llm developers and thus often lacks transparency and completeness. for instance, openai's contamination study for gpt-4 (openai, 2023) only covered the pre-training data and omitted later finetuning stages. llama 2 @xcite only reported contamination statistics for two of the 20+ benchmarks used in their evaluation. in addition, the implementation details of their contamination identification remain unclear. overall, existing internal contamination studies tend to lack sufficient transparency, with minimal sharing of contamination measurements across all evaluation benchmarks, as well as training data details and code to reproduce the results. this prevents the wider research community from fully auditing the credibility of reported metrics and model capabilities. this paper introduce a practically applicable pipeline that enables the community to identify potential data contamination from their benchmarks. specifically, our method uses search engines and common crawl as a proxy for contamination identification, i.e, if a test example is found verbatim in search engine and common crawl, we consider it a "contaminated" sample. this is based on the observation that modern llms' pre-training data rely heavily on online resources and common crawl @xcite . our method provides a practical and affordable solution for contamination detection that avoids the need for llms' training data (inaccessible in most cases) and avoids the expensive local indexing of huge corpora that often requires multi-petabyte storage (details in §4). with this method, we present an open contam-ination analysis for over 15 popular large language models on six common multiple-choice nlp benchmarks, to provide comprehensive measurements of benchmark contamination and its impact on model evaluation. the analysis includes a range of foundation models such as llama @xcite , llama-2, yi @xcite , mistral @xcite , baichuan @xcite , and qwen @xcite across multiple model sizes (7b, 13b, 30b, 34b, 65b, 70b parameters) as well as instruct-tuned models built on these foundations like llama-2 chat and mistral-instruct. six widely used multi-choice benchmarks are assessed: winogrande @xcite , ai2_arc @xcite , commonsenseqa @xcite , hellaswag @xcite , mmlu @xcite , and c-eval @xcite . our analysis reveals the following key findings: 1) we detect varying levels of data contamination across benchmarks, with 1% to 45.8% of examples showing verbatim overlap with common crawl; 2) by comparing the contamination degree between common crawl dec 2020 to oct 2023, we find data contamination grows rapidly through time; 3) data contamination can significantly inflate model performance: accuracy increases of 14% and 7% were found on c-eval and hellaswag, and a slight increase was found on mmlu; 4) we find larger models tend to benefit more from data contamination than smaller models, perhaps due to their more powerful memorisation capacities; 5) finally, we show our results align well with llama's original contamination reports, demonstrating the effectiveness of our method. our data and code can be found in https://github.com/liyucheng09/ contamination_detector . ## data contamination what is data contamination? data contamination refers to the phenomenon that examples from the test set are also found in the training data. this might lead to the evaluation failing to accurately reflect models' capabilities, as models can cheat by memorising instead of learning to generalise. there are two primary types of data contamination @xcite : input-only contamination refers to cases where only the input appearing in the pretraining corpus, and input-and-label contamination occurs when both inputs and their labels are present. the latter is generally more problematic, as models can directly memorise input-output pairs. but the first may still cause issues as models may learn from the context. how common is data contamination? data contamination appears to be quite widespread across commonly used nlp benchmark datasets based on findings from recent studies. @xcite and @xcite audited well-known big language corpora such as c4, the pile, and redpajama, revealing contamination rates ranging from 0% to over 50% on glue and superglue benchmarks. the gpt-3 study @xcite found over 90% of examples in quac, squadv2, and drop were flagged as contaminated. flan @xcite evaluations identified 7 out of 26 datasets exhibiting a serious contamination ratio of 50% and over. llama-2 @xcite reported over 16% of mmlu examples are contaminated and about 11% are seriously contaminated (more than 80% token leakage). gpt-4 (openai, 2023) uses academic exams and nlp benchmarks for model evaluation. while 4 out of 34 exams were found to have zero contamination (e.g., leetcode and bar exam), 9 out of 34 showed over 20% of instances marked as dirty examples. @xcite provide a comprehensive collection of evidence of data contamination in nlp datasets and models, where data contamination is found in hundreds of widely-used datasets, popular pre-training corpora, and state-of-the-art llms. how to identify data contamination? @xcite take a straightforward approach to detect exact matches between test set examples and the pretraining data after normalising for capitalisation and punctuation. exact match here means the entire input of an evaluation text is found in the training data. the gpt-3 paper @xcite uses n-gram overlap to identify contamination, treating any examples with 13-gram cooccurrence in both test sets and training data as dirty examples. llama-2 matches on verbalised and tokenized input to allow a token-level approach to identify contamination. it also involves a "skipgram budget" to allow slight variants in overlapping. overall, existing approaches usually use substring matching between evaluation examples and training data to identify data contamination. however, if we have no access to the training data, which is often the case for most recent closed models, it is extremely difficult to reveal contamina-tion by observing models themselves. pioneering studies propose to identify data contamination by measuring perplexity of test examples @xcite , asking models to reconstruct test examples verbatim @xcite , or examining models' preference on test sample ordering @xcite . in this paper, we use common crawl as a proxy for the training data of llms, as it often constitutes a significant part of it, thereby avoiding the need to access the full training dataset. to what extent does data contamination affect model evaluation? while contaminated data can potentially inflate scores, models do not necessarily perform worse on clean subsets or better on dirty subsets across all datasets. the degree of impact likely depends on many factors like the dataset characteristics, model scale, and nature of the pretraining data. for instance, gpt-3 @xcite showed a marginal 1-2% performance drop on clean subsets for piqa and record, compared to a significant 6% drop on clean set of squad as 94% of its test examples were contaminated. @xcite found a significant association between a code problem's presence on github and gpt-4's pass rate for that problem. but on other academic tests, gpt-4 showed little performance difference on the clean and contaminated test sets (openai, 2023). @xcite reported a 15.3 point and a 9.8 point gap from llama-2 70b by comparing the its performance on the clean and dirty sets of hellaswag and mmlu-humanities. ## benchmarks for language models clean and robust benchmarks are important to guide further progress of various models in nlp. popular benchmarks used to evaluate large language models include: comprehensive: mmlu, big bench @xcite , agi eval @xcite , c-eval commonsense reasoning: piqa @xcite , siqa @xcite , hellaswag, winogrande, arc, open-bookqa @xcite , com-monsenseqa world knowledge: naturalquestions @xcite , triviaqa @xcite reading comprehension: squad (rajpurkar et al., 2018), quac @xcite , boolq @xcite math: gsm8k @xcite , math @xcite code: humaneval @xcite , mbpp @xcite the construction of many of these relies heavily on online materials, therefore they are highly prone to data contamination as their source spreads on the internet. here we analyse six representative multi-choice qa benchmarks: mmlu, c-eval, winogrande, commonsenseqa, arc, and hel-laswag. these benchmarks have been selected due to their varied sources and potential susceptibility to data contamination. mmlu, arc, and c-eval, which are academic test-based benchmarks, were compiled from online .docx/.pdf files using techniques like ocr, typically assumed to be less affected by data contamination as such files are often not indexed by online crawlers. however, c-eval stands out as it is a non-english (chinese) benchmark, offering an opportunity to assess the impact of non-english benchmarks on language models. winogrande, uniquely human-authored from scratch, allows examination of whether manually created benchmarks are less prone to data contamination. commonsenseqa and hellaswag, both internet-sourced, differ in their source popularity; while commonsenseqa is built upon the less influential conceptnet, hellaswag is sourced from the more popular wikihow. this selection of benchmarks provides a comprehensive overview of how different sourcing and construction methods might influence the presence and extent of data contamination in language model evaluations. ## our approach the central goal of data contamination analysis is to categorise test samples as either clean or contaminated and then evaluate models separately on the clean and contaminated samples to assess the impact of contamination on the performance metrics. in this section, we describe our methodology to identify contaminated test samples. the basic idea is to check whether test examples appear verbatim in both search engine and common crawl. we use search engine and common crawl because they are accessible, affordable and often comprise the majority of pre-training data for large language models, e.g., common crawl constitutes over 80% in gpt-3 and llama training data @xcite and the remainder of pre-training data also relies heavily on online resources. we identify data contamination in two steps. first, 1) we use the bing search api to check if test examples appear verbatim online, which indicates their potential inclusion in llms' pre-training data. second, 2) we verify if the page containing verbatim test examples we found in step 1) were also indexed in common crawl. here we check the presence of test examples in both search engine and common crawl to address a possible false positive issue. and since the second step only involves url search within common crawl instead of string retrieval, it avoids the expensive local indexing of the entire common crawl. the search window of our contamination detection starts from 2017 and ends at the knowledge cutoff with respect to different llms. this is realised via adjusting the freshness parameter in the bing api and using the appropriate indexes of common crawl during the identification. note that we use the release date as the end of the search window for llms for which we are not aware of their knowledge cutoff. to construct the search queries, we verbalise examples accordingly and make sure the question and the correct answer are involved in the queries. for example: question: the flaw in anderson's act theory was that some considered it ____. ## contamination statistics for multi-choice benchmarks our analysis reveals varying levels of data contamination across six multi-choice qa benchmarks, as shown in we also illustrate how data contamination increases over time, as shown in figure 2 . in the figure, benchmarks such as commonsenseqa and winogrande maintain very low rates of contaminated data, with increases of just 0.3% and 0.2% over the past three years. however, benchmarks collected from academic tests like arc, mmlu, and c-eval have experienced a substantial increase in contamination, with up to 21% of examples flagged as contaminated during the same period. this shows how test content in academic benchmarks can easily propagate across the internet, which can be a serious issue for academic test based language model benchmarks. we also observe a moderate 8.3% increase for hellaswag, further demonstrating the increasing risk of data contamination for internet sourced benchmarks. in figure 3 , we illustrate where these hellaswag contaminated test samples come from. we discover that data contamination manifests in a centralised fashion, which means contaminated test samples are not evenly distributed across domains. instead, they are significantly concentrated in specific domains and rare in others. this finding is meaningful as it reveals the possibility that blocking specific domains during training data collection might alleviate the issue of data contamination. you can find more domain analysis and contamination examples in appendix a. ## impact of contamination on model performance to assess how data contamination impacts model evaluation, we test popular large language models on contaminated and clean splits of each benchmark. as shown in the previous section, we categorise benchmarks into four subsets: 1) the clean set; 2) not clean set; 3) input-only contaminated set; and 4) input-and-label contaminated set. note that 'not clean' = input-only + input-and-label contamination. since commonsenseqa and winogrande are shown to be just marginally contaminated, we focus on mmlu, c-eval, hellaswag and arc in these experiments. we only report input-only contaminated performance for mmlu, as the other benchmarks have too few samples of this type to yield robust results. following previous implementations @xcite , we use a zero-shot setting for hellaswag and arc where only the questions and choices are given in the input, and a 5-shot setting for mmlu and c-eval where 5 demonstrations are given in the prompts. we employ the third party llms evaluation platform opencompass (opencompass, 2023) in our experiments to provide in-context demonstrations, prompts, and metrics computing. we use perplexity to obtain the inference result, i.e., taking the choice with the lowest perplexity as the predicted answer. the results are presented in however, on mmlu tasks we observe no consistent enhancement across models. we also find that larger language models appear more capable of exploiting data contamination to achieve better performance. for instance, llama-2 70b displays increased metrics on most contaminated subsets. in contrast, the 13b llama-2 only outperforms on contaminated arc. in addition, llama-2 70b achieves a larger advantage on contaminated sets (5%) compared to 3% inflation of the 7b variant. this could be due to the more powerful memorisation capacity in larger language models @xcite . finally, we find that input-only contamination does not lead to inflation of metrics. this suggests that contamination has little effect when it does not give away the answer. surprisingly, we also observe that models demonstrate significantly worse performance on the input-only contaminated set. this may be because the absence of their labels online suggests that no one has provided a solution for these test samples, because they are inherently more challenging. input-and-label contamination, on the contrary, often leads to more notable accuracy increases, making it the key issue to address for data contamination. non-english benchmark. in what is the threshold of overlap for a test example to affect model prediction? we illustrate how the meteor score, which measures sentence similarity, correlates with model performance on test samples. the meteor metric measures the similarity between two sentences. for instance, a test sample with a meteor score of 0.8 indicates high equivalence between that test case and sentences in training data. in figure 4 , we group test samples by meteor score and present the accuracy achieved on those groups by llama-2 70b across four benchmarks. on arc, hel-laswag, and c-eval, a general upwards accuracy trend emerges as meteor rises, indicating that models attain higher metrics when more verbatim overlapping samples exist in the training data. in essence, substantial text duplication enables exploitation through memorisation, inflating model scores. ## conclusion this paper conducted an extensive data contamination analysis for popular large language models on six multi-choice qa benchmarks. we identified varying levels of test set contamination, ranging from 1% to 47% across benchmarks. we also found that data contamination can lead to increased metrics: data contamination in arc and hellaswag generally allows models to achieve significantly higher accuracy, but contamination in mmlu has less of an impact on model's performance, although it did increase performance on a specific sub-set of mmlu. our findings offer a transparent perspective on data contamination, emphasising its significance as an urgent issue within the evaluation community.