A Taxonomy for Data Contamination in Large Language Models (2024)

Abstract: Large language models pretrained on extensive web corpora demonstrate remarkable performance across a wide range of downstream tasks. However, a growing concern is data contamination, where evaluation datasets may unintentionally be contained in the pretraining corpus, inflating model performance. Decontamination, the process of detecting and removing such data, is a potential solution; yet these contaminants may originate from altered versions of the test set, evading detection during decontamination. How different types of contamination impact the performance of language models on downstream tasks is not fully understood. We present a taxonomy that categorizes the various types of contamination encountered by LLMs during the pretraining phase and identify which types pose the highest risk. We analyze the impact of contamination on two key NLP tasks—summarization and question answering—revealing how different types of contamination influence task performance during evaluation.

## introduction advancements in machine learning have traditionally relied on benchmark datasets to evaluate and compare model performance @xcite . with the surge of large language models (llms) in recent years, these benchmarks are now leveraged to showcase remarkable abilities across diverse tasks. however, the shelf life of benchmarks is incredibly low, with @xcite demonstrating that newer models with updated training cutoff dates are iteratively rendering existing benchmarks stale. the presence of internet-sourced data in both pretraining and evaluation datasets increases the risk of data contamination @xcite and challenges the notion of fair evaluation for models pretrained on massive corpora. both gpt-3 and c4 training corpora were found to contain test data for several benchmarks @xcite @xcite , raising serious concerns about the validity of evaluation scores for many pretrained models @xcite . the research community lacks consensus on best practices for data contamination, and different works define contamination in subtly different ways. without standardization of terminology, it is difficult to develop best practices for contamination-or even to characterize the problem at all. to address this gap, we suggest a formal definition of contamination and taxonomize subtypes of contamination ( § 2). we map prior work on both the detection and impact of contamination into this taxonomy, revealing several understudied forms of contamination ( § 2.3). we also measure the impact of different types of contamination on downstream summarization ( § 4) and qa ( § 5) performance through continued pretraining experiments assessing indirect/approximate test set contamination effects. our findings reveal that for gpt-2 large models, it is often the case that having in-domain data present during training is as beneficial as having the test data present during training. moreover, we observe that certain contamination types exhibit task-dependent effects on evaluation performance, further complicating decontamination best practices. our findings enable recommendations for identifying and mitigating problematic contamination during llm development to ensure reliable evaluations ( § 7). ## taxonomy consider a model m : @xmath0, given an input of some type @xmath1, outputs text @xmath2. while x can be of any format, we will restrict ourselves to cases where y is in the space of the natural language (y ⊆ σ * for some alphabet in σ). let d be the test set, consisting of |d| examples ⟨x i , y i ⟩. ## methodology in all our experiments, we employ gpt-2 large @xcite . 3 this will be referred to as the initial model. since the pretraining corpus for gpt-2 is not publicly accessible, there is a chance that these learned weights of gpt-2 might be contaminated. consequently, the outcomes of our experiments serve as a conservative estimate or lower bound on the effects of data contamination. for each of our datasets, we create train/in-domain/test splits of equal size, aiming to establish a fair and comparable evaluation environment. to disentangle the effects of exposure to test data during pretraining from those of prior task understanding, we constructed an in-domain data split, allowing us to train models on task-relevant but uncontaminated data for comparison against the various contaminated settings. to partially mitigate the potential recency bias from continued pretraining, we incorporate an additional 10,000 samples of open ai's webtext @xcite into the continued pretraining data. during continued pretraining, we use a blocksize of 1024 tokens with a batchsize of 1. for finetuning, the training data is seen sample by sample. to obtain deterministic results during our experiments, we set the temperature to zero and capped the maximum completion length at 200 tokens. ## case study: summarization for this case study, we use the following summarization datasets: xsum @xcite , samsum @xcite , and cnn/daily mail @xcite . we explore 5 contamination settings: ## case study: question answering for this case study, we consider open-ended qa with squad @xcite and multiple-choice qa with the children's book test (cbt) @xcite . we explore 6 contamination settings: ## analysis unsurprisingly, the cheating and verbatim contamination settings consistently outperform the baseline across both tasks. the in-domain settings' consistent outperformance of the baseline underscores the advantages of exposure to related samples during pretraining @xcite . far more concerning is that several approximate contamination settings outperform both the base-line and their respective in-domain settings, suggesting that the model in these settings benefits not only from seeing in-domain text but from unfairly leveraging prior knowledge of the test examples. in particular, the noised setting, which is generally not detectable with existing decontamination methods, produces scores inflated over baseline in all datasets, and scores more than one standard deviation above its corresponding in-domain setting in several datasets. the masked setting generally performs around or worse than the baseline, possibly due to the more extreme formatting mismatch between this data and the test data. we expect that the masked setting may be encountered in the wild if a file of outputs for the dataset is in the pretraining data; the limited impact of this contamination on downstream performance is thus good news, though more investigation would be necessary to conclusively say masked contamination is not a concern. for many of the contaminated settings and their corresponding in-domain settings, the effect of approximate contamination is not greater than affect of in-domain data seen during pretaining. however, research has shown that memorization in llms significantly grows as the size of the model increases @xcite . the number of times a sample has been duplicated in the pretraining corpora has also been shown to increase a model's memorization capabilities @xcite . some behavior is task-or dataset-specific, emphasizing that there is no one-size-fits-all approach to data curation: the importance of removing each type of contamination from the pretraining corpus is at least partially linked to the specific task's formatting. however, some types of approximate contamination do lead to inflated scores, emphasizing that considering a more broad definition of contamination when de-contaminating pretraining corpora is a worthwhile endeavor. ## conclusion our analysis highlights the importance of data format, with models performing better when pretraining data matches the evaluation format. we also observe task-specific effects, with certain contamination methods benefiting particular tasks more than others. additionally, we find that some latestage pretraining contamination can actually be unhelpful to downstream performance, if it occurs in a substantially different format from the downstream task. our findings underscore gaps in current decontamination practices, which primarily focus on full-dataset-level contamination and are often unable to detect approximate or noisy contamination. we demonstrate that different types of contamination can have variable effects on model performance, highlighting the need for careful consideration during training and evaluation. with the creation of our taxonomy, we hope to promote standardization regarding the definition and categories of contamination within the research community, facilitating clear communication and collaboration, while also enabling precise detection and mitigation of contamination in pretraining data. we recommend researchers decontaminating pretraining corpora for llms prioritize developing techniques that address noisy evaluation data, while also ensuring rigorous scrutiny to prevent any shuffled or interleaved evaluation data from inadvertently persisting in the pretraining data. it is not enough to merely remove instances of the full test dataset in the pretraining corpus; fragments or noised versions of the test set can also inflate performance. we hope our work inspires future work on detecting and mitigating specific types of contamination.