Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation (2024)

Abstract: Data contamination has garnered increased attention in the era of Large language models (LLMs) due to the reliance on extensive internet-derived training corpora. The issue of training corpus overlap with evaluation benchmarks—referred to as contamination—has been the focus of significant recent research. This body of work aims to identify contamination, understand its impacts, and explore mitigation strategies from diverse perspectives. However, comprehensive studies that provide a clear pathway from foundational concepts to advanced insights are lacking in this nascent field. Therefore, we present the first survey in the field of data contamination. We begin by examining the effects of data contamination across various stages and forms. We then provide a detailed analysis of current contamination detection methods, categorizing them to highlight their focus, assumptions, strengths, and limitations. We also discuss mitigation strategies, offering a clear guide for future research. This survey serves as a succinct overview of the most recent advancements in data contamination research, providing a straightforward guide for the benefit of future research endeavors.

## introduction data contamination refers to the accidental or deliberate inclusion of evaluation or benchmark data in the training phase of language models, resulting in artificially high benchmark scores @xcite . this issue, while longstanding-stemming from the foundational ml principle of separating training and test sets-has garnered increased attention with the advent of large language models (llms). these models are trained on vast corpora sourced from the web @xcite , heightening the risk that training data may inadvertently encompass instances from evaluation benchmarks @xcite ; chowdhery * equal contribution. evaluation benchmark (pre)training corpora evaluation benchmark (pre) training corpora distribution distance clean evaluation contaminated evaluation 1. tb-scale token size 2. (commonly) internetbased source 1. small-scale token size 2. (commonly) internetbased source rq1: what is the impact of data contamination? rq2: how to detect data contamination? rq3: what is mitigation stragetry towards it? figure 1: basic illustration of data contamination and the research questions related to it. clean evaluation is defined as having no overlap between the pretraining corpora and the benchmarks, and contaminated evaluation is defined as significant overlap between it. @xcite . such contamination of evaluation benchmarks can obscure the true generalization performance of llms, as it might artificially inflate benchmark scores by testing the models' ability to "memorize" and "recall" rather than "reason" or "generalize". given the increasing concerns regarding potential contamination of evaluation benchmarks and its broader impact on downstream task performance recently, numerous studies have aimed at identifying and mitigating data contamination in these benchmarks, and understanding its impact on our perception of model capabilities. research on data contamination could be broadly categorized into two main areas: (i) investigations of models trained with open-source data, and (ii) studies relevant to models developed using proprietary data. generally, having access to training data, or the lack thereof, has a profound influence on modern contamination research. in this paper, we present the very first comprehensive analysis of the growing field of data contamination detection and mitigation. our objective is to delve into the downstream impacts of data contamination, investigate existing methods for detecting data contamination, and discuss a range of mitigation strategies. the paper is structured as outlined in figure 1 . we start by establishing the background of data contamination ( §2) and discussing the effect of contamination ( §3). following this, we provide a detailed analysis of current methods for detecting data contamination ( §4). we categorize these methods and critically examine the assumptions each relies on, highlighting their the prerequisites and limitation for their application. subsequently, we explore strategies for mitigating data contamination ( §5), tackling potential hurdles and proposing avenues for future investigations in this domain. together with concurrent studies on data contamination @xcite , this paper aims to furnish nlp researchers with an in-depth, systematic understanding of data contamination issues, thereby making a significant contribution to enhancing the integrity of evaluations in the field 1 . 1 the related materials at https://github.com/ yale-nlp/lm-contamination-survey . ## background to provide a comprehensive understanding of data contamination, this section delves into its definition, the urgency of addressing it, and its implications across different types of language models. what is data contamination? data contamination occurs when benchmark or test set data are inadvertently included in the training phase. this issue is particularly relevant when evaluating llms that have been partially trained with a test set from a benchmark, potentially leading to an inflated performance score. this phenomenon, known as data contamination, is critical for ensuring fairness and unbiased evaluation in modern llms. ## impacts of contamination the contamination effect refers to the extent to which a model, exposed to contaminated data during its training phase, is influenced by this data in its performance on downstream tasks. research in this area typically involves selecting a base model and a fixed pretraining corpus, while varying mixture of contaminated data @xcite . this approach allows for observing how changes in the data mix affect downstream task performance. additionally, this area of research is often connected with evaluating the models' ability to memorize information and recall their parametric knowledge @xcite @xcite . ## detecting data contamination in this section, we discuss various methods for detecting data contamination. we begin with the traditional retrieval-based method, which primarily employs n-gram tokenization and string-matching for detection. this approach is often documented in technical reports by proprietary companies. subsequently, we introduce several modern methods predominantly developed by the academic community. these methods typically detect contamination indirectly and implicitly, without requiring full ac-cess to the training corpora. ## mitigating data contamination without specific mitigation strategies, the development of new benchmarks-often released publicly on the internet-does not resolve contamination issues, as newer models can access this data. consequently, several studies have proposed mitigation approaches to address this problem. in this section, we will introduce these strategies from the perspectives of benchmark construction, updating, encryption, and protection. @xcite propose to construct evaluation benchmarks from the most recent texts, thus minimizing the risk of overlap with the pre-training corpora. @xcite introduce a dynamic evaluation protocol that utilizes directed acyclic graphs to generate eval-uation samples of varying complexities, aiming to address the static and potentially contaminated nature of existing benchmarks. besides, @xcite provide clean-eval, which utilizes llms to paraphrase and back-translate contaminated data, creating a set of expressions that convey the same meaning in varied forms. this process generates a candidate set from which low-quality samples are filtered out using a semantic detector. the final selection of the best candidate from this refined set is based on the bleurt @xcite score, ensuring the chosen expression is semantically similar to the original data but articulated differently. furthermore, @xcite also suggest providing a diverse set of prompts for testing, which offers a dynamic evaluation to mitigate data contamination. @xcite suggests that test data released to the public should be safeguarded through encryption using a public key, and the distribution of derivatives should be strictly prohibited by the licensing agreement. to implement this, the recommended approach is toencrypt the test data before uploading it. this can be efficiently done by compressing the data into a password-secured archive. @xcite and @xcite emphasize the critical need to safeguard the ground truth labels of test datasets. these labels can inadvertently be exploited during the training phase, or even intentionally after being rephrased. providing both the question and its context is an effective strategy to prevent such deliberate contamination. ## discussion and future directions besides addressing the impact, detection, and mitigation of previously introduced data contamination, this section will also explore the topic at a higher level. we aim to offer more insights into the current challenges, the necessity, and the robustness of detecting data contamination methods. we will also discuss how these concepts can be applied in more realistic settings. additionally, we will consider data contamination as an overarching research direction and explore potential future pathways for this field. ## conclusion in this paper, we present an extensive and meticulously organized survey on the topic of data contamination in large language models. we start by laying the groundwork with a discussion on the effect of contamination, setting the stage for a deeper examination of various data contamination detection methods. we critically analyze the assumptions underlying these methods, highlighting their limitations and the prerequisites for their application. subsequently, we explore strategies for mitigating data contamination, addressing potential challenges and suggesting directions for future research in this area. our goal is to provide a comprehensive guide for nlp researchers seeking a systematic understanding of data contamination. we also aim to underscore the critical importance of this field, advocating for increased attention due to its pressing relevance.