Data Contamination: From Memorization to Exploitation (2022)

Abstract: Pretrained language models are typically trained on massive web-based datasets, which are often “contaminated” with downstream test sets. It is not clear to what extent models exploit the contaminated data for downstream tasks. We present a principled method to study this question. We pretrain BERT models on joint corpora of Wikipedia and labeled downstream datasets, and fine-tune them on the relevant task. Comparing performance between samples seen and unseen during pretraining enables us to define and quantify levels of memorization and exploitation. Experiments with two models and three downstream tasks show that exploitation exists in some cases, but in others the models memorize the contaminated data, but do not exploit it. We show that these two measures are affected by different factors such as the number of duplications of the contaminated data and the model size. Our results highlight the importance of analyzing massive web-scale datasets to verify that progress in NLP is obtained by better language understanding and not better data exploitation.

## introduction pretrained language models are getting bigger and so does their capacity to memorize data from the training phase @xcite . a rising concern regarding these models is "data contamination"-when downstream test sets find their way into the pretrain corpus. for instance, @xcite examined five benchmarks and found that all had some level of contamination in the c4 corpus @xcite ; @xcite flagged over 90% of gpt-3's downstream datasets as contaminated. eliminating this phenomenon is challenging, as the size of the pretrain corpora makes studying them difficult @xcite , and even deduplication is not straightforward @xcite . it figure 1 : we pretrain bert on wikipedia along with both the labeled training and test sets (denoted seen) of a downstream task (e.g., sst). then, we fine-tune this model on the same training set for that task. we compare performance between samples seen and unseen during pretraining to quantify levels of memorization and exploitation of labels seen in pretraining. we apply our methodology to bert-base and large, and experiment with three english text classification and nli datasets. we show that exploitation exists, and is affected by various factors, such as the number of times the model encounters the contamination, the model size, and the amount of wikipedia data. interestingly, we show that memorization does not guarantee exploitation, and that factors such as the position of the contaminated data in the pretrain corpus and the learning rate affect these two measures. we conclude that labels seen during pretraining can be exploited in downstream tasks and urge others to continue developing better methods to study large-scale datasets. as far as we know, our work is the first work to study the level of exploitation in a controlled manner. ## our method: assessing the effect of contamination on task performance to study the effect of data contamination on downstream task performance, we take a controlled approach to identify and isolate factors that affect this phenomenon. we assume that test instances appear in the pretrain corpus with their gold labels, 1 and that the labeled training data is also found in the pretrain corpus. 2 we describe our approach below. we pretrain an mlm model on a general corpus combined with a downstream task corpus, containing labeled training and test examples. we split the test set into two, adding one part to the pretrain corpus (denoted seen), leaving the other unobserved during pretraining (unseen). for example, we add the following sst-2 instance @xcite : we then fine-tune the model on the same labeled training set, and compare performance on the seen and unseen test sets. as both test sets are drawn randomly from the same distribution, differences in performance indicate that the model exploits the labeled examples observed during pretraining (fig. 1 ). this controlled manipulation allows us to define two measures of contamination: mem is a simple measure of explicit memorization. we consider the mlm task of assigning the 1 our focus is on classification tasks, but our method can similarly be applied to other tasks, e.g., question answering. 2 we recognize that these assumptions might not always hold; e.g., the data might appear unlabeled. such cases, while interesting, are beyond the scope of this paper. pretraining design choices simulating language model pretraining under an academic budget is not an easy task. to enable direct comparisons between different factors, we pretrain medium-sized models (bert-{base,large}) on relatively small corpora (up to 600m tokens). we recognize that some of the results in this paper may not generalize to larger models, trained on more data. however, as data contamination is a prominent problem, we believe it is important to study its effects under lab conditions. we hope to encourage other research groups to apply our method at larger scales. ## which factors affect exploitation? we study the extent to which pretrained models can memorize and exploit labels of downstream tasks seen during pretraining, and the factors that affect this phenomenon. we start by examining how many times a model should see the contaminated data in order to be able to exploit it. we pretrain bert-base on mlm using a combined corpus of english wikipedia (60m tokens), and increasing numbers of sst-5 copies @xcite . to facilitate the large number of experiments in this paper, we randomly downsample sst-5 to subsets of 1,000 training, seen and unseen instances. we train for one epoch, due to the practical difference between the number of times the task data appears in the corpus and the number of times the model sees it. for example, if a contaminated instance appears in the corpus once, but the model is trained for 50 epochs, then in practice the model encounters the contaminated instance 50 times during training. further exploration of the difference between these two notions is found in app. a. see app. d for experimental details. we describe our results below. exploitation grows with contaminated data duplicates both mem and expl levels increase in proportion to the contaminated data, reaching 60% mem and almost 40% expl when it appears 200 times (fig. 2 , left ). this suggests a direct connection between both mem and expl and the number of times the model sees these labels. this finding is consistent with several concurrent works, which show similar connections in gpt-based models. these works study the impact of duplication of training sequence on regeneration of the sequence @xcite , and the effect on few-shot numerical reasoning @xcite . one explanation for this phenomenon is the increase in the expected number of times labels are masked during pretraining. 5 to check this, we pretrain bert-base with 100 copies of sst-5 and varying probabilities of masking the label. our results (fig. 2 , right) show that the higher this probability, the higher mem and expl values. these results motivate works on deduplication @xcite , especially considering that casual language models (e.g., gpt; @xcite are trained using next token prediction objective, and so every word in its turn is masked. in the following, we fix the number of contaminated data copies to 100 and modify other conditions-the size of the wikipedia data and the model size (base/large). we also experiment with two additional downstream tasks: sst-2 and snli @xcite . all other experimental details remain the same. fig. 3 shows our results. ## discussion and conclusion we presented a method for studying the extent to which data contamination affects downstream fine-tuning performance. our method allows to quantify the explicit memorization of labels from 9 https://github.com/schwartz-lab-nlp/ data_contamination the pretraining phase and their exploitation in finetuning. recent years have seen improvements in prompt-based methods for zero-and few-shot learning @xcite @xcite . these works argue that masked language models have an inherent capability to perform classification tasks by reformulating them as fill-in-the-blanks problems. we have shown that given that the language model has seen the gold label, it is able to memorize and retrieve that label under some conditions. prompt-tuning methods, which learn discrete prompts @xcite or continuous ones @xcite , might latch on to the memorized labels, and further amplify this phenomenon. this further highlights the importance of quantifying and mitigating data contamination. zexuan zhong, dan friedman, and danqi chen. 2021. factual probing is [mask]: learning vs. learning to recall. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 5017-5033, online. association for computational linguistics. as noted in sec. 3, the number of times an instance appears in the corpus is a different notion than the number of times the model sees it during training. the latter also takes into account the number of training epochs. for example, if an instance appears in the corpus once, but the model is trained for 50 epochs, than practically the model sees it 50 epochs appears seen expl 1 10 10 2.07% 5 10 50 6.87% we do so by performing second-stage-pretraining @xcite on a combined corpus of wikipedia and 10 copies of sst-5. we train one model for one epoch, and the other for 5 epochs. results are shown in tab. 1. in the second experiment we fix the number of times the model sees sst-5, and change the number of times it appears in the corpus. we do so by performing second-stage-pretraining for one epoch on a combined corpus of wikipedia and changing number of copies of sst-5. results are shown in tab. 2. we observe that expl levels of the models which saw the contamination 50 times are rather similar. on the contrary, expl levels of the model which saw the data 10 times is 5% lower. these results indicate the number of times contamination is seen during training have great influence on expl. in the main experiments presented in this paper we train for one epoch in order to eliminate the difference between the two notion (appears vs. seen).