NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023)

Abstract: In this position paper we argue that the classical evaluation on Natural Language Processing (NLP) tasks using annotated benchmarks is in trouble. The worst kind of data contamination happens when a Large Language Model (LLM) is trained on the test split of a benchmark, and then evaluated in the same benchmark. The extent of the problem is unknown, as it is not straightforward to measure. Contamination causes an overestimation of the performance of a contaminated model in a target benchmark and associated task with respect to their non-contaminated counterparts. The consequences can be very harmful, with wrong scientific conclusions being published while other correct ones are discarded. This position paper defines different levels of data contamination and argues for a community effort, including the development of automatic and semi-automatic measures to detect when data from a benchmark was exposed to a model, and suggestions for flagging papers with conclusions that are compromised by data contamination.

## introduction at the core of nlp as a discipline, there is rigorous evaluation on different tasks. the experimental protocols involve strict control over the data, especially test data, which needs to be totally unseen during development, but also over training and development data. this is essential to assess the performance of a model in zero-shot, few-shot, or fully supervised settings. since fine-tuning and prompting of large language models (llms) became commonplace @xcite it has been increasingly difficult to enforce those strict protocols. pretraining llms is expensive, and therefore, most of the time, researchers use llms trained by thirdparty entities @xcite , which are agnostic to the target tasks where those llms are going to be used. with the growing scale of llms @xcite the need for data has been solved by crawling the internet, reaching trillions of tokens @xcite , and making it very hard to know whether a specific benchmark was used to train the llm. this is applicable to all models, even if they document the source of the data at a high level, but especially for closed models with no or insufficient documentation. data contamination has two consequences. the first one is that the performance of an llm when evaluated on a benchmark it already processed during pre-training will be overestimated, causing it to be preferred with respect to other llms. this affects the comparative assessment of the quality of llms. the second is that papers proposing scientific hypotheses on certain nlp tasks could be using contaminated llms, and thus make wrong claims about their hypotheses, and invalidate alternative hypotheses that could be true. this second consequence has an enormous negative impact on our field and is our main focus. there are several measures that the community could take. a possible solution would be to avoid all research involving datasets which include published test data, and focus on datasets where the test data labels are not public. this solution will severely affect the number of nlp tasks for which benchmarks exist, at least until new benchmarks that avoid data leakage are produced. @xcite presents preventative strategies to avoid contamination in the future. in this position paper, we propose a complementary line of action which seeks to measure and document data contamination cases, specifying llm, benchmark and evidence supporting contamination. this solution involves a registry of contamination cases 1 , collaborative manual work and research on automatic approaches. in addition, conferences should devise mechanisms to ensure that papers don't include conclusions involving contamination, and to flag past work where contamination has been discovered after publication. the paper starts by introducing background, followed by a definition of data contamination, contamination at different steps, methods to measure data contamination and a call for action. ## background detection of contamination cases has been traditionally done by directly analyzing the training data @xcite , but the current scale of the pre-training data makes it difficult @xcite . without proper documentation and search tools like roots @xcite it is very difficult for any researcher to actually know whether their datasets are compromised on a given model. more recently, this task became even harder, as the best-performing llms are deployed as products, and therefore, their training corpora are kept secret. in this case, it has been shown that the high memorization abilities of llms can be used to generate portions of the training texts @xcite . using this memorization property, @xcite show that chatgpt generates portions of popular nlp benchmarks. furthermore, llms memorization has been studied on data-leakage scenarios @xcite . regarding data contamination cases, @xcite exposed that the c4 corpus @xcite , a corpus used to pre-train several llms such as t5 @xcite , contained the test splits of several benchmarks that were crawled from github. moreover, brown et al. (2020) acknowledged a bug in their filtering script that caused the contamination of several benchmarks during the gpt-3 training. furthermore, openai (2023) stated that parts of the bigbench @xcite benchmark were inadvertently mixed into the training set, enough to stop them from evaluating the model on it. they also mention that they included parts of the training sets of math @xcite and gsm-8k @xcite as training data to improve mathematical reasoning @xcite . therefore, the performance results reported for gsm-8k cannot be taken as zero-shot results when compared to other models. recently, @xcite reported that several benchmarks have already been com-promised in chatgpt, including the popular conll2003 @xcite . there are several preprints that evaluate chatgpt on conll03 @xcite @xcite and at least one conference paper published on acl 2023 that evaluates @xcite and @xcite on the same benchmark @xcite . appendix a shows evidence for data contamination for those llms, and casts doubts on the conclusions of those papers. ## defining data contamination in general, data contamination refers to any breach in the strict control of datasets required by the experimental protocol. in this paper, we focus on the specific case where a llm has processed the evaluation benchmark during its pre-training. however, different types of contamination exist and each of them has different implications. in this section, we present three types of contamination: guideline, text and annotation. guideline contamination happens when the annotation guidelines for a specific dataset are seen by the model. usually, for specialized annotations, highly detailed guidelines are required. the guidelines can usually be publicly found on the internet, even for datasets that are not public or require buying a license for their use, ace05 @xcite for example. the more details the guidelines have the more information and examples they provide. a model aware of the guidelines for a specific task or dataset has advantages over a model without such information. we should consider the guideline contamination, especially on zero and few-shot evaluations. raw text contamination happens when the original text (previous to annotation) is seen by the model. some examples of this type of contamination are the datasets based on wikipedia texts. wikipedia is commonly used as a source of pretraining data, but, it is also a frequent source of text to create new datasets. multiconer 2 @xcite , a named entity recognition dataset based on wikipedia links and wikidata information, is an example of this phenomenon. models that have already seen wikipedia in its original form (including the markup annotations) have more information to better identify a part of the annotations (the entity boundaries) of the dataset. as pointed out by @xcite , other datasets built from the web such as imdb @xcite and cnn/dailymail @xcite can be also compromised. this kind of contamination should be taken into account when developing automatically annotated datasets. annotation contamination happens when the annotations (labels) of the target benchmark are exposed to the model during training. depending on the splits of the benchmark that have been exposed, we can have the following cases: (1) when the evaluation split is involved, the experiment is completely invalidated. this is the most harmful level of contamination. (2) when the train or development splits are involved, this would not affect comparisons with other models that have been developed using those same splits, but it does invalidate conclusions claiming zero-shot or few-shot performance. ## contamination on different steps currently, the standard procedure to train and deploy language models has three main steps: pretraining a language model, fine-tuning the model to follow instructions and/or align with human feedback; and an iterative improvement step after deployment. data contamination does not only occur in the pre-training step of llms, but can occur later in the training pipeline. ## measuring data contamination for the reasons we already mentioned, it is necessary to measure the existent data contamination cases and to document relevant contamination evidence. in order to achieve this goal, we differentiate two cases. in the first case, we would have open models where there is public access to all the training data, including text used in pre-training, but also, if the llm was trained on them, instruction tuning datasets and deployment datasets. in the second case, we would have closed models for which there is no access to training data. ## call for action we want to encourage the nlp community to: (1) develop auto-or semi-automatic measures to detect when data from a benchmark was exposed to a model; (2) build a registry of data contamination cases, including the evidence for the contamination; as the problem affects our entire field, we also want to encourage the community to participate in workshops related to this topic, as for example, the 1st workshop on data contamination 2 . we think that developing the ideas that will arise from this community will play an important role in future nlp evaluations. ## limitations in this paper, we address the problem of data contamination that occurs when evaluating llms on standard academic benchmarks. however, we are aware that there could exist other issues in current evaluations, but, they are out of the scope of this position paper. related to our proposed solutions, we are aware that these are early-stage solutions and that the proposed effort is really challenging, therefore we call for further discussion and research on topics related to this issue.