Confounders in Instance Variation for the Analysis of Data Contamination (2024)

Abstract: Test contamination is a serious problem for the evaluation of large language models (LLMs) because it leads to the overestimation of their performance and a quick saturation of benchmarks, even before the actual capability is achieved. One strategy to address this issue is the (adversarial) generation of variations, by including different exemplars and different rephrasings of the questions. However, these two interventions can lead to instances that can be more difficult (accumulating on the expected loss of performance by partly removing the contamination) but also to instances that can be less difficult (cancelling the expected loss of performance), which would make contamination undetectable. Understanding these two phenomena in terms of instance difficulty is critical to determine and measure contamination. In this paper we conduct a comprehensive analysis of these two interventions on an addition task with fine-tuned LLAMA-2 models.

## introduction large language models (llms) @xcite have transformed natural language processing, but face evaluation challenges, especially with publicly available benchmarks. a key issue is data contamination @xcite , where training data contains test instances. a model trained on this data has seen part of the test set, which can have important effects in the evaluation of the model, leading to inflated performance measures @xcite . recently, it has been seen that the actual issue of data contamination could be more predominant, as it is not only present by exact copies of the test instances in the training data. studies such as the one by @xcite show that even rephrased or translated test instances in training data can improve performance, indicating potential contamination. however, the effect of the difficulty of these template 1 can you please add [term 1 ] and [term 2 ] together? 2 find the sum of [term 1 ] and [term 2 ]. 3 add up two numbers: [term 1 ] and [term 2 ]. 4 please work out the total of [term 1 ] and [term 2 ]. 5 please determine the numeric sum of [term 1 ] and [term 2 ]. 6 proceed to identify the aggregated total of the numbers [term 1 ] and [term 2 ]. 7 perform an addition operation on the numerical values [term 1 ] and [term 2 ]. the main contributions are: the following sections detail our experimental design, methodology, and results. ## background several methods have been used to address data contamination. prior to big tech companies close sourced their models and the training data, a common approach was trying to look for evaluation instances in the training data. string matching and embedding similarity are two techniques that have been commonly used for this purpose. ope-nai used 8-gram matching of test instances and training dataset for gpt-2 model @xcite . for gpt-3 @xcite the same approach has been taken and all data points from the evaluation sets that had a 13-gram collision in the pre-training common crawl (c4) dataset were removed to tackle contamination. as contamination can involve minor variations of the examples, calculating cosine similarity between embeddings of test and training items can also be used for finding cases in which the test item has been rephrased or expressed in a different language @xcite . but string matching and even embedding matching are not able detect rephrased test items effectively in general @xcite . more sophisticated and effective techniques employ embedding similarity search to identify the top-k samples similar to a given test sample and then prompting a powerful llm such as gpt-4 to determine if any of the k samples are too similar to the test case. for closed source models where no information regarding the training set is provided, none of the above mentioned methods are applicable. introducing new contamination-free benchmarks such as lasteval @xcite , wikimia @xcite , kieval @xcite , livecodebench @xcite , termite @xcite might seem a reliable solution for the problem, but as @xcite mentioned, these new benchmarks can get contaminated as soon as they are publicly available or even just when used for evaluating closed source models by the creators of the benchmark themselves for the first time. in addition, building a high quality benchmark is a time consuming process and can not be done overnight. consequently, the idea of continuously generating new variation has taken ground. clean-eval @xcite intends to 'purify' current benchmarks by rephrasing the test items. while a drop in the performance of llms on the rephrased data points is considered as a sign of decontamination, the role of difficulty has been neglected in their analysis. ## methodology data contamination occurs when instances from the test set are found in the train set of ai models. for example, if a model is tested on the question "what is 123 + 456?" but has seen the same question (and answer) during training, it might simply recall the answer rather than 'compute' it again. even if rephrased forms of test items like "calculate the sum of 123 and 456" or "what do you get when you add 456 to 123?" exist in the training data, the evaluation is still compromised. these rephrased forms can inadvertently aid the model, causing an overestimation of its true capabilities. on the other hand, testing on the rephrased form of original test items is suggested by the researchers to mitigate the contamination problem. yet to the best of our knowledge, the role of difficulty of original test items and their variants has not been studied. also, what matters more, the change in the exemplar or rephrasing the template? for instance, solving "find the result of 9876 + 54321" might naturally be harder than "compute 12 + 34," regardless of rephrasing. dataset construction to explore these considerations, we designed a dataset of addition problems varying in complexity. specifically, we generated 1,000 addition pairs (each different pair is an exemplar) for numbers ranging from five to fourteen digits. each exemplar's intrinsic difficulty was determined by the sum of the number of digits of the addends. for example, the intrinsic difficulty of "829 + 4531" is 7. instance templates to produce varied instances, we asked gpt-4 to rephrase each addition problem in ten different ways. after excluding three ambiguous rephrasings, we used the remaining seven clear templates (see ## results as shown in previous studies, llms are sensitive to prompts, i.e., the way that the request is formulated. figure 1 shows that even for a simple task such as addition, rephrasing the question influences model performance. we can observe that template 2 in average is the easiest and template 7 is the most difficult version of rephrasing addition among our 7 templates. consequently, as rephrasing a test item can change its difficulty level, this should be considered when this approach -rephrasing test items-is taken to address contamination. a lower performance of models on the rephrased test items might be simply due to the higher difficulty level of them and may not be a sign of their purity. figure 3 demonstrate this effect more clearly. as it can be seen, there are cases that the performance of the model for one or more templates when tested on the non-contaminated validation 1 2 3 4 5 6 7 template 0.0 0.2 0.4 0.6 0.8 average performance without fine tuning finetuned-tested on validation-set finetuned-tested on the test-set figure 4: llama2 7b model average performance comparison when tested on the non-contaminated validation-set, the contaminated test-set, train-set, and test-set excluding training instances set is higher than some templates when tested on the contaminated test-set. this phenomenon can be seen for 5-digit, 6-digit, 7-digit, and 11-digit additions (figure 3 clearly demonstrates this when the solid lines cross the dashed lines). figure 5 shows the performance of the fine-tuned models. it is noticeable that performance of the fine-tuned model on templates that have not been used in fine-tuning is affected differently when finetuned on templates 2 or 7. humans are not highly sensitive to the way a question is formulated and in those cases that they are, it is expected that if they learn the hardest rephrased form of a task, they can easily cover the easier variations too. figure 6 shows that this does not hold for llms. finetuning on the easiest template shows better performance on 3 other templates (templates 1, 3, 4 and 5) however, for the case which we fine-tuning on the hardest template the resulting model performs better only on one other template (number 6). ## conclusions our study explored the effects of data contamination and instance variation on the performance of large language models (llms), specifically using fine-tuned llama-2 models on an addition task. key findings from this investigation answered our research questions and highlighted important insights. regarding template difficulty (rq1), we show that the performance of llms is significantly influenced by the template or rephrasing of test items. easier templates, such as template 2, consistently yielded higher performance, while more complex templates, like template 7, posed greater challenges. when analysing the impact of finetuning on contaminated data (rq2), we observed that performance on non-contaminated templates varies after fine-tuning. contrary to expectations, some templates showed higher performance on noncontaminated validation sets compared to contaminated test sets, indicating that contamination and difficulty levels are deeply intertwined. finally, focusing on template diversity and contamination (rq3), we saw that fine-tuning on an easier templates improved performance on other templates more consistently than fine-tuning on harder templates. this challenges the assumption that learning from more difficult examples would generalize better to simpler variations. additionally, lower template diversity in the training set amplified the contamination effect, suggesting that less diverse contamination scenarios have a stronger influence on model performance. these findings highlight the relevance of considering both the difficulty and diversity of rephrased instances when evaluating llm performance. in particular, the change of rephrasing and exemplars can have confounding effects masking contamination or suggesting contamination where there is not. our results suggest that addressing data contamination effectively requires more nuanced strategies that accommodate these factors. for future work, we are investigating the effects of contamination and instance variation across more complex and diverse nlp tasks.