CLEAN – EVAL : Clean Evaluation on Contaminated Large Language Models (2024)

Abstract: We are currently in an era of fierce competition among various large language models (LLMs), continuously pushing the boundaries of benchmark performance. However, genuinely assessing the capabilities of these LLMs has become a challenging and critical issue due to potential data contamination. In this paper, we propose a novel and valuable method, Clean-Eval, which mitigates the issue of data contamination and evaluates the LLMs more cleanly. Clean-Eval employs a neural-based model to paraphrase and back-translate the contaminated data into a candidate set, generating expressions with the same meaning but in different surface forms. A semantic detector is then used to filter those generated low-quality samples to narrow down this candidate set. Candidates with moderate BLEURT scores against the original samples are selected as the final evaluation set. According to human assessment, this set is almost semantically equivalent to the original contamination set but expressed differently. We conduct experiments on 20 existing benchmarks across diverse tasks, and results demonstrate that Clean-Eval substantially restores the actual evaluation results on contaminated LLMs under both few-shot learning and fine-tuning scenarios.

## introduction in recent years, llms have made breakthroughs in handling complex and nuanced scenarios, achieved superior performance in some professional and academic benchmarks, and attracted many resources from industry and academia @xcite @xcite . this subsequently opens the arms race era of llms, and various llms are continuously launched, such as @xcite , llama2 @xcite partial work was done when wenhong zhu was interning at facemind corporation. ## clean-eval the framework of our method is shown in figure 2. our methodology comprises three primary stages. initially, we concentrate on gathering existing benchmarks to assess llms. in the subsequent phase, we meticulously cleaned contamination in the collected benchmarks. this involves paraphrasing samples using the creative capacities of the llms and performing multi-language backtranslation on the contaminated data. in the final phase, we use the semantic detector to filter the outcomes of the contamination cleanup, eliminating subpar results and selecting the ultimate results based on the bleurt score. ## evaluation setting nearly all llms operate with proprietary training datasets, making it challenging to ascertain whether the data being tested is free from contamination. we introduce an experimental framework for simulating data contamination to address this issue. ## analysis 6.1 ablation study in ## case study in this case, the paraphrased sentence successfully conveys the essence of the original while introducing some variation. the transformation maintains a positive sentiment, emphasizing the documentary's coolness and appeal to the audience. back-translation aims to ensure that the paraphrased sentence retains its intended meaning. the back-translated sentence aligns well with the paraphrased version. the key elements, such as the documentary's uniqueness, coolness, and audience appeal, are preserved. the combined approach of paraphrasing and back-translation effectively enhances the original sentence. the paraphrased version introduces a nuanced expression, and the subsequent back-translation successfully captures the intended meaning. the final output maintains a positive tone and successfully communicates the documentary's appeal. ## conclusion data contamination is an urgent problem for the development of llms society. downloading and trying contaminated models can be a waste of time for both researchers and developers. to save their time, this paper intends to mitigate the issue of data contamination in llms by introducing the clean-eval method. this approach leverages existing datasets to create a new evaluation dataset, effectively mitigating the impact of contamination. experimental results demonstrate the method's success in accurately assessing model capabilities. clean-eval holds promise in enhancing transparency and reliability in evaluating llms. future work can be dedicated to co-training a data contamination detector in a neural-based framework with clean-eval in a multi-tasking fashion. additionally, we hope to open-source various versions of intentionally contaminated llms and their contamination information for research purposes.