P a C o ST : Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models (2024)

Abstract: Large language models (LLMs) are known to be trained on vast amounts of data, which may unintentionally or intentionally include data from commonly used benchmarks. This inclusion can lead to cheatingly high scores on model leaderboards, yet result in disappointing performance in real-world applications. To address this benchmark contamination problem, we first propose a set of requirements that practical contamination detection methods should follow. Following these proposed requirements, we introduce PaCoST, a Paired Confidence Significance Testing to effectively detect benchmark contamination in LLMs. Our method constructs a counterpart for each piece of data with the same distribution, and performs statistical analysis of the corresponding confidence to test whether the model is significantly more confident under the original benchmark. We validate the effectiveness of PaCoST and apply it on popular open-source models and benchmarks. We find that almost all models and benchmarks we tested are suspected contaminated more or less. We finally call for new LLM evaluation methods.

## introduction large language models (llms) have brought about a paradigm shift in the domain of natural language processing, yielding notable enhancements across various evaluation benchmarks @xcite and demonstrating proficiency in professional examinations @xcite . these advancements primarily stem from extensive training on vast and diverse datasets sourced from multiple origins. however, the substantial volume of data has given rise to significant concerns regarding benchmark contamination, where benchmarks for llm evaluation are inadvertently or deliberately included in model training. this contamination presents considerable obstacles in accurately gauging the capabilities of llms. while efforts are being made to address this issue by removing benchmarks from training datasets and conducting contamination studies, these endeavors face numerous limitations @xcite @xcite . these limitations include narrow focus on specific benchmarks and reliance on the trustworthiness of vendors. moreover, the competitive dynamics within the field, coupled with copyright considerations, have resulted in recent model releases lacking accompanying contamination studies @xcite . hence, there is an urgent necessity for independent methods to audit llms for the presence of benchmark datasets, eliminating the dependence on model providers' cooperation. simultaneously, there has been a growing interest in heuristic membership inference algorithms designed to reverse-engineer aspects of the training dataset @xcite , thereby providing insights into potential test set contamination @xcite . despite their promise, these heuristic approaches often lack definitive proof of contamination and tend to rely on assumptions that may be too stringent. moreover, the majority of these methods concentrate less on detecting benchmark contamination. as elaborated in section 3.1, inherent challenges, such as the need for lengthy trained segments and the necessity of establishing thresholds, impede the adaptation of previous methods for detecting benchmark contamination. in this study, we introduce a novel approach named pacost (paired confidence significance testing) designed for the detection of benchmark contamination in open-source llms. our method entails a three-step statistical analysis, capable of identifying benchmarks within the model's training data. specifically, our approach involves constructing counterparts for each data instance with similar distribution, followed by statistical analysis of corresponding confidence scores to ascertain whether the model exhibits significantly higher confidence when presented with original benchmarks. we operate under the assumption that the model tends to demonstrate greater confidence when responding to questions it has been trained on. to validate our method rigorously, we conduct a series of controlled experiments. subsequently, we employ pacost across a diverse array of publicly accessible llms, scrutinizing various benchmarks to reveal contamination outcomes. our experimental observations indicate that, across the board, there are suspicions of contamination to varying degrees in both models and benchmarks. consequently, we advocate for the adoption of a benchmark-free evaluation approach as a means to mitigate this contamination issue. our contributions can be summarized as follows: ## method we introduce pacost, a novel benchmark contamination detection method that emphasizes the distinction between contaminated and clean data without relying on thresholds. our approach leverages the disparity in model behavior between original and rephrased instances, focusing on confidence rather than traditional performance metrics like accuracy @xcite . by conducting statistical analysis on confidence, we can robustly identify contamination. pacost comprises three key steps: rephrasing preparation, confidence estimation, and significance testing. through this method, we provide a clear and unique approach to detecting benchmark contamination in models. ## conclusion in this work, we introduce the issue of benchmark contamination in llms and propose several essential criteria that an effective benchmark contamination detection method should meet. we highlight that all existing detection methods fall short of satisfying all of these requirements. we then propose a benchmark contamination detection method named pacost, which uses significantly higher confidence scores as an indicator of contamination. we conduct various experiments to demonstrate the effectiveness of our method. additionally, we ap-ply our method to popular llms and benchmarks and reveal a significant problem of benchmark contamination across almost all benchmarks and llms we examined.