LNE -Blocking: An Efficient Framework for Contamination Mitigation Evaluation on Large Language Models (2025)

Abstract: The problem of data contamination is now almost inevitable during the development of large language models (LLMs), with the training data commonly integrating those evaluation benchmarks even unintentionally. This problem subsequently makes it hard to benchmark LLMs fairly. Instead of constructing contamination-free datasets (quite hard), we propose a novel framework, LNE-Blocking , to restore model performance prior to contamination on potentially leaked datasets. Our framework consists of two components: contamination detection and disruption operation. For the prompt, the framework first uses the contamination detection method, LNE , to assess the extent of contamination in the model. Based on this, it adjusts the intensity of the disruption operation, Blocking , to elicit non-memorized responses from the model. Our framework is the first to efficiently restore the model’s greedy decoding performance. This comes with a strong performance on multiple datasets with potential leakage risks, and it consistently achieves stable recovery results across different models and varying levels of data contamination. We release the code at https://github.com/RuijieH/LNE-Blocking to facilitate research.

## introduction in the era of fierce development with large language models (llms), it has been a popular research topic across many areas such as chain-ofthought reasoning @xcite , machine translation @xcite , code generation @xcite , and even spatial reasoning @xcite . despite the fact that llms are usually strong on many tasks, natural language processing (nlp) practitioners secretly face a common problem called data contamination when conducting nlp research and engineering which frequently relies on benchmark data that could be potentially contaminated in llms. data contamination, also called data leakage, occurs when the test data is inadvertently included in the model's training data @xcite . this causes the model to perform exceptionally well on the leaked test data. due to the immense scale and diverse origins of the pre-trained datasets used for llms, even when developers do not intentionally introduce contamination, llms are more vulnerable to data contamination. as a result, preventing benchmark data contamination in llms becomes highly challenging. this prevents nlp developers and researchers from honestly judging the llms. to conduct the contamination-free evaluation of models, prior works @xcite @xcite have assessed model performance by creating or reconstructing new datasets that are free from leakage. however, these approaches are associated with significant labor costs and do not completely eliminate the risk of these datasets being inadvertently leaked in newly released models. one less studied problem, defined as contamination mitigation evaluation, is conducting the contamination-free evaluation of models on datasets already at risk of leakage. for contamination mitigation evaluation, ted @xcite spends significant time sampling and generating multiple responses on existing benchmarks, then removes similar responses to assess the model's genuine performance under sampling methods. in contrast, this paper proposes the lne-blocking framework, which adaptively restores the model's true performance under varying contamination levels. it operates online during generation, without relying on sampled responses, and directly assesses the model's genuine performance under greedy decoding. specifically, the lne-blocking framework explicitly decouples contamination mitigation into contamination detection and disruption operations, as shown in figure 1 . the contamination detection strategy, lne (length normalized entropy), determines the degree of contamination based on the model's output, while the disruption operation, blocking, intervenes in the original generation process by suppressing the token with the highest response probability during decoding. overall, the framework adjusts the frequency of disruption operations based on the model's contamination level, prompting the model to generate non-memorized content for the current sample. to the best of our knowledge, we are the first to propose a method for assessing the model's genuine performance under greedy decoding. additionally, experiments demonstrate that our approach is highly robust across different tasks and models, even with varying levels of contamination. to this end, we make three key contributions: ## motivation contaminated models often exhibit a high lexical overlap between their output and the ground truth, which is indicative of memory phenomena @xcite . as shown in figure 2 , as the degree of contamination increases, the overlap between the output generated by greedy decoding and the ground truth significantly rises. this behaviour reflects the model's tendency to memorize the training data rather than generalizing it. one solution @xcite to assess the genuine performance of such contaminated models involves generating diverse outputs through multiple sampling and filtering out similar samples. then, they expect to derive non-memorized answers that stem from the model's generalization abilities, rather than its memorized knowledge. however, answers based on memory phenomena tend to have a very high likelihood within the decoding process, meaning that obtaining nonmemorized, generalized answers requires a large number of samplings. ted @xcite has found that at least 50 samples are necessary to achieve satisfactory performance estimates. this process is both highly random and time-consuming. nevertheless, the inherent randomness of the sampling process presents a fundamental limitation: it makes consistent generation of non-memorized answers across models with varying contamination levels challenging, even when employing the recommended 50 sampling attempts. particularly for heavily contaminated models, this approach fails because memorized outputs dominate the sampling distribution, and the probability of obtaining sufficient non-memorized samples is critically low for reliable evaluation. this fundamental limitation is mild cont. ## related work data contamination detection the issue of data contamination in large language models (llms) gained attention in the context of gpt-3 (brown, 2020), where the vast pre-training corpus inevitably overlapped with evaluation benchmarks. meanwhile, as models are iteratively improved by using data coming from users, they overlook the problem of indirect data leakage @xcite . following this, some work @xcite @xcite exposed the serious consequences of data contamination and urged attention to this problem. to address this, min-k% prob @xcite calculates the average of the smallest k% probabilities of generated tokens and flags potential contamination if this average exceeds a certain threshold. similarly, perplexity @xcite is also used to detect contamination, assuming that leaked data tends to produce lower perplexity scores. ## methodology the section discusses how to utilize lne for assessing the degree of contamination, how to employ blocking for disruption, and how to integrate both methods to construct a framework for contamination mitigation evaluation. ## conclusion in this paper, we propose the lne-blocking framework to address the challenge of data contamination in large language models (llms). by decoupling detection and disruption, the framework first restores the model's greedy decoding performance after contamination mitigation. through extensive experiments, we demonstrate that lne-blocking effectively restores model performance under greedy decoding, achieving robust and consistent promising results across diverse tasks and contamination levels. this work provides a practical and efficient solution for contamination mitigation, offering a new direction for fair benchmarking and reliable evaluation of llms.