L asting B ench: Defend Benchmarks Against Knowledge Leakage (2025)

Abstract: The increasing size and complexity of large language models (LLMs) raise concerns about their ability to “cheat” on standard Question Answering (QA) benchmarks by memorizing task-specific data. This undermines the validity of benchmark evaluations, as they no longer reflect genuine model capabilities but instead the effects of data leakage. While existing methods detect such leakage, they fail to address the long-term challenge of mitigating it. In this paper, we introduce LastingBench, a novel approach to reinforce and safeguard existing benchmarks against knowledge leakage. Our method involves identifying leakage points through perturbation-based detection, followed by counterfactual rewriting to disrupt memorization while preserving the benchmark’s original evaluative intent. We demonstrate that our approach significantly reduces memorization effects in long-context QA benchmarks, providing a more accurate assessment of model reasoning and generalization abilities. Our experiments show that LastingBench not only uncovers substantial leakage in benchmarks like HotpotQA but also yields a more reliable evaluation of state-of-the-art models, ensuring that benchmarks remain effective and resilient over time.

## introduction the rapid advancement of llms has introduced critical challenges to the reliability and validity of qa evaluation benchmarks @xcite @xcite . due to the opaque and large-scale nature of llm training pipelines, these models often memorize parts of benchmark datasets @xcite @xcite @xcite . this unintended data leakage enables models to "cheat" during evaluation, producing correct answers without genuine understanding. as a result, evaluation scores on standard benchmarks may no longer reflect genuine model capabilities, but instead represent latent memorization effects. this poses a serious threat to fair and meaningful measurement in nlp research @xcite @xcite . to address this issue, recent efforts have proposed various techniques to detect benchmark contamination, including perplexity-based analysis @xcite , promptbased probing @xcite , and guided completions @xcite . while useful for identifying leakage, these approaches do not offer long-term solutions, especially as llms continue to grow in size and training data volume. a common reactive strategy is to retire leaked benchmarks and create new ones, as seen in systems like livecodebench @xcite . however, this strategy is both costly and unsustainable, requiring continuous data collection and annotation @xcite @xcite . moreover, previously valuable benchmarks that are no longer maintained risk becoming obsolete, leading to wasted effort and lost evaluative insights. therefore, the research community needs a robust pipeline not just for detecting contamination, but for actively recovering and reinforcing existing benchmarks @xcite . such a pipeline should enable benchmarks to remain reliable and resilient over time-regardless of future advances in model architectures, capabilities, or dataset scale @xcite . this would ensure evaluations focus on true improvements in reasoning and generalization, rather than incidental training data exposure. in this paper, we propose lastingbench , a novel framework to reinforce and safeguard existing benchmarks against knowledge leakage, specifically targeting the long-context qa domain. rather than continuously building new benchmarks, lastingbench enhances existing ones by identifying leakage points and rewriting them using counterfactuals, preserving the benchmark's original purpose while disrupting model memorization. our methodology begins with perturbation-based detection. by systematically perturbing the original context and the question, we assess whether a model relies on internal memorization rather than contextual clues. when leakage is detected, we locate critical evidence segments-the minimal context required to justify the answer-using enriched chain-of-thought (cot) queries from an llm. these segments are then rewritten into plausible yet semantically contradictory counterfactuals. this makes the benchmark robust against memorization while maintaining its reasoning challenge. we apply lastingbench to several longcontext qa benchmarks and find widespread memorization, especially in hotpotqa. our experiments reveal that larger models, such as gpt-4o and llama-4-maverick, exhibit higher levels of leakage. after applying our rewriting pipeline, we observe a substantial drop in inflated performance metrics, indicating a shift toward evaluations based on genuine reasoning and generalization. our contributions can be summarized as follows: ## knowledge leakage: detection and current landscape in this section, we systematically investigate the phenomenon of knowledge leakage-whether a model's correct answers stem from memorized training data rather than contextual reasoning. our study aims to identify and quantify large-scale knowledge leakage in existing long-context qa benchmarks. ## experiments in this section, we evaluate lastingbench through experiments. we use the same experimental setup as in section 2. ## conclusion in this paper, we introduced lastingbench, a novel framework designed to address the critical challenge of model cheating and knowledge leakage in qa benchmarks. lastingbench combines perturbation-based detection with counterfactual rewriting to identify and repair leakage points-disrupting memorization while preserving the original evaluative intent of the benchmark. our experiments on long-context qa benchmarks reveal widespread data leakage across a range of models, including both frontier and smaller opensource llms. by applying lastingbench, we substantially reduce memorization effects and provide a more faithful evaluation of models' reason-ing and generalization capabilities. this work offers a robust and sustainable approach to reinforce benchmark integrity, ensuring that evaluations remain meaningful and resilient against the evolving capabilities of llms.