Investigating Data Contamination in Modern Benchmarks for Large Language Models (2024)

Abstract: Recent observations have underscored a disparity between the inflated benchmark scores and the actual performance of LLMs, raising concerns about potential contamination of evaluation benchmarks. This issue is especially critical for closed-source models and certain open-source models where training data transparency is lacking. In this paper we study data contamination by proposing two methods tailored for both open-source and proprietary LLMs. We first introduce a retrieval-based system to explore potential overlaps between evaluation benchmarks and pretraining corpora. We further present a novel investigation protocol named Testset Slot Guessing (TS-Guessing), applicable to both open and proprietary models. This approach entails masking a wrong answer in a multiple-choice question and prompting the model to fill in the gap. Additionally, it involves obscuring an unlikely word in an evaluation example and asking the model to produce it. We find that certain commercial LLMs could surprisingly guess the missing option in various test sets. Specifically, in the MMLU benchmark, ChatGPT and GPT-4 demonstrated an exact match rate of 52% and 57%, respectively, in guessing the missing options in benchmark test data. We hope these results underscore the need for more robust evaluation methodologies and benchmarks in the field.

## introduction large language models (llms) have demonstrated exceptional performance across a wide range of nlp tasks, and the nlp community has witnessed the emergence of several impressive llms. notably, there are robust proprietary llms, including the gpt-* @xcite , claude (anthropic, 2023), and bard @xcite , among others. in addition to these proprietary models, there are numerous open-source llms, such as llama @xcite , mpt @xcite , falcon @xcite , and mistral @xcite . however, with the increasing compute scale (including data) used to train these models, concerns have arisen regarding the extensive use of crawled web data, often at a terabyte scale. this extensive training data may, in turn, potentially include instances of evaluation benchmarks @xcite @xcite , many of which are also constructed from internet sources. research has demonstrated that the use of evaluation benchmark data in training sets (i.e., contamination) can artificially inflate performance metrics, regardless of whether contamination occurs during pretraining @xcite or fine-tuning @xcite . consequently, it becomes imperative for the research community to develop methods for detecting potential data contamination in these models @xcite . one of the most commonly used methods to detect data contamination has been n-gram matching @xcite @xcite . particularly, a number of previous works have employed n-gram tokenization to partition large documents into smaller segments, subsequently assessing their similarity to benchmark data @xcite . however, this approach is heavily reliant on having full access to the training corpus. this dependency poses a significant challenge in estimating data contamination for models where the training data is not disclosed @xcite @xcite @xcite . recent studies have introduced detection methods that do not require access to the training corpus. these methods, however, might be constrained to a dataset-level granularity as noted by @xcite ; @xcite or require fine-tuning of open-source models @xcite . given these limitations, there is an evident need for developing new methodologies to detect potential contamination in both open-source ## related work retrieving from large corpora retrieving from large corpus is an emerging topic in the era of llms. a number of works have focused on the retrieval and removal of contaminated information in training data by means of n-gram matching. specifically, recent work has focused on building indexing tools for large corpora @xcite @xcite , which allows efficient retrieval. additionally, previous work including gpt-3 (appendix c; brown et al., 2020) utilized a 13-gram tokenization strategy for both training and benchmark data for decontamination purposes. similarly, palm (chowdhery et al., 2022) employs an 8-gram approach, considering data as contaminated if there is a 70% overlap with 8-grams from the test set. open-source models like llama (touvron et al., 2023a) adopt a methodology akin to gpt-3's, while llama 2 (touvron et al., 2023b) (section a.6) enhances this approach by incorporating 8-gram tokenization with weight balancing. moreover, dodge et al. (2021) discusses documenting the large corpus c4 and benchmarking to detect data contamination, while elazar et al. (2023) provides a detailed analysis of various aspects of open training data including c4, redpajama, pile, the stack, etc, and providing analysis of potential contamination on glue and su-perglue benchmarks. besides the research conducted on english-only corpora, blevins and zettlemoyer (2022) investigate language contamination in cross-lingual settings. while n-gram matching can provide some level of detection for contaminated data, recent work has found that many test examples can remain undetected using such methods @xcite . futhermore, @xcite also investigates code contamination both in the surface-level and semantic-level. ## tradeoff between retrieval and ts-guessing the first method (retrieval) detailed involves constructing an ir system, derived from gpt-3's methodology and extended to llama, which is aimed at identifying and retrieving contaminated data from commonly utilized pretraining corpora. our second approach, named ts-guessing, offers a novel solution suitable in scenarios where such access is restricted, and is applicable to both closed-source and open-source models without requiring training data transparency. in a comparative analysis between two method-ologies, retrieval and ts-guessing, we observe distinct differences in various aspects. retrieval requires access to the full training corpus and is characterized by a high time consumption, whereas ts-guessing does not require such access and is less time-consuming. the target model for retrieval is open-source with transparency in training data, in contrast to ts-guessing which can apply to both closed-source and open-source models. retrieval-based method is cost-effective and scalable, yet it demands high maintenance, whereas ts-guessing is easy to scale with low maintenance requirements. retrieval is deemed generally more reliable due to direct information retrieval from verified sources and is suitable for knowledge qa and traditional benchmarks, while may be less suitbale to detect contamination in reasoning benchmark. ts-guessing, while suitable for a broad range of benchmarks, including those requiring advanced reasoning, is considered less reliable since it relies on inferred knowledge rather than direct retrieval. ## conclusion and future work we introduce two approaches for investigating data contamination in several widely-used contemporary evaluation benchmarks. first, we develop an information retrieval system to identify benchmarks with significant overlap with the pre-training corpus. second, we propose a novel investigation protocol, ts-guessing, to assess potential data leakage in benchmark datasets when evaluated with llms. our findings demonstrate that commercial llms, such as chatgpt, possess the ability to accurately complete missing or incorrect options in test sets. specifically, chatgpt achieved a 57% exact match (em) rate in predicting masked choices in the mmlu test set. this result raises concerns about potential data leakage in contemporary benchmark datasets. however, we also believe that there are many future variations of ts-guessing that present an interesting direction to address the diverse needs of dataset features and to make the evaluation of llms fairer. we can obscure specific information segments to prompt language models into inferring the missing content on the benchmark, or even render it quantifiable akin to perplexity. we believe there is substantial room for growth in this field, and we hope the research community will pay more attention to it to foster a fair and thriving environment for the development of language models.