Data Contamination Calibration for Black-box LLM s (2024)

Abstract: The rapid advancements of Large Language Models (LLMs) tightly associate with the expansion of the training data size. However, the unchecked ultra-large-scale training sets introduce a series of potential risks like data contamination, i.e. the benchmark data is used for training. In this work, we propose a holistic method named Polarized Augment Calibration (PAC) along with a new to-be-released dataset to detect the contaminated data and diminish the contamination effect. PAC extends the popular MIA (Membership Inference Attack) — from machine learning community — by forming a more global target at detecting training data to Clarify invisible training data. As a pioneering work, PAC is very much plug-and-play that can be integrated with most (if not all) current white- and black-box LLMs. By extensive experiments, PAC outperforms existing methods by at least 4.5%, towards data contamination detection on more 4 dataset formats, with more than 10 base LLMs. Besides, our application in real-world scenarios highlights the prominent presence of contamination and related issues.

## introduction as is widely acknowledged, the rapid advancements of large language models (llms) in natural language tasks are largely attributed to the incredible expansion of the size of the training data @xcite . despite the massive successes, this unmanaged expansion has introduced a series of significant issues that are yet explored, particularly data contamination. this issue arises notably when the benchmarking data is inadvertently included in the training sets. this contamination leads to misleading evaluation results @xcite , thus deducing difficulties in acquiring effective and secure consequently, pac compares the polarized distance of z with its adjacent samples z, where the polarized distance is a spatial measurement jointly considering far and near probability regions. in this work, we position the training data detection for llms as an extension of the membership inference attacks (mia) @xcite in the literature of machine learning. mia targets distinguishing whether the given data samples are members (training data) or non-members (not be trained). the previous line of work can generally be categorized into score-based (i.e., calibrationfree) @xcite @xcite and calibration-based @xcite @xcite methodologies. despite the promise, these approaches hardly suffice for current llms. the conventional setup in machine learning may generally focus on a small-scale training set, accompanied by global confidence distribution differences between members and non-members. however, this assumption no longer holds in llms, leading to a situation where non-members can also exhibit misleadingly high confidence levels, as shown in figure 2 . in addition, the mia approaches generally rely on training an external reference or proxy model using member data distribution approximated to the targeted model. this, unfortunately, has contradicted the black-box setup of many current llms. @xcite , where pac significantly enhances the salience of differences between members and non-members. with this paper, we make two major efforts: (i)-a holistic scheme named polarized augment calibration (pac) to resolve the challenges of black-box training data detection and (ii)-a brand new dataset for newly released llms. notably, the current related methods mostly assess the overfitting metric at individual points. upon revisiting the issue of typical mia, we find that the geometric properties of the samples may better reflect the latent differences introduced by training. for members, they should exhibit two characteristics: (i) high confidence at individual points; and (ii) being part of a poorly generalized local manifold. in other words, if a sample lies in a poor-generalization region but shows anomalous confidence, it suggests that the model is merely memorizing (or overfitting) rather than truly generalizing. based on such observations, as depicted in figure 1 , we propose polarized augment calibration (pac), which explores the confidence discrepancies in local regions through data augmentation tech-niques, aimed at detecting those overfitted training samples. upon this, to address the bias of existing global confidence metrics, we have developed a brand-new evaluation score named polarized distance, as a polarized calculation of flagged tokens with far and near local regions in the probability space. we further provide an in-depth explanation of pac detection from a theoretical perspective. compared to previous relevant methods relying on fully probability-accessed llms, we introduce a new probabilistic tracking method to extract probabilities under limited conditions(e.g., openai api) for the first time.2 this becomes the trailblazers to extend our detection to a black-box setup where only access to partial probabilities is permitted, enabling pac to be applied to almost all llms. for the benchmark, the current available benchmark for the contamination problem is very limited. we construct and introduce stackmia tailored for reliable and scalable detection of the latest llms. it ensures reliability by adopting a timebased member/non-member classification similar to @xcite , where members are visible during pre-training and non-members are not. stack-mia is dynamically updated, by offering detailed timestamps in order to quickly adapt to any up-todate llm through our curation pipeline. by contrast, the existing benchmark wikimia @xcite does not possess these properties, making it hard to use, especially on the llms released post-2023. last but not least, we exhibit extensive experiments on 10 commonly used models to evaluate the pac against six existing major baselines. these results demonstrate that pac outperforms the strongest existing baseline by 5.9% and 4.5% in the auc score respectively on stackmia and wikimia. pac further expresses superior robustness under conditions of ambiguous memory or detection of fine-tuning data. to further validate the effectiveness of pac in real-world applications, we provide a set of case studies for data contamination and different security risks @xcite on chatgpt and gpt-4. the conclusions from these examples highlight the ubiquity of security risks upon widespread deployment @xcite . our contributions can be summarized as: ## problem definition and preliminary work we first provide a comprehensive formal definition of the training data detection for llms in the context of a two-stage training process (kenton and toutanova, 2019). upon this foundation, we introduce a new dynamic data benchmark (section 3.2) that can be applied to recently released llms. by open-sourcing the benchmark upon publication, we expect to use it to foster further research in the community. ## methodology we introduce polarized augment calibration (pac), an efficient and novel calibration-based training data detection method. the key idea is to construct adjacent samples using easy data augmentation for calibrating a generalized distribution and design a brand-new polarized distance to enhance the salience. we further propose a probabilistic tracking method suitable for models with partially inaccessible logits for the first time. ## case study: date contamination to further uncover the potential risks of existing llms (large language models) through pac, we selected two logical reasoning datasets, gsm8k @xcite and aqua @xcite , and one ethical bias investigation dataset, toxi-gen @xcite on gpts llms. as depicted in ## limitations while pac shows promising results in detecting training data contamination in llms, its full potential is yet to be realized due to certain constraints. the limited availability of detailed training data information from llms providers restricts comprehensive validation across diverse models, underscoring the method's novelty yet implicating its untapped applicability. additionally, the efficacy of pac could be further enhanced with a more varied dataset, suggesting its adaptability and scope for refinement in varied llms. however, our current computational resources limit the extent of experiments, particularly on larger-scale llms, hinting at the method's scalability potential yet to be fully explored.