Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method (2024)

Abstract: As the scale of training corpora for large language models (LLMs) grows, model developers become increasingly reluctant to disclose details on their data. This lack of transparency poses challenges to scientific evaluation and ethical deployment. Recently, pretraining data detection approaches, which infer whether a given text was part of an LLM’s training data through black-box access, have been explored. The Min-K% Prob method, which has achieved state-of-the-art results, assumes that a non-training example tends to contain a few outlier words with low token probabilities. However, the effectiveness may be limited as it tends to misclassify non-training texts that contain many common words with high probabilities predicted by LLMs. To address this issue, we introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection. We compute the cross-entropy (i.e., the divergence) between the token probability distribution and the token frequency distribution to derive a detection score.We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text. Experimental results on English-language benchmarks and PatentMIA demonstrate that our proposed method significantly outperforms existing methods. Our code and PatentMIA benchmark are available at https://github.com/zhang-wei-chao/DC-PDD .

## introduction a critical element contributing to the effectiveness of large language models (llms) is the large volume of data used for pretraining. in many cases, model developers are reluctant to disclose information about their training corpus @xcite @xcite @xcite . this lack of transparency complicates the assurance that all ethical and legal standards are met. the pretraining corpus may contain unauthorized private information or copyrighted content @xcite . indeed, openai and nvidia face lawsuits over copyright issues related to their training data @xcite . moreover, a lack of transparency around the pretraining data used prevents us from properly addressing the data contamination problem @xcite and, hence, from determining whether an llm's performance is due to genuine task understanding or to prior exposure to test data. we focus on the following key question: how can we detect if a black-box llm was pretrained on a given text, considering that its training data is undisclosed? the pretraining data detection problem can be viewed as an instance of the membership inference attack (mia) task @xcite , where the primary objective is to determine if a particular text was part of a target llm's training corpus. prevailing methods to tackle this problem are based on the idea that a text's token probability distribution can reveal its inclusion in the training set. e.g., the min-k% prob method @xcite is based on the hypothesis that non-training examples tend to have more tokens assigned lower probabilities than training examples do. min-k% prob relies on the assumption that data with higher probability is more likely to be training data. language models trained with a cross-entropy loss function tend to favor high-frequency tokens when conducting next-token prediction, which will also lead to llms generally predicting higher probabilities for high-frequency tokens @xcite ). as the conceptual example shown in figure 1 , x 1 is a nontraining text and x 2 is a training text. we can see that the lowest raw token probabilities for x 1 are higher than those for x 2 , which may be because the words in x 1 (e.g., "boys", "great") are generally more common than the words in x 2 (e.g., "erudite", "conundrum"). therefore, min-k% prob will calculate a detection score of -0.88 for x 1 and -2.94 for x 1 , which means that x 1 is more likely to be considered a training text than. this is contrary to the actual situation. inspired by the divergence-from-randomness theory @xcite , we introduce a divergence-based calibration method, named dc-pdd, to calibrate the token probabilities for pretraining data detection. the basic idea underlying divergence-from-randomness is that the higher the divergence of the within-document termfrequency of a word in a document from its frequency within the collection, the more information the word carries. in our scenario, the withindocument term-frequency can be interpreted as the target llm's predicted probability for each token with regard to the text to be detected, to which we refer as the token probability distribution. the frequency of a word within the collection refers to the frequency of each token in the target llm's pretraining corpus, to which we refer as the token frequency distribution. according to the divergencefrom-randomness theory, the higher the divergence between these two distributions, the more informative the tokens are in indicating that the text was part of the model's training corpus, rather than solely relying on token probabilities as the indicator for detection. like prior works @xcite , we assume that we only have access to the target llm as a black box: we can compute token probabilities for the text to be detected but have no access to the internals of the llm (e.g., weights and activations). we first obtain the token probability distribution by querying the llm with the text. next, we use a large-scale publicly available corpus as a reference corpus to obtain an estimation of the token frequency distribution since an llm's pretraining corpus is not accessible usually. we then calibrate the token probabilities by comparing the token probability distribution to the token frequency distribution. based on the calibrated token probabilities, we derive a score for pretraining data detection. finally, a predefined threshold is applied to the score to determine whether the text was included in the llm's pretraining corpus. figure 1 (b) illustrates that dc-pdd assigns a score to text that better reflects whether it is training data or non-training data (i.e., a training text should have a higher score than a non-training text). in contrast to other calibration methods @xcite , dc-pdd neither requires additional reference models nor extra access requirements on the target llm. ## experimental settings benchmarks and models. to evaluate the performance of dc-pdd, we conduct experiments on three benchmarks mentioned in baselines. we consider the following methods as our baselines, each predicting whether an example was included in training set based on: (i) ppl: the perplexity of the example. (ii) lowercase: the ratio of the example's perplexity to that of the lowercased example. (iii) zlib: the ratio of the example's perplexity against its zlib entropy. pythia, qwen1.5, gpt-neox and gpt-3, <@xmath0, @xmath1. for the reference corpus d ′ to compute the token frequency distribution, we take a subset of c4 @xcite ) (≈ 15gb) for english text detection and take a subset of chinesewebtext @xcite (≈ 15gb) for chinese text detection. for hyperparameter a settings, we set it to 0.01 for wikimia and patentmia detection tasks, and to 10 for book-mia. since we take the auc score as our evaluation metric, we do not need to determine a specific threshold τ in our method. for the baseline implementation, we set @xmath2. correspondingly, the hyperparameter k in min-k%++ prob is also set to 20 for fair comparison. for the smaller reference model setting, we employ opt-350m as the smaller model for opt-6.7b, pythia-70m for pythia-6.9b, llama-7b for llama-13b, gpt-neo-125m for gpt-neox-20b, baichuan-7b for baichuan-13b and qwen1.5-7b for qwen1.5-14b. ## experimental results here, we report our main results, several ablation studies, and additional experiments investigating factors influencing detection performance. ## conclusion in this work, we proposed dc-pdd to improve methods that directly rely on token probabilities for pretraining data detection, which tend to misclassify non-training texts containing many common words as training texts. the key idea of dc-pdd is to calibrate the token probabilities and thereby make them more informative signals for detection. the calibration process is achieved by computing the cross-entropy (i.e., the divergence) between the token probability distribution and the token frequency distribution. experiments demonstrate the superior performances of dc-pdd compared to various baselines. in future work, we want to detect whether an llm was pretrained on a given corpus (corpus-level detection), rather than just on a piece of text (sample-level detection).