Paper: Detecting Pretraining Data from Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The methodology sounds dubious. It doesn't take consideration of the data ordering in the training phase. For instance, in the paper, authors showed that learning rate affects the result a lot. In the late stage of training, the learning rate will be small which could impact how well model remember the data.

Author response:
We find the reviewer has some misconceptions of our method and setting. As clearly stated in abstract/intro and other sections, we study the pretraining data detection problem of **black-box language models: we only have black-box access to an LLM without knowing the pretraining data. It means we don’t know the data ordering, learning rates, and pretraining data distribution of the target model.** We study the problem under this black-box assumption because state-of-art LMs such as OpenAI GPT does not reveal any detail of their pretraining process and pretraining data. We show our method could be effectively applied to detecting copyrighted books in OpenAI text-davinci-003 (in Section 6). 

Our main results in Table 1 are a fair comparison with baseline approaches using the *same model*. The learning rates and data factors are the same for each of these comparisons. Consequently, the fact that Min-K Prob provides a consistent margin regardless of the source model shows that it robustly works better than other approaches. Note that we are not claiming Min-K prob can solve the training detection problem perfectly in any scenario (such as the low learning rate scenario as the author pointed out), but Min-K prob is a better detection method than existing methods.

Reviewer comment (Experiments):
It also doesn't consider the data distribution/mixture in the training. e.g. llama training duplicates wikipedia data.

Author response:
We find the reviewer has some misconceptions of our method and setting. As clearly stated in abstract/intro and other sections, we study the pretraining data detection problem of **black-box language models: we only have black-box access to an LLM without knowing the pretraining data. It means we don’t know the data ordering, learning rates, and pretraining data distribution of the target model.** We study the problem under this black-box assumption because state-of-art LMs such as OpenAI GPT does not reveal any detail of their pretraining process and pretraining data. We show our method could be effectively applied to detecting copyrighted books in OpenAI text-davinci-003 (in Section 6). 

Our main results in Table 1 are a fair comparison with baseline approaches using the *same model*. The learning rates and data factors are the same for each of these comparisons. Consequently, the fact that Min-K Prob provides a consistent margin regardless of the source model shows that it robustly works better than other approaches. Note that we are not claiming Min-K prob can solve the training detection problem perfectly in any scenario (such as the low learning rate scenario as the author pointed out), but Min-K prob is a better detection method than existing methods.

Reviewer comment (Evaluation):
High probability of a small piece of text doesn't mean that model is trained entirely on the content. The piece of text might appear somewhere else.

Author response:
We would like to clarify some misunderstandings regarding our approach. Our method is based on the hypothesis that an unseen example is likely to contain a few outlier words with low probabilities under the LLM, while a seen example is less likely to have words with such low probabilities. This is a key distinction – **our focus is on the identification of texts with extremely low probability words, rather than high probability ones.** In other words, our method identifies a text segment as a potential seen example if a significant portion of text exhibits non-low probabilities, rather than *"a small piece of text has high probability"*. Furthermore,  we have systematically validated it across seven different datasets with known ground truths. These datasets have been meticulously constructed to include both seen and unseen data, verified to be either included or excluded from the model’s pretraining corpus. Our results demonstrate that our method consistently outperforms previous baselines.

Reviewer comment (Novelty):
Lack enough of novelty

Author response:
We respectfully request that the reviewer provide further details regarding their concerns about the lack of novelty in our work. Could you direct us to any related works for comparison?

We are **the first to study the pretraining data detection problem of large-scale black-box language models.** We highlight three primary areas of innovation:

* Creation of a Dynamic Benchmark: We introduce a dynamic benchmark specifically designed for this new problem domain. This benchmark is the first of its kind and is tailored to assess the effectiveness of data detection methods in the context of large-scale language models.

* Development of the detection method MIN-K% PROB: Our proposed method, MIN-K% PROB, is a significant departure from existing detection techniques. Unlike previous methods that require training a reference model on data akin to the pretraining corpus, MIN-K% PROB requires no knowledge of the pretraining data and no additional training. We demonstrate the superior performance of our methods compared to existing baselines across all datasets. 

* Real-World Application: We apply our research to real-world applications, including the detection of copyrighted books and contaminated downstream examples. These case studies showcase their practical relevance in addressing critical issues in the field.

Reviewer comment (Experiments):
The method used here doesn't take consideration of the data ordering in the training phase. For instance, as is shown in the paper that learning rate affects the prediction a lot. In the late stage of training, the learning rate would be small which could impact how well model remember the data.

Author response:
We find the reviewer has some misconceptions of our method and setting. As clearly stated in abstract/intro and other sections, we study the pretraining data detection problem of **black-box language models: we only have black-box access to an LLM without knowing the pretraining data. It means we don’t know the data ordering, learning rates, and pretraining data distribution of the target model.** We study the problem under this black-box assumption because state-of-art LMs such as OpenAI GPT does not reveal any detail of their pretraining process and pretraining data. We show our method could be effectively applied to detecting copyrighted books in OpenAI text-davinci-003 (in Section 6). 

Our main results in Table 1 are a fair comparison with baseline approaches using the *same model*. The learning rates and data factors are the same for each of these comparisons. Consequently, the fact that Min-K Prob provides a consistent margin regardless of the source model shows that it robustly works better than other approaches. Note that we are not claiming Min-K prob can solve the training detection problem perfectly in any scenario (such as the low learning rate scenario as the author pointed out), but Min-K prob is a better detection method than existing methods.

Reviewer comment (Theory):
On Page 8, second paragraph, the authors report that their accuracy is proportional to the pretraining corpus size. They cite a paper on memorization of neural networks and argue that outliers are memorized better. The question is: with the same theory, can the authors justify why the outliers of a small dataset are not memorized better than the outliers of a large dataset? This is what they are reporting, but it cannot be supported by the cited theory.

Author response:
> Page 8, second paragraph. Authors report that their accuracy is proportional to the pretraining corpus size. They also cite a paper on memorization of NNs, and argue that outliers are memorized better. My question is: with the same theory, can Authors justify why the outliers of a small dataset are not memorized better than the outliers of a large dataset? Because that is what they are reporting. This cannot be supported by the cited theory.

Thanks for the insightful question! To clarify, Feldman et al., 2021 [1] state that the learning algorithms tend to memorize the outliers for a better generalization. Zhang et al. 2021 [2] use counterfactual memorization to capture the source of memorization at test time and empirically identify a long tail of examples (i.e., outliers) with high memorization scores.

To connect them to our experiment, our hypothetical explanation is as follows. In a larger dataset, downstream data, comprising a smaller fraction (0.01%), more resembles an outlier distribution and, hence, might be memorized more effectively compared to its proportion in a smaller dataset (10% downstream data). Although Feldman et al., 2021[1] did not explicitly mention this observation, our additional experiments with in-distribution data support our hypothesis, showing a reversal of dataset effect. **We have updated the paper (Section 6.2)** to state more clearly about the connection to Feldman et al., 2021[1] (we've highlighted the changes in red)

[1] Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954–959, 2020.

[2] Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938, 2021

Reviewer comment (Theory):
The most important question is why the proposed method works. The authors have not answered this pivotal question. Why not use the occurrence probability of all the words, the occurrence probability of the entire sentence, or the occurrence probability of sentences that appear in the pretraining corpus and filter out the rest? The question is “WHY” the proposed heuristic works.

Author response:
> My last question (and the most important one) is that why the proposed method works? Authors have not answered this pivotal question. Why not using the occurrence probability of all the words...

Our method is based on the intuition that trained data tends to contain fewer outlier tokens with very low probabilities compared to other baselines. If a sequence has been trained, gradient descent on this sequence almost ensures a reduction in the losses of all tokens, making it unlikely for tokens with extremely low probabilities to exist under the trained language model. Conversely, untrained sequences are prone to contain unexpected tokens with lower probabilities, although these sequences are still valid.

Reviewer comment (Evaluation):
A recently published paper (“Elephants Never Forget: Testing Language Models for Memorization of Tabular Data.” Sebastian Bordt, Harsha Nori, Rich Caruana. NeurIPS 2023 Second Table Representation Learning Workshop) claims that LLMs memorize training sets, although not verbatim. This implies that the proposed method by the authors will fail to detect such cases, since the authors have stated they would only look into the exact match case.

Author response:
> I suggest the authors look into the paper below---published about 10 days ago---the paper claims that LLMs memorize training sets, although not verbatim. So the proposed method by the authors will fail to detect such cases---I know that the authors have said they would only look into the exact match case.

We would like to clarify that our study **is not limited to detecting exact match cases**. As explicitly stated in Section 2.2 under 'benchmark setting', our research addresses the detection of pretraining data in both exact match and **paraphrased scenarios**. As shown in Table 1, our method is effective for detecting paraphrases of training examples. This demonstrates our approach's capability to identify variations of training data, not just verbatim instances. Thanks for pointing us to the related works. **We have included discussion of this paper in our updated version** (Section 2.2, highlighted in yellow).

Reviewer comment (Novelty):
On Page 8, in the paragraph “Data occurrence”, it appears the authors are not aware that this is not a new finding. The cited work [1] (“Large Language Models Struggle to Learn Long-Tail Knowledge.” ICML 2023) reports comprehensive experiments on this subject.

Author response:
> In Page 8, paragraph “Data occurrence”, it looks Authors are not aware that this is not a new finding. [1] reports comprehensive experiments on this subject.

Thank you for sharing [1]. **We have incorporated it into our submission when discussing our findings (Section 6.2, highlighted in yellow)** It's worth noting that we arrived at this conclusion through a different approach compared to [1]: While [1] measures memorization by conducting a data extraction attack [2] and reporting the reconstruction fidelity, we utilize a membership inference attack. This divergence in methodologies, yet convergence in findings, also adds to the credibility of our results.

Furthermore, our study extends beyond data occurrences. We conduct additional experiments analyzing how various factors, such as learning rates and dataset size, influence the overall performance. This broader scope allows us to offer a more comprehensive perspective on how the pretraining design choices affects detection difficulty.

[1] Large Language Models Struggle to Learn Long-Tail Knowledge. ICML 2023.
[2] Carlini, et al. "Extracting training data from large language models." USENIX Security 2021.

Reviewer comment (Experiments):
I presume the authors only work with English? That itself is not a weakness, of course, but it should be clearly stated. The "weakness" part of working with only English, is that I think lower-resourced languages may contradict the underlying hypothesis propping up the MIN-K PROB method: That is, because English is such a large percentage of training corpora for multilingual models, I'm not sure how this would work for other languages where the token probabilities given by LLMs are likely to be very low in general. How does this square with the authors' original hypothesis, and newly proposed method?

Author response:
Thanks for your feedback! We've revised Section 8, the conclusion section, to include a note that our evaluations were exclusively conducted on English datasets. We also highlight in this section the need for future research to focus on the evaluation of low-resource languages (**Changes in the paper have been highlighted in yellow in the updated PDF**).

While our current focus has been on English, we are optimistic about the applicability of our methods to texts that constitute a minor fraction of the pretraining data, such as non-English corpora. For example, in our dataset contamination experiments, we detected test set contamination, which is only **0.01%** of the pretraining data and our method shows effectiveness in these settings. 

The primary challenge we face in extending our evaluation to non-English languages is the complexity and effort required to construct a robust non-English dataset tailored for membership inference attack detection. We welcome suggestions for relevant datasets in non-English languages and would be very happy to test our method on them.

Reviewer comment (Writing):
[S4.1] You say you use "different lengths (32, 64, 128, 256)", but do not specify what -- is this characters? tokens? words? S5.1 says "words" specifically.

Author response:
The length refers to “words” and our examples are sentence-level. Thanks for pointing it out. **We have updated the paper (Section 2.2) to make it clear.**

Reviewer comment (Writing):
[S6.2] I find the verbiage around "in-distribution" and "outlier" contaminants to be a bit awkward. If I'm understanding you correctly, Fig5 (a) shows the contaminates that you correctly detect in the continued-pretrained model, (b) shows contaminated unrelated to the continued-pretrained model, but still included in pre-training. Is (a) the "outlier", simply because its not supposed to be included in pre-training data, and thus (b) is the "in-distribution" because this sort of data is presumed to be the type of thing we'd expect in pretraining data?  Maybe reword to "downstream task data" and "control data", or something of the like?

Author response:
Yes. (a) is the outlier because the downstream task data is not supposed to be included in pretraining data, while (b) is the sort of data we'd expect in pretraining. **We have updated the wording in the paper.** Thanks for your suggestion!

Reviewer comment (Experiments):
Only one dataset is used for experiments.

Author response:
We evaluate our proposed method and baselines on **7 datasets**, including 1 WikiMIA dataset, 5 datasets from our dataset contamination detection experiments (BoolQ, Commonsense QA, IMDB, Truthful QA in Section 5) and 1 copyright book detection experiments (in Section 6). Across 7 datasets, we observe that our proposed method is consistently better than the baselines.

Reviewer comment (Reproducibility):
Please correct me if I am wrong. The proposed method seems to only applicable to LLMs which provide logits of outputs. Can it be applied to SOTA LLMs such as GPT-4 and Claud?

Author response:
Our method requires output logits from LLMs. OpenAI models such as text-davinci-003 provide such logits. For example, Section 5 shows the effectiveness of our method in detecting copyrighted books from text-davinci-003 (0.88 AUC). 

As stated by OpenAI (https://openai.com/blog/new-models-and-developer-products-announced-at-devday), GPT-4 will provide output logits in the next few weeks. We look forward to experimenting with our method on GPT-4.

Reviewer comment (Evaluation):
Please correct me if I am wrong. The proposed method seems to have difficulty in detecting the texts which are very well written or very badly written.

Author response:
The proposed method is **effective across any type of text**. including very well-written and very badly written texts. Our method relies on a hypothesis: an unseen example is likely to contain a few outlier words with low probabilities under the LLM, while a seen example is less likely to have words with such low probabilities. This hypothesis is not limited to specific writing styles or quality levels.