Paper: Proving Test Set Contamination in Black-Box Language Models (ICLR-2024)

Reviewer comment (Theory):
I'm most concerned about the definition of contamination used in this paper. Currently, the most popular definition of contamination follows the n-gram analysis. In real-world scenarios when training large language models, it's hardly seen to directly feed original data samples in their original ordering as shown in Figure 1. The application of this work could be greatly limited.

Author response:
N-gram overlap is a commonly utilized measure of contamination in the literature; however, it should be noted that it acts more as a measurement tool rather than as a definition of contamination. N-gram overlap may fail to distinguish between coincidental overlap and genuine contamination in some situations, and has been observed to potentially lead to false positives under certain conditions. For example, the Stanford Question-Answering Dataset (SQuAD v2) uses background information derived from Wikipedia, and in the GPT-3 paper, Brown et. al. (2020) find high N-gram overlap for this reason, even if labels are not present in the data and no true contamination exists. Please see pg. 43 of that paper for examples of false positives in their contamination study. 

We believe that the definition introduced in our paper, of contamination as a statistical dependence between the model and the dataset, is precise and formal, and better captures the notion of contamination as a transfer of information between the test set and the model—not simply a correlation that appears because both the test set and pre-training data share information. 

While verbatim contamination of ordered data does not encompass all forms of contamination, we found that the presence of ordered test sets in pre-training data is surprisingly common. A search of The Pile, a large open-source language modeling dataset, yielded numerous instances of real-world datasets embedded with examples appearing in-order; see our top-level comment for an example. 

In pre-training, shuffling of the data occurs at the document level, and is not typically applied to the data within a document. Files collected from the internet would typically be treated as singular documents in dataset construction pipelines. 

More importantly, the use of ordering allows us to give provable guarantees of contamination, which is more difficult to achieve for other, less direct forms of contamination. Our work is the only existing contamination detection method for language models to give guarantees of this kind.

Reviewer comment (Experiments):
From Figure 3, it seems that the parameters for shards and permutations are sensitive and have to be carefully selected when being applied to other test sets.

Author response:
Regarding concerns about sensitivity of the test to the shard count, figure 3 (left) shows that a wide range of shard counts (between 10 and 150) attain p-values below 1e-4. Once the p-value is low enough that statistical significance is attained, there is no added benefit to lowering the p-value. Therefore, the plot suggests that the test is robust to shard count, so long as the shard count is not too low (so that the t-test can still be used reasonably), and not too high (so that there are enough examples per shard to get sufficient signal from log-prob differences.)

Similarly, figure 3 (right) shows that increasing the permutation count monotonically decreases the p-value, and that the p-value stabilizes beyond about 25 permutations per shard. This suggests that the test is not sensitive to the permutation count, provided that the permutation count is not too low. Our empirical results use a permutation count of at least 50. We welcome further discussion on this point to ensure we fully understand and address your concerns.

Reviewer comment (Experiments):
The paper only targets direct sentence appearance in the pre-training stage. What about instruction-tuning data in the SFT stage?

Author response:
Detecting contamination at the instruction fine-tuning stage would be interesting follow-up work, but is complicated by the fact that examples are commonly shuffled in this setting, and so we cannot test against a known example order. In this setting, heuristic methods may prove to be more effective.

Reviewer comment (Writing):
Could you further explain "high false positives" in existing n-gram-based analyses?

Author response:
N-gram overlap is a commonly utilized measure of contamination in the literature; however, it should be noted that it acts more as a measurement tool rather than as a definition of contamination. N-gram overlap may fail to distinguish between coincidental overlap and genuine contamination in some situations, and has been observed to potentially lead to false positives under certain conditions. For example, the Stanford Question-Answering Dataset (SQuAD v2) uses background information derived from Wikipedia, and in the GPT-3 paper, Brown et. al. (2020) find high N-gram overlap for this reason, even if labels are not present in the data and no true contamination exists. Please see pg. 43 of that paper for examples of false positives in their contamination study.

Reviewer comment (Experiments):
Although a more efficient sharded rank comparison test is proposed, the computational complexity is still considerable. For example, testing 49 files using 1000 permutations per shard can take 12 hours for LLaMA2.

Author response:
We'd like to address the concern regarding the computational complexity of our test. It's important to note that the test is a one-time process for any given model and dataset; once the p-values are computed, there is no need for recalculation. Our findings indicate that a number of permutations beyond 30-50 per shard offers diminishing returns, as shown in Figure 3 (right).

Furthermore, the test's design allows for easy parallelization. Each shard permutation can be evaluated independently, enabling the use of inexpensive commodity hardware to run the test significantly faster.

Reviewer comment (Theory):
The method relies on a strong assumption of data exchangeability, which may not hold in real-world datasets.

Author response:
Regarding the assumption of data exchangeability, this is a strictly weaker condition than the commonly held assumption of independent and identically distributed (I.I.D.) data in machine learning. Most datasets satisfy this assumption to some extent.

Reviewer comment (Theory):
If a dataset is not exchangeable, how effective is the method?

Author response:
We acknowledge the validity of our test hinges on data exchangeability. However, depending on the source of non-exchangeability, it is often the case that a dataset can be altered slightly so that our test is still valid. For example, a common source of non-exchangeability is the presence of ascending IDs (e.g. as in SQuAD and HumanEval). We can adjust the data—by either removing these IDs or permuting the examples while keeping IDs constant—to retain the test's applicability. This is discussed in more detail in the revised paper.

Reviewer comment (Experiments):
How sensitive is the proposed test to the model size?

Author response:
Thank you for your question regarding the impact of model size on the performance of the test. Existing work on memorization in language models suggests that larger models memorize their training data more strongly. For example, Carlini et. al. (2023) show that both model size and repetitions in the training data increase the extractability of training data sequences. Our empirical results show that the power of our test increases dramatically in the number of repetitions (duplication rate), and we posit that our test enjoys a similar increase in power for larger model sizes, since larger models likely memorize example order more strongly.

We present preliminary results on the impact of model size on the power of our test. We evaluated three models of increasing parameter counts trained on the same data mixtures as in section 4.1, on the test sets present in the pretraining data at a duplication rate of 1. These results suggest that the test performs better with larger models.

| Parameter Count | Average Logarithm of P-Value |
|-----------------|---------|
| 355M          | -1.427  |
| 774M          | -1.825 |
| 1500M          | -12.783  |

Reviewer comment (Experiments):
I have some questions about the definition of test set contamination below.

Author response:
Regarding the definition of test set contamination, we specifically address verbatim contamination, where both inputs and labels from the test set appear in the training data in order. While other forms of contamination can be studied, such as indirect contamination, we study verbatim contamination as this form of contamination is more amenable to provable detection with statistical guarantees.

Reviewer comment (Experiments):
In Figure 1 the authors show test set contamination for BoolQ. But the examples there are unlabeled. Are the authors targeting unlabeled test set contamination i.e. the input is present in the pretraining data but not the label?

Author response:
Your second question is regarding the omission of labels In Figure 1. We omitted labels in the figure for readability, and the actual test does incorporate the labels.