Paper: Elephants Never Forget: Testing Language Models for Memorization of Tabular Data (ICLR-2024)

Reviewer comment (Experiments):
This work does not provide strong evidence supporting the validity of their proposed approaches. I think one main takeaway of this paper is that some models are pretrained with some datasets, so their performance is not indicative of. But this takeaway is based on the validity of their proposed approaches. I think the authors need to address this more.

Author response:
With regard to the validity of our tests, please see the global comment ‘The validity and relevance of our tests’. 

In addition, note that we have been careful to use (1) datasets that are public and overwhelmingly likely to have been used in pre-training, and (2) datasets that are very unlikely or impossible to have been used in pre-training.  There is a clear difference in the results for these two different kinds of datasets.  We also intentionally include FICO as a dataset where there is significant public discussion of the data and some of its properties, but where the data table itself has not been made public. As might be expected, on this dataset our tests detect prior experience but fail to detect verbatim memorization.

Reviewer comment (Writing):
The descriptions of the testing approaches are vague and not rigorous. Writing down the testing approaches with simple math equations could help. For example, in page 6, I can’t understand what it means by “we can perform a t-test between the similarity of model completions with actual vs. random rows."

Author response:
We agree that there is room for improvement and will rewrite the sections describing the different tests. For details on the different tests, please also see Supplement Section E which gives the prompt structures of all the tests, and the revised Supplement Section F which provides the test results in a more explicit form.

Reviewer comment (Experiments):
The authors (claims to) show data contamination exists in some datasets. However, I am not sure whether those datasets are commonly used to benchmark the language model. Thus I am not sure whether the findings are important (if they are valid).

Author response:
Yes, the datasets in our paper have been used to benchmark language models. Please see the papers referenced under ‘The relevance of our work’ in the global comment.

Reviewer comment (Evaluation):
Evaluation metric for knowledge, learning, and memorization is unclear. In Table 1, the evaluation results are categorized into three categories (✓, X, and ?), but the metric for the categorization is not given. It is probably a better idea to show the raw values (e.g., accuracy) than using categories to give the reader a direct comprehension of the degree of memorization on each dataset. Notations such as "✓" could be misleading as it may be confused as perfect memorization. The appendix gives raw accuracies for Row Completion Test, Feature Completion Test, and First Token Test, why raw accuracies for Feature Names, Feature Values, and Header Test are not provided as well?

Author response:
*“raw accuracies for Feature Names, Feature Values, and Header Test are not provided ”*

We empirically found that the language models either (1) correctly list all the feature names, or (2) completely fail to list the feature names. The same holds true for the Feature Values test (third row of Table 3) and also for the header test. This is why we decided to depict the test results qualitatively. We agree that we did not make this clear enough in the paper. The revised Supplement Section F now depicts detailed results of the different tests.

Reviewer comment (Evaluation):
The differentiation between learning and memorization is not clear: the authors use feature distributions to examine learning, but memorization can also result in a high similarity of the generated data's feature distributions to the original data. Learning is defined as the model's ability to perform tasks in the current paper, but task performance is heavily affected by memorization and may fail to reflect true learning. Even with considerable discussion, the paper does not seem to arrive at a conclusion about how learning can be clearly assessed.

Author response:
*“the authors use feature distributions to examine learning, but memorization can also result in a high similarity of the generated data's feature distributions to the original data. Learning is defined as the model's ability to perform tasks in the current paper, but task performance is heavily affected by memorization and may fail to reflect true learning. Even with considerable discussion, the paper does not seem to arrive at a conclusion about how learning can be clearly assessed.”*

Great point. We find it important to introduce the learning/memorization distinction to make clear that there are many different tasks that the model might be able to perform with the data that are not indicative of memorization (see also ‘The distinction between Knowledge, Learning and Memorization’ above). We are reasonably confident to have identified cases where the LLMs have learned, but not memorized. The reviewer asks whether we are able to test for learning under the assumption that there is memorization. This is a very interesting question (and also a question that is hard to answer). With the tests that we propose in this paper, we do not claim to be able to do this.

Reviewer comment (Evaluation):
Evaluation of memorization needs to take the nature of data fields into consideration. Some data fields in the tabular dataset are considerably harder to memorize verbatim or to predict exactly (such as measurement values) than other simpler fields (categorical values such as sex, occupation, nation). For numerical values, it may be more reasonable to measure the relative distance from the predicted value to the true value than using exact match (perhaps in a similar vein as the "first token test" in the paper but more principled). Under the current evaluation protocol, it is likely that datasets containing more easy fields are more likely to be judged as memorized. To compare the degree of memorization across datasets, it seems necessary to perform some kind of "normalization" before measuring memorization, for example, selecting a fixed number of categorical and numerical fields from each dataset. Results in Table 3 could suffer from this limitation as well.

Author response:
*“Evaluation of memorization needs to take the nature of data fields into consideration. [...] For numerical values, it may be more reasonable to measure the relative distance from the predicted value to the true value than using exact match (perhaps in a similar vein as the "first token test" in the paper but more principled). Under the current evaluation protocol, it is likely that datasets containing more easy fields are more likely to be judged as memorized. To compare the degree of memorization across datasets, it seems necessary to perform some kind of "normalization" before measuring memorization, for example, selecting a fixed number of categorical and numerical fields from each dataset. Results in Table 3 could suffer from this limitation as well.”*

This is an interesting observation, but we tend to disagree. In fact, it is precisely the exact matches between rare numerical values in the data and the generations by the LLM that allow us to develop valid tests for memorization. The fact that tabular datasets often contain such values (as opposed to free-form text) is what makes tabular data such an interesting testbed for memorization (and this is also why the results in Table 3 are valid). For a dataset that only contains easy fields such as categorical variables with few different values, it would be very hard to distinguish a powerful learner and memorization. Measuring the relative distance from the predictive value to the true value could result in a test for learning, but would be unlikely to result in a valid test for memorization.

Reviewer comment (Evaluation):
Evaluation of memorization needs to be evaluated separately for the training and test split. It may be possible that the training sets are memorized more than the test set due to more exposure on the internet. Memorizing the test set definitely compromises evaluation, but memorizing the training set may not always compromise evaluation.

Author response:
*“Evaluation of memorization needs to be evaluated separately for the training and test split. ”*

Very good point. The revised Supplement Section F now contains separate evaluations for the datasets where separate train and tests splits are available. We do not find any significant differences.

Reviewer comment (Evaluation):
Connection between memorization and downstream performance is not reliably established. The main observation from Section 5 is that for datasets with a high degree of memorization, LLM performs better than decision tree and logistic regression, while for datasets with a low degree of memorization the reverse is true. Such observation alone may not be sufficient to conclude that memorization compromises evaluation, because there is no evidence that LLM cannot perform better than decision tree and logistic regression under no memorization. It would be much better to solicit new test sets for the tasks to use in evaluation, which can be used to show exactly how much performance gap is caused by memorization. In case finding new examples is difficult, perhaps one can modify the values of the fields known to be irrelevant to the label in existing examples, and that may break the reliance on memorization in LLMs.

Author response:
*“Connection between memorization and downstream performance is not reliably established. [...] It would be much better to solicit new test sets for the tasks to use in evaluation, which can be used to show exactly how much performance gap is caused by memorization.”*

The SOTA accuracy for Kaggle Titanic using ensembling is approximately 0.85 (see https://www.kaggle.com/code/pliptor/how-am-i-doing-with-my-score/report). GPT-4 achieves an accuracy of 0.98. We think that this is fairly indicative of memorization. We agree, however, that the approach in Section 5 is only a first step, and that one could think of more advanced evaluation strategies (such as the one suggested by the reviewer). In general, however, there are many possible downstream tasks, and it is not possible for tests of memorization and prior exposure to consider (or even be relevant) for all of these.  Because of this, we believe it is important to have tests that are also reasonably task-agnostic.

Reviewer comment (Evaluation):
"We emphasize the importance of verifying data contamination before applying LLMs": the implication of data contamination is not reliably demonstrated in Section 5. Also, from the current discussion, it is not very clear how to interpret the test results on knowledge, learning, and memorization together. For example, if knowledge and learning show positive results and memorization show negative results, should we conclude that there is data contamination or not? And could the performance on downstream tasks be trusted in this situation? It can be argued that knowledge and learning will not directly compromise evaluation on downstream tasks, so there may not be as much need to evaluate them compared to memorization. I would suggest allocating more space in the paper for extended experiments and discussions on memorization, which is the ultimate reason why people are concerned about data contamination.

Author response:
*“if knowledge and learning show positive results and memorization show negative results, should we conclude that there is data contamination or not? And could the performance on downstream tasks be trusted in this situation?”*

The setting that we are interested in here is when a paper proposes a new prompting or fine-tuning strategy, and then demonstrates that this strategy leads to improved performance on some datasets. If the model already shows positive results for learning on these datasets, then this raises the question of whether the results are really due to the new prompting/fine-tuning strategy, or reliant on the fact that the LLM has already learned about the dataset during pre-training. Even when testing does not show memorization, prior learning can significantly bias the results that are obtained in experiments by allowing the LLM to perform advanced tasks with the data distribution that it would not have been able to perform on a novel dataset without prior exposure. Papers that study tabular learning with LLMs are, in most cases, purporting to show results that would be obtained without prior data contamination (see also ‘The relevance of our work’ above).

Reviewer comment (Experiments):
The paper only studies ChatGPT-3.5 and 4. Those are very likely to be strongly correlated in terms of the data used, which harms the representativeness of the study.

Author response:
We have now extended our code and prompts to work with arbitrary language models (some adaptation of prompts is required between base models and chat models, and the capabilities of chat models to follow instructions vary). The revised Supplement Section F contains results with Llama2-7b and Llama2-70b.

Reviewer comment (Experiments):
As there is no ground-truth knowledge on whether a particular dataset was seen at training, it is impossible to strictly verify the findings. Including an LM trained on a known dataset would allow us to verify the used methods.

Author response:
We agree that this is an interesting direction in which our work could be extended. Unfortunately, the LLMs for which the training data and schedule are transparent are rather small (<100b Parameters). At the same time, memorization is known to occur especially for very large models. This is evident, for example, in Supplement Table 4, which shows that GPT-4 has memorized significantly more than GPT-3.5. 
It is also evident in the Figures in the revised Supplement Section F, where it can be seen that Llama2-7b is the model that has memorized the least.

Reviewer comment (Theory):
Another related issue: the work is mostly relevant when we consider closed-data models w/ a black-box API access. This scenario reflects a dominant situation at the moment, but it is not given that this will not/should not change.

Author response:
We completely agree. We also believe that this is a very important and relevant scenario (see also ‘The validity and relevance of our tests’ above).

Reviewer comment (Writing):
"we also identify a regime where the language model reproduces important statistics of the data, but fails to reproduce the dataset verbatim": It's not clear to me what this statement means. See also Question 1 below.

Author response:
Here we are referring to the behavior of GPT-3.5 on the California Housing dataset. On this dataset, the samples provided by the model match the correlations in the training data (Figure 3). At the same time, the average n-gram match between the samples and the training data is only 3.1/10 (Table 3) and there is no evidence for verbatim memorization except for the header of the dataset (Table 1).

Reviewer comment (Experiments):
Figure 3: Why are some results with gpt-3.5 and some with gpt-4?

Author response:
Figure 3 depicts selected examples. The full results, for both gpt-3.5 and gpt-4, are in Supplement D.