Paper: Is Memorization Actually Necessary for Generalization? (ICLR-2024)

Reviewer comment (Experiments):
I feel fix-1 changes the entire story. First, "lack of model convergence" sounds like gradient descent hasn't converged, which is slightly misleading. Second and more importantly, it is natural to expect that changing the optimization objective or including different regularization techniques, architectures, etc would impact the findings. After all, the previous work focuses on one particular setup, which would of course make their findings difficult to transfer to new setups.

Author response:
We agree with the reviewer, that changing setups can change findings. And that is precisely our point. [1] made a general claim that memorization was necessary for generalization. If their claim is valid, changing the specifics of training pipeline (such as adding regularization) should not matter. However, we see that this is not the case. Our results show that using regularization alone reduces the impact of memorization on test accuracy. This means that the notion memorization is necessary for generalization is flawed.

Reviewer comment (Evaluation):
Because the setup has changed in certain ways, one may expect different train points to be memorized. Hence, the change in the accuracy should be computed after the points memorized *in this setup* are removed (not the ones reported in the reference paper).

Author response:
We thank the reviewer for this comment. Upon closer inspection, we saw that our setup memorized a strict subset of the points that were memorized by [1]. So our setup did not memorize any new points, just a smaller portion of the original.

Reviewer comment (Evaluation):
More than half of the original memorized points are included in the new training set. In other words, the post-pruning training dataset has a bigger size, which would naturally incur a smaller change in accuracy; hence comparing the reported accuracy change (0.54) against the original number (2.54) would not be fair.

Author response:
We thank the reviewer for this comment.
In the CIFAR-100 experiments, even though we only removed half the training points, the drop in accuracy is not at all proportional. The accuracy drop decreased from 2.54 to 0.54, a five-fold decrease.

Furthermore, in our CIFAR-10 experiments, we show that removing _all_ the 3,347 points from the memorized bucket still only incurs a rather small 0.4±0.2\% drop in accuracy (Figure 1 of the appendix.). This means our original findings still hold.

Reviewer comment (Writing):
_Entire sub-populations were removed alongside the memorized points._ <-- What is a sub-population?

Author response:
Please refer to Section 2.2.

Reviewer comment (Evaluation):
_original paper reported a drop of $2.54 \pm 0.20%$. However, we can see that after training the models to convergence, the value halved to $1.78 \pm 0.32%$._ <-- Does the number really halved?

Author response:
Thank you for this suggestion. We have corrected it.

Reviewer comment (Experiments):
Regarding errors stated in the paper — “Lack of Convergence: Models”: Why should models be trained to their maximum test set accuracy? You don't have access to the test set during training!

Author response:
Thank you for pointing this out. We had meant the validation set. We have fixed this in the paper.

Reviewer comment (Evaluation):
Regarding errors stated in the paper — "Sub-population Shift": The argument does not necessarily contradict the previous work because it aligns with what they have been arguing in their paper: memorization is necessary due to the long-tail distribution. Both mislabeled and sub-population data points are part of the long tail; therefore, it does not necessarily mean that the model memorized the points associated with sub-populations. They have high influence scores because they are rare in the data distribution, and the model exhibits a similar behavior of memorization.

Author response:
We appreciate the reviewer's input. The key idea we aimed to convey is that regardless of whether a set of points is memorized, eliminating entire sub-populations from the training data will lead models to misclassify test data from the same sub-population. This aligns with a fundamental principle in machine learning: models need exposure to data from the same distribution as the one they are tasked with predicting.

To illustrate, suppose there is a dataset with 95 white cats and 5 black cats. Whether the model memorizes information about black or white cats, removing the entire sub-population of black (or white) cats will result in decreased accuracy, as the model will misclassify black (or white) cats during testing. Therefore, the decline in accuracy is not solely attributable to memorization but rather to the removal of entire sub-populations.

Our research demonstrates that Feldman and Zhang's approach captures numerous sub-populations, not just the memorized points. Consequently, when they eliminate both the sub-populations and the memorized points, a decrease in accuracy is observed.

Reviewer comment (Theory):
There is an argument in the paper that the definition of memorization from the original work is incorrect, but there is no strong justification for this claim, and it is deferred to future research. I would suggest the authors first review their observations and results and then use them to formulate a more robust definition of memorization. In that case, your work would be better supported.

Author response:
Thank you for this comment. Our argument is an extension of the above response. Since the threshold to define memorization (set at 0.25) is arbitrary, and it captures a significant number of sub-populations. Does it mean that the captured points are even memorized? While reformulating the definition from memorization from scratch is definitely an interesting problem, our goal is to show that even if we accept the current definition, the final findings are not correct (i.e., memorization is necessary for generalization).

Reviewer comment (Experiments):
The critique regarding subpopulation shifts questions the validity of ignoring the central hypothesis of Feldman and Zhang's work, which emphasized the importance of memorizing near-singleton subpopulations. In particular, when you put examples from small subpopulations back into the original training set in order to allow for the test set examples from the same subpopulation to perform well, that defeats the entire purpose of the original paper where they claim that because some test set subpopulations have only one or two members in the training set, therefore they need to be memorized to perform well. So the results on the subpopulation shift are unfounded in my opinion, but I do agree with the results on model convergence and data leakage particularly, but I do not think that the effect is as prominent as the original claim of the paper.

Author response:
We thank the reviewer for this comment. The point we were trying to make is, whether or not a group of points is memorized, removing entire sub-populations from training data will cause models to mis-classify the test data from the same sub-population. This is in fact one of the most fundamental principles of machine learning: models must have observed data from the same distribution of those that they are asked to predict. 

For example, consider a dataset of 95 white cats and 5 black ones. Whether black cats or white cats are memorized, removing the entire sub-population of black (or white) cats will result in a drop in accuracy as models will misclassify the black (or white) cats at test-time. So the drop in accuracy can not be attributed to memorization, but instead, the removal of entire sub-populations.

And as we show in our work, Feldman and Zhang's method captures a large number of sub-populations, in addition to the memorized points. Therefore, when they remove the sub-populations, alongside the memorized points, they observe a drop in accuracy.

Reviewer comment (Experiments):
The paper fails to replicate a key analysis from Feldman and Zhang's work, specifically the varied levels of model accuracy against memorization thresholds, limiting the robustness of the current findings. I think that this work needs to do a proper analysis to replicate figure 2 of the paper by Feldman and Zhang where they have various levels of model accuracy with respect to memorization value threshold on ImageNet CIFAR-100 and MNIST datasets, and they show that removing random examples is better than removing memorized examples. In my opinion, this is a critical figure and a critical analysis that the authors need to present, whereas at this point, the authors have only presented results at one single point, which does not account for a solid justification that a prior method does not succeed.

Author response:
We want to thank the reviewer for this comment. We followed your suggestion and recreated the experiment across different thresholds on CIFAR-10. We used this data set for two reasons. 1) to show that our results are generalizable 2) Because CIFAR-10 models are faster to train than CIFAR-100 models.
We show our results below:
| Threshold | Remove Random (Accuracy %) | Remove Memorized (Accuracy %) | Difference (Accuracy %) |
|:---------:|:--------------------------:|:-----------------------------:|:-----------------------:|
|    0.25   |         94.81±0.15         |           94.4±0.15           |        0.4±0.2        |
|    0.35   |         95.06±0.15         |           94.8±0.13           |        0.29±0.14        |
|    0.45   |         95.08±0.17         |           94.91±0.13          |         0.15±0.2        |
|    0.55   |         95.07±0.13         |           94.98±0.14          |        0.14±0.17        |
|    0.65   |         95.03±0.13         |           94.97±0.13          |        0.07±0.22        |
|    0.75   |         95.13±0.11         |           95.04±0.13          |        0.11±0.15        |
|    0.85   |         95.02±0.13         |           95.05±0.11          |        -0.09±0.13       |
|    0.90   |         95.16±0.17         |           95.11±0.13          |        0.05±0.23        |

The Table shows the test set accuracies for models when the memorized points (Memorized Bucket from Table 2 that does not suffer from sub-population and near-duplicate issues) were removed, when the same number of random points are removed from the training data, and the mean difference (with standard deviation) between the two. We can make a few observations:
1. The difference in accuracies is very small.
2. Even after removing all the memorized points at the 0.25 threshold (a total of 3,347), there is a less than half a percent drop accuracy (of 0.4±0.2\%).
3. For most memorization thresholds above 0.35, the mean difference is smaller than the standard deviation. This means that drop in accuracy when removing memorized points is not statistically significant.

As a consequence of these three observations, we can conclude memorized points do not significantly contribute towards test accuracy. And therefore, memorization is not necessary for generalization. We encourage the reviewer to refer to our results in the newly added Figure 4 (in the Supplementary Material) of the paper.

Reviewer comment (Experiments):
Would the accuracy loss remain lower if an additional 516 random data points replaced the omitted 499 examples, keeping the training set size constant?

Author response:
We thank the reviewer for this comment. We follow the same methodology laid out in the original Feldman and Zhang's work so that it's an apple-to-apple comparison. Specifically, we always compare a model trained on a reduced dataset (with memorized points removed) with another model with an equal number of random points removed as well.

Finally, thank you for identifying the issue with ICLR format. We fixed it and this saved us a few lines :)

Reviewer comment (Experiments):
Can the authors replicate the original paper's results across different memorization thresholds, particularly on the CIFAR-100 dataset, and possibly on other datasets as well? This would provide a more comprehensive view of the memorization effects at varying levels.

Author response:
We want to thank the reviewer for this comment. We followed your suggestion and recreated the experiment across different thresholds on CIFAR-10. We used this data set for two reasons. 1) to show that our results are generalizable 2) Because CIFAR-10 models are faster to train than CIFAR-100 models.
We show our results below:
| Threshold | Remove Random (Accuracy %) | Remove Memorized (Accuracy %) | Difference (Accuracy %) |
|:---------:|:--------------------------:|:-----------------------------:|:-----------------------:|
|    0.25   |         94.81±0.15         |           94.4±0.15           |        0.4±0.2        |
|    0.35   |         95.06±0.15         |           94.8±0.13           |        0.29±0.14        |
|    0.45   |         95.08±0.17         |           94.91±0.13          |         0.15±0.2        |
|    0.55   |         95.07±0.13         |           94.98±0.14          |        0.14±0.17        |
|    0.65   |         95.03±0.13         |           94.97±0.13          |        0.07±0.22        |
|    0.75   |         95.13±0.11         |           95.04±0.13          |        0.11±0.15        |
|    0.85   |         95.02±0.13         |           95.05±0.11          |        -0.09±0.13       |
|    0.90   |         95.16±0.17         |           95.11±0.13          |        0.05±0.23        |

The Table shows the test set accuracies for models when the memorized points (Memorized Bucket from Table 2 that does not suffer from sub-population and near-duplicate issues) were removed, when the same number of random points are removed from the training data, and the mean difference (with standard deviation) between the two. We can make a few observations:
1. The difference in accuracies is very small.
2. Even after removing all the memorized points at the 0.25 threshold (a total of 3,347), there is a less than half a percent drop accuracy (of 0.4±0.2\%).
3. For most memorization thresholds above 0.35, the mean difference is smaller than the standard deviation. This means that drop in accuracy when removing memorized points is not statistically significant.

As a consequence of these three observations, we can conclude memorized points do not significantly contribute towards test accuracy. And therefore, memorization is not necessary for generalization. We encourage the reviewer to refer to our results in the newly added Figure 4 (in the Supplementary Material) of the paper.

Reviewer comment (Evaluation):
"and therefore, [memorization] is not necessary for generalization." I'm curious why the authors claim this if the effect still exists. Is it not statistically significant away from being 0? I would rephrase the abstract to be a bit more conservative.

Author response:
1. "I would rephrase the abstract to be a bit more conservative."

    Thank you for the suggestion. We reworded the abstract as per your request.

Reviewer comment (Experiments):
I didn't find in [1] nor in this paper how training accounts for the removal of data, i.e., in Table 2, are all models trained on the same number of points? Doing otherwise would seem... incorrect?

Author response:
2. "I didn't find in [1] nor in this paper how training accounts for the removal of data..."

    In Table 2, the models in the first two columns (Original Result and Model Convergence) are trained on exactly the \textit{same} points. Specifically, around 1,000 points were removed from the dataset. The change in test accuracy (due to memorization removal) goes down from 2.54 ± 0.20\% (first column) to 1.78 ± 0.32\% (second column). This shows that training models to convergence alone significantly reduces the impact of memorization on the test set (i.e., memorization decreases while generalization increases). Similarly, the last two columns (Leakage+Shift Bucket and Memorized Bucket) were trained on approximately the same number of points, with 500 points removed. This shows that the removal of sub-populations and duplicates from the training data was responsible for the drop in accuracy, instead of the memorized points themselves.