Paper: Memorization Through the Lens of Curvature of Loss Function Around Samples (ICLR-2024)

Reviewer comment (Evaluation):
Limited to memorization: The baseline by Feldman & Zhang calculates both the memorization scores for the training examples, and influence scores for test examples. There is no natural extension of curvature as a metric to influence scores.

Author response:
Authors: We thank the reviewer for helping us think in this direction. We feel that using contrastive loss and calculating curvature on this loss might help us get influence scores. We leave this exploration for future work, but thank the reviewer for identifying another application of our method. We want to highlight that if we were to use contrastive loss, most methods mentioned in the prior work section would not be applicable (SSFT [Maini1], CL [NorthCutt1], Inconfidence [Carlini1], FZ [Feldman2], AUM [Pleiss1], Learning Time) since they require logits to work. However, since curvature only requires a loss, we would still be able to capture the scores.

Reviewer comment (Experiments):
Baseline: No comparison with any baseline is presented. Simple baselines like learning time, second split forgetting time etc. should be compared to.

Author response:
Authors: We have added both these comparisons to Table 2. From Table 2, we can see that Curvature outperforms SSFT for both MNIST and CIFAR-10 for all percentages of mislabeling considered. However, SSFT performs better than Curvature for CIFAR-100, but as the authors note in their text, SSFT can be combined with other methodologies. We show that combining SSFT and Curvature results in a combination that outperforms both SSFT and Curvature independently and significantly. We have provided the same table below for reference, with the experiments for LT and SSFT performed using the author's github repo. Additionally, we also note that SSFT results shown in the paper fail to capture the novel failure mode we caught, that of duplicate samples with differing labels.      

&nbsp;     
We thank the reviewer for pointing us to a strong baseline, and for helping us strengthen out paper by improving our results by combining it with SSFT. However, we would still like to reemphasize that our paper focuses on establishing curvature as a method for measuring memorization, and the application of detecting mislabeled samples is shown merely as corroboration that curvature does indeed detect memorization. There are not many baselines available for memorization, but we compare our method against the most established method currently, FZ scores, and show that we capture failure modes that they fail to, despite being 3 orders of magnitude more efficient. 

|  Dataset |   Method  | Corruption |         |         |        |        |        |
|:--------:|:---------:|:----------:|---------|---------|--------|--------|--------|
|          |           | 1%         | 2%      | 4%      | 6%     | 8%     | 10%    |
| MNSIT    | Inconf.   | 99.4\%     | 99.0\%  | 98.4\%  | 97.7\% | 97.1\% | 96.1\% |
|          | CL        | 99.7\%     | 99.3\%  | 99.1\%  | 98.9\% | 98.9\% | 98.9\% |
|          | SSFT      | 99.9\%     | 99.9\%  | 99.8\%  | 99.7\% | 99.7\% | 99.5\% |
|          | LT        | 98.7\%     | 99.5\%  | 98.4\%  | 98.5\% | 97.7\% | 97.3\% |
|          | Curv      | 100.0\%    | 100.0\% | 99.9\%  | 99.9\% | 99.9\% | 99.9\% |
| CIFAR10  | Inconf.   | 84.2\%     | 82.5\%  | 81.8\%  | 81.6\% | 81.5\% | 81.5\% |
|          | CL        | 92.4\%     | 93.1\%  | 93.4\%  | 93.4\% | 91.7\% | 93.4\% |
|          | SSFT      | 94.5\%     | 94.1\%  | 93.2\%  | 92.5\% | 91.6\% | 90.0\% |
|          | LT        | 87.3\%     | 82.5\%  | 84.0\%  | 83.4\% | 82.2\% | 83.0\% |
|          | Curv      | 97.4\%     | 96.6\%  | 95.5\%  | 94.4\% | 94.1\% | 92.9\% |
|          | Curv SSFT | 97.5\%     | 96.8\%  | 96.2\%  | 96.1\% | 95.2\% | 94.8\% |
| CIFAR100 | Inconf.   | 85.0\%     | 84.0\%  | 83.5\%  | 83.6\% | 83.6\% | 83.4\% |
|          | CL        | 83.1\%     | 84.2\%  | 85.3\%  | 86.3\% | 84.3\% | 84.6\% |
|          | SSFT      | 94.8\%     | 93.7\%  | 93.2\%  | 91.7\% | 91.9\% | 91.0\% |
|          | LT        | 85.2\%     | 85.2\%  | 81.8\%  | 80.8\% | 79.8\% | 78.5\% |
|          | Curv      | 90.9\%     | 89.6\%  | 88.3\%  | 86.8\% | 85.7\% | 84.3\% |
|          | Curv SSFT | 96.0\%     | 94.5\%  | 94.2\%  | 93.1\% | 92.9\% | 91.9\% |

Reviewer comment (Experiments):
Results of comparison with other simple baselines?

Author response:
Authors: We have added both these comparisons to Table 2. From Table 2, we can see that Curvature outperforms SSFT for both MNIST and CIFAR-10 for all percentages of mislabeling considered. However, SSFT performs better than Curvature for CIFAR-100, but as the authors note in their text, SSFT can be combined with other methodologies. We show that combining SSFT and Curvature results in a combination that outperforms both SSFT and Curvature independently and significantly. We have provided the same table below for reference, with the experiments for LT and SSFT performed using the author's github repo. Additionally, we also note that SSFT results shown in the paper fail to capture the novel failure mode we caught, that of duplicate samples with differing labels.      

&nbsp;     
We thank the reviewer for pointing us to a strong baseline, and for helping us strengthen out paper by improving our results by combining it with SSFT. However, we would still like to reemphasize that our paper focuses on establishing curvature as a method for measuring memorization, and the application of detecting mislabeled samples is shown merely as corroboration that curvature does indeed detect memorization. There are not many baselines available for memorization, but we compare our method against the most established method currently, FZ scores, and show that we capture failure modes that they fail to, despite being 3 orders of magnitude more efficient. 

|  Dataset |   Method  | Corruption |         |         |        |        |        |
|:--------:|:---------:|:----------:|---------|---------|--------|--------|--------|
|          |           | 1%         | 2%      | 4%      | 6%     | 8%     | 10%    |
| MNSIT    | Inconf.   | 99.4\%     | 99.0\%  | 98.4\%  | 97.7\% | 97.1\% | 96.1\% |
|          | CL        | 99.7\%     | 99.3\%  | 99.1\%  | 98.9\% | 98.9\% | 98.9\% |
|          | SSFT      | 99.9\%     | 99.9\%  | 99.8\%  | 99.7\% | 99.7\% | 99.5\% |
|          | LT        | 98.7\%     | 99.5\%  | 98.4\%  | 98.5\% | 97.7\% | 97.3\% |
|          | Curv      | 100.0\%    | 100.0\% | 99.9\%  | 99.9\% | 99.9\% | 99.9\% |
| CIFAR10  | Inconf.   | 84.2\%     | 82.5\%  | 81.8\%  | 81.6\% | 81.5\% | 81.5\% |
|          | CL        | 92.4\%     | 93.1\%  | 93.4\%  | 93.4\% | 91.7\% | 93.4\% |
|          | SSFT      | 94.5\%     | 94.1\%  | 93.2\%  | 92.5\% | 91.6\% | 90.0\% |
|          | LT        | 87.3\%     | 82.5\%  | 84.0\%  | 83.4\% | 82.2\% | 83.0\% |
|          | Curv      | 97.4\%     | 96.6\%  | 95.5\%  | 94.4\% | 94.1\% | 92.9\% |
|          | Curv SSFT | 97.5\%     | 96.8\%  | 96.2\%  | 96.1\% | 95.2\% | 94.8\% |
| CIFAR100 | Inconf.   | 85.0\%     | 84.0\%  | 83.5\%  | 83.6\% | 83.6\% | 83.4\% |
|          | CL        | 83.1\%     | 84.2\%  | 85.3\%  | 86.3\% | 84.3\% | 84.6\% |
|          | SSFT      | 94.8\%     | 93.7\%  | 93.2\%  | 91.7\% | 91.9\% | 91.0\% |
|          | LT        | 85.2\%     | 85.2\%  | 81.8\%  | 80.8\% | 79.8\% | 78.5\% |
|          | Curv      | 90.9\%     | 89.6\%  | 88.3\%  | 86.8\% | 85.7\% | 84.3\% |
|          | Curv SSFT | 96.0\%     | 94.5\%  | 94.2\%  | 93.1\% | 92.9\% | 91.9\% |

Reviewer comment (Experiments):
How does the value of h and n in the curvature calculation affect the results?

Author response:
Authors: We have added the hyperparameter tuning results in the appendix and provide the same in the table below. The hyperparameter tuning results show the cosine similarity match between FZ scores and curvature score for all (CS) and top 5000 (Top-K CS) examples on CIFAR10 dataset. From the Table we have the following take away, the proposed method is pretty robust to hyper parameter choices. Next, we see that the choice of n is less important than the choice of h. But overall, the scores are very robust.

|       h      |      n  |      Top-K CS  |      CS   |
|--------------|---------|----------------|-----------|
|      0.01    |     5   |     0.89       |     0.80  |
|      0.001   |     5   |     0.90       |     0.82  |
|      0.0001  |     5   |     0.90       |     0.82  |
|      0.01    |     10  |     0.89       |     0.80  |
|      0.001   |     10  |     0.90       |     0.82  |
|      0.0001  |     10  |     0.90       |     0.82  |
|      0.01    |     20  |     0.89       |     0.80  |
|      0.001   |     20  |     0.90       |     0.82  |
|      0.0001  |     20  |     0.90       |     0.82  |

Reviewer comment (Experiments):
This paper could benefit from a stronger demonstration of the utility of the proposed metric, beyond the simple corrupted label prediction experiment.

Author response:
We want to emphasize that in this paper we wanted to introduce the idea of measuring memorization with input loss curvature. We believe that extreme memorization and overfitting is one of the most significant insights into deep neural networks, as evidenced by the extensive work done in references [Zhang1, Feldman1, Feldman2].  We establish curvature as a reliable and efficient metric to measure memorization, and show both quantitative and qualitative corroborations of the reliability of curvature as a memorization metric.     
&nbsp;


For practical applications, we show three in the paper which we believe are quite significant. One is detecting mislabeled samples, and that is an area of significant research [Maini1, Northcutt1, Carlini1]. Secondly, we show an as of yet unobserved failure mode on CIFAR100 and ImageNet, two of the most widely used datasets to test different algorithms for vision algorithms, that of duplicate samples with differing labels. This shows how curvature can be used to find failure modes especially for datasets used in the industry that may not have been vetted by many research works. Thirdly, we also note the outliers in FashionMNIST dataset show a bias towards high contrast clothes, despite them being not more prototypical of clothing in real life. This highlights that our analysis can reveal biases present in the dataset.    
&nbsp;


There are many downstream applications of being able to measure and study memorization, and the reviewer lists some that we are currently working on as extensions. However, this work forms the bases of all those future explorations, which we and hopefully the community will pursue. We feel that this manuscript is self-contained in its claim, justification and example applications, and hope that the reviewer would reconsider the paper on its own merit, instead of the potential applications that can exist as a result of this manuscript.      
&nbsp;

References:      
Zhang1 - Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. “Understanding deep learning requires rethinking generalization”. In ICLR 2017,  

Feldman1 - Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954–959, 2020. 

Feldman2 - Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881–2891, 2020. 

Maini1 - Pratyush Maini, Saurabh Garg, Zachary Lipton, and J Zico Kolter. Characterizing datapoints via second-split forgetting. Advances in Neural Information Processing Systems, 35:30044–30057, 2022. 

NorthCutt1 - Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411, 2021a. 

Carlini1 - Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv:1910.13427, 2019a.

Reviewer comment (Theory):
Could the curvature be extended to compute some kind of influence metrics from training to test examples like the FZ scores or other data valuation metrics?

Author response:
Authors:  We thank the reviewer for helping us think in this direction. We feel that using contrastive loss and calculating curvature on this loss might help us get influence scores. We leave this exploration for future work, but thank the reviewer for identifying another application of our method. We want to highlight that if we were to use contrastive loss, most methods mentioned in the prior work section would not be applicable (SSFT [Maini1], CL [NorthCutt1], Inconfidence [Carlini1], FZ [Feldman2], AUM [Pleiss1], Learning Time) since they require logits to work. However, since curvature only requires a loss, we would still be able to capture the scores.  


Feldman2 - Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881–2891, 2020. 

Maini1 - Pratyush Maini, Saurabh Garg, Zachary Lipton, and J Zico Kolter. Characterizing datapoints via second-split forgetting. Advances in Neural Information Processing Systems, 35:30044–30057, 2022. 

NorthCutt1 - Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411, 2021a. 

Carlini1 - Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv:1910.13427, 2019a. 

Pleiss1 - Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:17044–17056, 2020

Reviewer comment (Experiments):
Unlike other metrics that measures in the output, logits, or weight space, the proposed metric probably depend more on the geometry of the inputs. Therefore, I think it is very valuable if the paper could present studies with different input domains.

Author response:
Authors: We agree with the reviewer. We think this discussion highlights the advantages of our method, namely it only requires being able to calculate loss, instead of logits which may not always be available. Thus, it can be extended to different domains and application with relative ease. We hope it will encourage the community to explore input loss curvature for different domains.

Reviewer comment (Experiments):
Do you observe the same behavior if the input-vs-logit space analysis in Fig. 7 is performed on a real dataset instead of a synthetic one?

Author response:
Authors: Visualizing the input vs logit space is tricky for real datasets due to the higher dimensional nature of both the input data (3072 dimensional for CIFAR100) and of logits (100 for CIFAR100). However, we see the same curvature trend for our toy dataset and CIFAR100 and ImageNet datasets. As a proxy for logit space one can look at softmax confidence which is used in [NorthCutt1], and learning time. These studies tend to suggest that logit margin maximization tends to happen on real datasets as well. However, it would be hard to visualize as clearly as in our toy dataset. 

&nbsp;

NorthCutt1 - Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411, 2021a.

Reviewer comment (Experiments):
It is described in the appendix how the two hyperparameters h and n are tuned. Can you show some hyperparameter tuning curves? In particular, I'm interested in not only what values are chosen, but also how robust the proposed metric is to hyperparameter changes.

Author response:
Authors: We have added the hyperparameter tuning results in the appendix and provide the same in the table below. The hyperparameter tuning results show the cosine similarity match between FZ scores and curvature score for all (CS) and top 5000 (Top-K CS) examples on CIFAR10 dataset. From the Table we have the following take away, the proposed method is pretty robust to hyper parameter choices. Next, we see that the choice of n is less important than the choice of h, and would recommend not using an h of less than 0.001.    
&nbsp;             

|       h      |      n  |      Top-K CS  |      CS   |
|--------------|---------|----------------|-----------|
|      0.01    |     5   |     0.89       |     0.80  |
|      0.001   |     5   |     0.90       |     0.82  |
|      0.0001  |     5   |     0.90       |     0.82  |
|      0.01    |     10  |     0.89       |     0.80  |
|      0.001   |     10  |     0.90       |     0.82  |
|      0.0001  |     10  |     0.90       |     0.82  |
|      0.01    |     20  |     0.89       |     0.80  |
|      0.001   |     20  |     0.90       |     0.82  |
|      0.0001  |     20  |     0.90       |     0.82  |

Reviewer comment (Theory):
The main weakness of the paper is that it does not provide a comprehensive discussion on what it means to say that an example is being memorized. For example, the work of Feldman & Zhang calls those points to be memorized where the model only performs well when those points are present in the training set. What is the precise notion of memorization that this paper aims to capture?

Author response:
This is a very relevant question, and we attempt to share the way we think of our metric with the reviewer here. The memorization that our manuscript is aiming to capture is conceptually quite similar to what Feldman and Zhang intend to capture with their metric. Specifically, both methods ask how sensitive the model is to a particular input. Feldman & Zhang propose to answer this by looking at probabilities in presence and absence of the sample, and we propose to answer this by measuring the sensitivity of the model to perturbation in the input, calculated via curvature.  
&nbsp;     


 A far more intuitive understanding of why we chose curvature arises from correlating the curvature of loss around a sample to the curvature of decision boundary around the sample in the input space. The classical machine learning illustration of overfitting, such as the one found [here](https://media.geeksforgeeks.org/wp-content/cdn-uploads/20190523171258/overfitting_2.png), shows that the network overfits by bending the decision boundary around certain sample points in the input space, indicating that these samples are being memorized by the network. In the example shown, the samples that would be counted as most memorized, would be the ones with the curviest decision boundaries around them, establishing a straightforward link between memorization score and curvature of decision boundary around the sample.  
&nbsp;     


Previous work such as [Dong1, Srinivas1] has shown the link between curvature of decision boundary and curvature of loss, in that regularizing loss curvature results in a flatter decision boundary. This link means that our metric estimates the curvature of the decision boundary around a sample in the input space, which makes intuitive sense as a measure of memorization.       
&nbsp;         



If the reviewer feel  that this would add clarity and intuitive visualization of our metric to the readers, we are happy to add this to the manuscript.      

&nbsp;         
References:       
Dong1 - Bin Dong, Haocheng Ju, Yiping Lu, and Zuoqiang Shi. "CURE: Curvature regularization for missing data recovery." SIAM Journal on Imaging Sciences 13, no. 4 (2020): 2169-2188.       

Srinivas1 Suraj Srinivas, Kyle Matoba, Himabindu Lakkaraju, and Francois Fleuret. "Efficient training of low-curvature neural networks." Advances in Neural Information Processing Systems 35 (2022): 25951-25964.

Reviewer comment (Experiments):
Most of the experiments in the paper are tied to a single model. It would be interesting to see how the proposed curvature-based memorization score behaves as one changes model architecture and/or size. Are key takeaways from this paper robust to such variations?

Author response:
In addition to ResNet18 we provide results on VGG 13 (with Batch Norm), DenseNet121 and MobileNetV2 for CIFAR100, trained with a weight decay of 1e-4. Curvature scores from each network is able to obtain a good match with FZ scores.  We also provide cosine similarity match cross architecture (between arch1 and arch2) for all the samples and for top 5000 FZ score samples in CIFAR100 in the following tables. These results have also been added to the appendix of the paper. The results suggest that curvature is quite robust, since we observe a good match between FZ and curvature score across different architecture and model sizes, including architectures withlower test set accuracy. 

    &nbsp;     

    |       Architecture  |      Test Set Acc  |      Train Acc  |      CS Top 5K FZ Score  |      CS   |
    |---------------------|--------------------|-----------------|--------------------------|-----------|
    |      ResNet18       |     75.09%         |     99.99%      |     0.90                 |     0.82  |
    |      DenseNet121    |     76.09%         |     99.98%      |     0.83                 |     0.74  |
    |      VGG13 BN       |     69.81%         |     99.98%      |     0.87                 |     0.75  |
    |      Mobilenet V2   |     66.21%         |     99.97%      |     0.90                 |     0.77  |

    &nbsp;      

    Cosine similarity between curvature scores of various architectures for top 5000 FZ score samples (Top-K CS) on CIFAR100 dataset (below).

    |                    |      ResNet18  |      VGG13 BN  |      DenseNet121  |      Mobilenet V2  |
    |--------------------|----------------|----------------|-------------------|--------------------|
    |      ResNet18      |     1.00       |     0.89       |     0.88          |     0.92           |
    |      VGG13 BN      |     0.89       |     1.00       |     0.83          |     0.88           |
    |      DenseNet121   |     0.88       |     0.83       |     1.00          |     0.85           |
    |      Mobilenet V2  |     0.92       |     0.88       |     0.85          |     1.00           |

    &nbsp;     

    Cosine similarity between curvature scores of various architectures for all samples on CIFAR100 dataset (below).
    |                    |      ResNet18  |      VGG13 BN  |      DenseNet121  |      Mobilenet V2  |
    |--------------------|----------------|----------------|-------------------|--------------------|
    |      ResNet18      |     1.00       |     0.83       |     0.83          |     0.87           |
    |      VGG13 BN      |     0.83       |     1.00       |     0.76          |     0.83           |
    |      DenseNet121   |     0.83       |     0.76       |     1.00          |     0.78           |
    |      Mobilenet V2  |     0.87       |     0.83       |     0.78          |     1.00           |

Reviewer comment (Novelty):
The paper claims that their proposed memorization score is significantly cheaper to compute compared to existing scores like Feldman & Zhang’s memorization score (see a question on this below). However, it appears that the *efficient* loss curvature calculation follows from the prior work, limiting the technical novelty of the contributions.

Author response:
Thank you for your question. Garg and Roy [R2] indeed was a source of inspiration for this work. We found their analysis of curvature for creating coresets interesting, and were inspired to see if we could use it as a measure of memorization. We found that their method did not work out of the box, and we had to make significant changes to establish a good metric, in essence reverting back to Hutchinson’s trace estimator form for calculating curvature [Hutchinson1]. Importantly, *their scores also failed to capture the failure mode of duplicated samples with differing labels*. We would like to highlight the following differences: 

    * In this manuscript, we focus on using curvature as a method for measuring memorization of  samples, and focus on the high curvature samples. In contrast, Garg and Roy 2023 [R2] focus on low curvature samples, and their application to data efficiency for coreset creation.  

    * We note that their method of calculating curvature does not give reliable results: 
        *   They used the adversarial direction to estimate curvature. We removed the adversarial direction assumption and returned to traditional Hutchinson’s trace estimator form, and used random Rademacher vectors instead for more reliable results. The cosine similarity with FZ scores on CIFAR100 for Garg and Roy [R2] is 0.17, while our method achieves 0.82. 

        * To further improve estimation, we average our results over 10 different random Rademacher variables while [R2] use a single direction. We get a good match for n=5,10,20 with FZ scores as seen in the hyperparameter tuning results from Table 3 in Appendix B. 

        * Their method only calculates curvature at the end of training, and the following table shows that that gives unreliable scores. For instance, the match with FZ scores for curvature calculated at the end of training, as shown in the table below, is only 0.18.  We average curvature during training to get reliable results, in order to allow for different decision boundaries that have been learned at different epochs as training progresses. This is the result of studying curvature dynamics during training in Section 4.4 and is quite evident if we form a GIF of the decision boundary in the input space. We have added that to the supplementary material, also found [here](https://imgur.com/a/IGDqwo0). The early epochs in the GIF show different hypotheses tried out by the network during the training process, and averaging over them allows us to not be over-reliant on any one hypothesis.

        * We show the results for using Garg and Roy’s method, our method limited to calculating curvature at end of training, and our method as outlined in the paper and show that we get considerably better matches (better cosine similarity CS) with FZ scores for both the most memorized samples, and for the entire dataset, both with (wd1) and without (wd0) weight decay.        
            &nbsp;        


            |       Method            |      Top 5K FZ CS (wd0)      |   Top 5K FZ CS (wd1)        |      CS with FZ for all data (wd0) |     CS with FZ for all data (wd1)      |
            |-------------------------|------------------------------|-----------------------------|------------------------------------|----------------------------------------|
            |      Garg and Roy [R2]  |     0.07                     |     0.24                    |     0.10                           |     0.17                               |
            |      Ours n=10 EOT      |     0.06                     |     0.28                    |     0.12                           |     0.18                               |
            |      Ours               |     0.82                     |     0.90                    |     0.73                           |     0.82                               |
    
        * We provide insights and strong empirical evidence for the use of high curvature samples in studying memorization of deep neural nets, and utilize it to identify a new failure mode that all other methods, including R2, fails to capture, as shown in Figure 19 in the Appendix.       
    &nbsp;       

    We thank the reviewer for helping us consolidate the differences, and have summarized this discussion in Appendix D4.

Reviewer comment (Writing):
There is significant scope for improvement in the quality of the presentation. For instance:  

* In Section 3.1, both $d$ and $D$ are used to represent the input dimension.  
* In the line after Eq. (2) $v_i$ represents the $i$-th coordinate of the random Rademacher vector $v$, while in Eq (3), $v_i$ represents $i$ random *vector*. Also, the line after Eq. (3) refers to $v$ as a Rademacher *variable* instead of *vector* (a similar issue in Section 3.2).  
* In Eq. (4), please consider using $\approx$ instead of $=$.  
* In Eq. (6), please make the dependence of the square norm on indices $i$ (via $v_i$) and $t$ (via $W_t$) explicit.  
* In Section 3.2, O(n) → O(nT) forward and backward passes?  

Besides the issues mentioned above, multiple sentences in the paper require paraphrasing as their meaning is not entirely clear, and thorough proofreading is required to eliminate various typos.

Author response:
Thank you we have updated the paper.

Reviewer comment (Experiments):
How does a memorizing network behave on duplicate examples with distinct labels? Which label is preferred by the model? Does the model encounter a decreasing loss curvature trend on such examples as training progresses?

Author response:
We believe that this is hard to predict, and highly dataset and sample dependent.  For instance, let’s look at the case where the same sample is labeled girl and baby in CIFAR100. We think different models might choose different labels, but one major factor that would impact this choice would be the number of samples in each of the conflicting classes, with the majority class having a higher chance of being predicted. Another factor would be if there is another similar image with an embedding close to our conflicted sample, as that would influence the model to preferentially predict its class for the conflicting sample as well. Further, for the case in ImageNet where there is an image labeled both wine and corkscrew, since both are present in the image, we believe that the area occupied by each class might also play a role. Overall, we believe that this is a very interesting question, but would end up being largely dataset and sample dependent.

Reviewer comment (Theory):
In Section 4.2, the authors state ``... These scores are likely to be independent of spurious correlations to curvature that other methods such as confidence of prediction might have, and hence serve as a good baseline.`` Could the authors elaborate on this statement on why Feldman & Zhang’s score is likely to be independent of spurious correlations (while other methods can potentially exhibit such correlations)?

Author response:
We meant that for baseline comparison, we wanted to use a method that is unrelated to curvature in concept or calculation. In concept, the curvature likely has some correlation to distance from boundary, which other methods like the inconfidence value is also correlated to. For having a truly unrelated measure, we use Feldman and Zhang since their description of memorization is simple and depends on the prediction on a sample in presence and absence of it during training. It does not depend on training statistics and hence cannot be correlated to curvature or other related training metrics, and therefore serves as a largely independent baseline.

Reviewer comment (Evaluation):
In Section 4.2, the authors claim that computing Feldman & Zhang (FZ) score is ``~3 x more computationally expensive``. Why is it **only** 3x more expensive when computing FZ scores requires training a large number of models?

Author response:
Thank you for identifying a major typo! We meant 3 orders of magnitude. We have fixed this in the paper. To clarify, our method complexity can be thought of as equivalent to training 10 standard models since we use n=10, while FZ method trains ~20000 models.

Reviewer comment (Writing):
In Section 4.3, the authors mention ``...We recommend that curvature analysis should be used in conjunction with other checks, for a holistic view of dataset integrity.`` Could the authors clarify which other checks they refer to?

Author response:
We meant that there are many unknown failure modes that unvetted datasets can suffer from. Before we did the curvature analysis, we were not aware of the failure mode of duplicate samples with differing labels. There can be other failure modes that we do not anticipate that our method might fail to capture, and hence we recommend performing holistic checks. To begin with, we recommend performing the checks we use to compare our method’s performance against in Table 2, Confident Learning [NorthCutt1] and Inconfidence score [Carlini1]. Other scores can be found in [Pleiss1], [Katharopoulos1], [Johnson1] as mentioned in the prior work, and maybe more or less applicable depending on the task and data style. Additionally, we added two new baselines to Table 2 based on reviewer feedback, that of Learning Time and SSFT [Maini1] and show that our method combined with SSFT gives better results than both methods independently. As demonstrated, these combinations can help us find failure modes that individual methods might be insensitive to, and that might cause impactful problems in the decisions made by these models when they are deployed.

Reviewer comment (Novelty):
Novelty: The proposed method has a high degree of overlap with Garg and Roy 2023 [R2], where the similar per-instance score is used for determining the curvature. Further, the score in their paper is able to find long-tailed and rare samples, which is the application demonstrated in this paper. It would be great if authors can please clarify the novelty of this work in comparison with Garg and Roy 2023 [R2].

Author response:
1. Novelty: The proposed method has a high degree of overlap with Garg and Roy 2023 [R2], where the similar per instance score is used for determining the curvature. Further, the score in their paper is able to and long-tailed and rare samples, which is the application demonstrated in this paper. It would be great if authors can please clarify the novelty of this work in comparison with Garg and Roy 2023 [R2]. 

    Authors: Thank you for your question. Garg and Roy [R2] indeed was a source of inspiration for this work. We found their analysis of curvature for creating coresets interesting, and were inspired to see if we could use it as a measure of memorization. We found that their method did not work out of the box, and we had to make significant changes to establish a good metric, in essence reverting back to Hutchinson’s trace estimator form for calculating curvature [Hutchinson1]. Their scores also failed to capture the failure mode of duplicated samples with differing labels. We would like to highlight the following differences: 

    * In this manuscript, we focus on using curvature as a method for measuring memorization of  samples, and focus on the high curvature samples. In contrast, Garg and Roy 2023 [R2] focus on low curvature samples, and their application to data efficiency for coreset creation.  

    * We note that their method of calculating curvature does not give reliable results: 
        *   They used the adversarial direction to estimate curvature. We removed the adversarial direction assumption and returned to traditional Hutchinson’s trace estimator form, and used random Rademacher vectors instead for more reliable results. The cosine similarity with FZ scores on CIFAR100 for Garg and Roy [R2] is 0.17, while our method achieves 0.82. 

        * To further improve estimation, we average our results over 10 different random Rademacher variables while [R2] use a single direction. We get a good match for n=5,10,20 with FZ scores as seen in the hyperparameter tuning results from Table 3 in Appendix B. 

        * Their method only calculates curvature at the end of training, and the following table shows that that gives unreliable scores. For instance, the match with FZ scores for curvature calculated at the end of training, as shown in the table below, is only 0.18.  We average curvature during training to get reliable results, in order to allow for different decision boundaries that have been learned at different epochs as training progresses. This is the result of studying curvature dynamics during training in Section 4.4 and is quite evident if we form a GIF of the decision boundary in the input space. We have added that to the supplementary material, also found [here](https://imgur.com/a/IGDqwo0). We can see from the first few epochs in the GIF, that the network is trying out different hypotheses in the form of different decision boundaries, on the way to the final one. Averaging over all these hypotheses allows us to be robust to any particular one and shows much more reliable results. 

        * We show the results for using Garg and Roy’s method, our method limited to calculating curvature at end of training, and our method as outlined in the paper and show that we get considerably better matches (better cosine similarity CS) with FZ scores for both the most memorized samples, and for the entire dataset, both with (wd1) and without (wd0) weight decay. 


            |       Method            |      Top 5K FZ CS (wd0)      |   Top 5K FZ CS (wd1)        |      CS with FZ for all data (wd0) |     CS with FZ for all data (wd1)      |
            |-------------------------|------------------------------|-----------------------------|------------------------------------|----------------------------------------|
            |      Garg and Roy [R2]  |     0.07                     |     0.24                    |     0.10                           |     0.17                               |
            |      Ours n=10 EOT      |     0.06                     |     0.28                    |     0.12                           |     0.18                               |
            |      Ours               |     0.82                     |     0.90                    |     0.73                           |     0.82                               |
    
        * We provide insights and strong empirical evidence for the use of high curvature samples in studying memorization of deep neural nets, and utilize it to identify a new failure mode that all other methods, including [R2], fails to capture, as shown in Figure 19 in the Appendix. 

    We thank the reviewer for helping us consolidate the differences, and have summarized this discussion in the section in Appendix D4.

Reviewer comment (Experiments):
Missing Baselines: The authors don’t compare the proposed method to the Maini et al. 2022 [R1] for identification of the noisy labelled samples, however the setup followed is similar to that of the Maini et al. 2022 paper. I request the authors to please provide the comparison or the reason for the omission of the comparison.

Author response:
2. Missing Baselines: The authors don’t compare the proposed method to the Maini et al. 2022 [R1] for identification of the noisy labelled samples, however the setup followed is similar to that of the Maini et al. 2022 paper. I request the authors to please provide the comparison or the reason for the omission of the comparison. 

    Authors: We wish to emphasize that Maini et. al is more focused on identifying typical, mislabeled, complex and rare samples, whereas our work focuses on measuring memorization, and use the application of mislabeled sample detection only as a corroboration of our metric. However, we feel that the reviewer is justified in asking for this baseline for the mislabeled sample detection application, as it is a very strong baseline. Hence, we have added it to the paper, using the open-source code available from the author's github repo. From Table 2, we can see that Curvature outperforms SSFT for both MNIST and CIFAR-10 for all percentages of mislabeling considered. However, SSFT performs better than Curvature for CIFAR-100, but as the authors note in their text, SSFT can be combined with other methodologies. We show that combining SSFT and Curvature results in a combination that outperforms both SSFT and Curvature independently and significantly. Additionally, we also note that SSFT results shown in the paper fail to capture the novel failure mode we caught, that of duplicate samples with difffering labels.

    &nbsp;    

    We thank the reviewer for pointing us to a strong baseline, and for helping us strengthen our paper by improving our results by combining it with SSFT. However, we would still like to reemphasize that our paper focuses on establishing curvature as a method for measuring memorization, and the application of detecting mislabeled samples is shown merely as corroboration that curvature does indeed detect memorization. 

    |  Dataset |   Method  | Corruption |         |         |        |        |        |
    |:--------:|:---------:|:----------:|---------|---------|--------|--------|--------|
    |          |           | 1%         | 2%      | 4%      | 6%     | 8%     | 10%    |
    | MNSIT    | Inconf.   | 99.4\%     | 99.0\%  | 98.4\%  | 97.7\% | 97.1\% | 96.1\% |
    |          | CL        | 99.7\%     | 99.3\%  | 99.1\%  | 98.9\% | 98.9\% | 98.9\% |
    |          | SSFT      | 99.9\%     | 99.9\%  | 99.8\%  | 99.7\% | 99.7\% | 99.5\% |
    |          | LT        | 98.7\%     | 99.5\%  | 98.4\%  | 98.5\% | 97.7\% | 97.3\% |
    |          | Curv      | 100.0\%    | 100.0\% | 99.9\%  | 99.9\% | 99.9\% | 99.9\% |
    | CIFAR10  | Inconf.   | 84.2\%     | 82.5\%  | 81.8\%  | 81.6\% | 81.5\% | 81.5\% |
    |          | CL        | 92.4\%     | 93.1\%  | 93.4\%  | 93.4\% | 91.7\% | 93.4\% |
    |          | SSFT      | 94.5\%     | 94.1\%  | 93.2\%  | 92.5\% | 91.6\% | 90.0\% |
    |          | LT        | 87.3\%     | 82.5\%  | 84.0\%  | 83.4\% | 82.2\% | 83.0\% |
    |          | Curv      | 97.4\%     | 96.6\%  | 95.5\%  | 94.4\% | 94.1\% | 92.9\% |
    |          | Curv SSFT | 97.5\%     | 96.8\%  | 96.2\%  | 96.1\% | 95.2\% | 94.8\% |
    | CIFAR100 | Inconf.   | 85.0\%     | 84.0\%  | 83.5\%  | 83.6\% | 83.6\% | 83.4\% |
    |          | CL        | 83.1\%     | 84.2\%  | 85.3\%  | 86.3\% | 84.3\% | 84.6\% |
    |          | SSFT      | 94.8\%     | 93.7\%  | 93.2\%  | 91.7\% | 91.9\% | 91.0\% |
    |          | LT        | 85.2\%     | 85.2\%  | 81.8\%  | 80.8\% | 79.8\% | 78.5\% |
    |          | Curv      | 90.9\%     | 89.6\%  | 88.3\%  | 86.8\% | 85.7\% | 84.3\% |
    |          | Curv SSFT | 96.0\%     | 94.5\%  | 94.2\%  | 93.1\% | 92.9\% | 91.9\% |

Reviewer comment (Experiments):
Missing Concrete Application: The section on curvature dynamics for training provides interesting insights. However I couldn’t find any specific experiments which demonstrate its practical applicability.

Author response:
3. Missing Concrete Application: The section on curvature dynamics for training provides interesting insights. However I couldn’t find any specific experiments which demonstrate its practical applicability. 

    Authors: Thank you for pointing out the insights in Section 4.4. The major reason for writing section 4.4 was to help the readers to understand why we need to average our scores every epoch as opposed to say, calculating curvature at the end of training. In that way, it serves as the base that informs our design of metric for measuring memorization. We realize that this was not coming across from our writing, and thank the reviewer for identifying the need for a better explanation. We have cleaned up Section 4.4 considerably, and believe it reads a lot better now. We have reduced the content to highlight the following key takeaway, and believe that it is now a lot easier for the reader to follow. 

     
    &nbsp;      
    Takeaway: At the early epochs of training, the boundary changes very rapidly and significantly. In fact, every epoch, the boundary can curve around a different subset of samples. This means that measuring the curvature at 1 epoch only will be unstable. Averaging the curvature over all of training helps us iron out this effect and get reliable curvature results. This is quite evident if we form a GIF of the decision boundary in the input space, and we have added that to the supplementary material, also found [here](https://imgur.com/a/IGDqwo0). We can see from the first few epochs in the GIF, that the network is trying out different hypotheses in the form of different decision boundaries, on the way to the final one. Averaging over all these hypotheses allows us to be robust to any particular one and shows much more reliable results.

Reviewer comment (Experiments):
I am curious if the above method can be used to identify the samples which have inconsistent captions for the Vision Language Based Methods.

Author response:
4. I am curious if can the above method be used to identify the samples which have inconsistent captions for the Vision Language Based Methods? 

    Authors: We believe this is possibly a very good application of our method we have not yet tried. We focused on vision datasets in this paper, and hope that this would encourage others to try our method in different domains.

Reviewer comment (Theory):
Further, it’s claimed in the paper that FZ scores can’t be used to find samples that are duplicate images with different samples. Can you please provide a concrete reason for this?

Author response:
5. Further, it’s claimed in the paper that FZ scores can’t be used to find samples that are duplicate images with different samples. Can you please provide a concrete reason for this? 

    Authors: We wish to clarify that we did not mean that FZ scores cannot calculate this, just that the FZ scores that have been released do fail to catch duplicate samples with different labels. We have edited the paper to make this clear. We cannot recreate FZ scores to try and improve their methodology to catch this failure mode since it requires training 20,000 models. However, we can think of some reasons why they may have failed to capture it. FZ scores are calculated by looking at the change in expected probability of correct prediction when the sample is removed from the training set. To make this computationally easier, the models perform subsampling of the dataset, and it is possible that the considered subsets missed the duplicate samples. It is also possible that each of the 20,000 models was not trained to full convergence for efficiency reasons.

Reviewer comment (Theory):
Is it possible to provide any theoretical results regarding the correctness of the curvature scores for finding the noisy labeled samples etc?

Author response:
6. Is it possible to provide any theoretical results regarding the correctness of the curvature scores for finding the noisy labeled samples etc? 

    Authors: In this manuscript, we use qualitative and quantitative results to show that curvature is a good measure of memorization. We are working on more theoretical findings as part of future work.

Reviewer comment (Experiments):
Toy example in section 4.4: The number of training points is far less than in the test set, which is not a similar case to your image datasets. The training data does not have enough points to accurately represent the underlying distribution. Additionally, adding noise to it makes it more challenging for the model to learn the decision boundary correctly.

Author response:
The imbalance in the design of this toy example is intentional. Our goal was to evaluate what the decision boundary looks like when networks memorize the data. The few, noisy training data encourages the network to memorize the training data. But, having clean abundant test data allows us to also measure generalization of the underlying true data distribution under extreme memorization and noisy samples. This setup allowed us to understand how curvature behaves with extreme memorization, with the following key takeaway: 

At the early epochs of training, the boundary changes very rapidly and significantly. In fact, every epoch, the boundary can curve around a different subset of samples. This means that measuring the curvature at 1 epoch only will be unstable. Averaging the curvature over all of training helps us iron out this effect and get reliable curvature results. This is quite evident if we form a GIF of the decision boundary in the input space, and we have added that to the supplementary material, also found [here](https://imgur.com/a/IGDqwo0). Please look at the early epochs from the GIF, we can see that the neural network is trying out different decision boundaries to try to minimize the loss, since multiple different boundaries give good results. Averaging over these different hypotheses was necessary to not rely too much on one particuaar hypothesis.

We thank the reviewer for identifying the need for a better explanation, and we have cleaned up Section 4.4 considerably, and believe it reads a lot better now. We have reduced the content to highlight this key takeaway, and believe that it is now a lot easier for the reader to follow.

Reviewer comment (Presentation):
Section 4.4: It is a lengthy section and difficult to follow the main points of it; the presentation format can be improved.

Author response:
We thank the reviewer for identifying the need for a better explanation, and we have cleaned up Section 4.4 considerably, and believe it reads a lot better now. We have reduced the content to highlight this key takeaway, and believe that it is now a lot easier for the reader to follow.

Reviewer comment (Presentation):
Figure 5: The labels on the axes of the figure are not clear, and I can't interpret the results. Having a distribution over the corrupted samples would be more informative.

Author response:
Thank you for identifying this readability issue, we have increased the font size and improved the readability of graphs in Figure 5

Reviewer comment (Presentation):
Tables: MEAN+STD missing in the tables.

Author response:
We have added a density plot visualizing the distribution of input loss curvature for clean and mislabeled samples in the appendix (See Figure 19).

Reviewer comment (Experiments):
Introduction (Regarding “weakly labeled or have noisy annotations”): Can you apply your method in unsupervised settings?

Author response:
One advantage of our method is that computing curvature of sample only requires a loss, as opposed to other methods that require logits. This renders our method scalable towards unsupervised settings, unlike SSFT or FZ scores. Thank you for highlighting an advantage of our method.

Reviewer comment (Experiments):
Table 2: Did you use weight decay for other methods as well? Can you show the result of FZ method?

Author response:
Yes, all results in Table 2 are using weight decay. For FZ scores however, we were limited to using the results the posted on their GitHub repository as it is not possible to recreate their experiments as they calculated their scores by training 20,000 models per dataset. For this reason, it is also not possible to add that as a baseline to the mislabeled sample detection in Table 2.

Reviewer comment (Evaluation):
Section 4.3: Can you elaborate on why ROC curves are more reliable?

Author response:
Metrics such as accuracy are very brittle to imbalance in datasets. For instance, if we had a dataset with very few samples of the positive class, such as in our case with mislabeled samples, predicting all negative would give us a hundred percent accuracy, despite nothing having been learnt. This can be avoided by using AUROC curves. We preferred AUROC curves over PR curves as PR curves over represent the smaller class, in our case the mislabeled class. There can be memorized samples in the dataset that were not synthetically mislabeled as well, and we wanted to give a more holistic comparison. However, we realize that this may be more application and user dependent than we estimated, and upon looking closer, we think that giving a generic recommendation is likely not the best. We have removed this line from the text, and thank the reviewer for taking the time to check the minutiae of our paper.

Reviewer comment (Experiments):
How is the threshold for the curvature of loss selected?

Author response:
In the paper we do not specifically calculate a threshold, we report the AUROC (Area Under ROC curve), as is commonly practiced in literature. AUROC/AUC is obtained by varying the threshold. The Area under the ROC curve gives a more holistic view than the ROC metric at some threshold. However, given an ROC plot there are several ways in literature to select a suitable threshold. For example, one method is to choose a threshold that maximizes the true positive rate (TPR) while minimizing false positive rate (FPR) i.e. the threshold that results in the top left point of the ROC curve. Another method chooses a threshold such that FPR is below a certain value or TPR is above a certain value. There are other works that inject known label-altered data and track their metric to identify the threshold. Any of these methods could be used if the user wants a threshold for their use case. However, it is common practice to just report AUROC numbers to show good separability between two classes.

Reviewer comment (Evaluation):
Does averaging the curvature over epochs remove the specific epoch signals? Isn't it better to study the difference between curvature evolution during time than taking an average?

Author response:
Thank you for your question, this is exactly what we were hoping to answer with the analysis done in Section 4.4. Different examples are memorized at each epoch as the model makes different boundaries in order to learn the data since there can be many correct boundaries. For instance, the boundaries at epochs, 20, 30 and 40 are different in Figure 7. There is inherent noise in how the network reaches the minima during learning as it samples different minibatches stochastically, which results in different curvature scores for samples at each epoch. Averaging over epochs allows us to average over all the decision boundaries the model looked at during its training and make aggregate decisions, which is more reliable. To support our claim, we can show that curvature scores calculated at the end of training are not good enough, and we revamp Section 4.4 to make this point clearer. We provide a link to a [GIF](https://imgur.com/a/IGDqwo0) to illustrate the development of input space curvature. From the early epochs in this GIF, we can note the different hypotheses tried out by the network, evidenced by changing decision boundaries.

Reviewer comment (Evaluation):
Figure 8: Which type of sample triggers the peak in curvature, and do memorized points contribute more to that?

Author response:
Figure 8 plots the average curvature of the training set across epochs. The curvature of all samples peak around the overfitting epoch, as explained in Section 4.4. However, from the same figure,  we can see that the curvature of the 5000 most memorized samples according to an independent measure (FZ scores) remains higher than the average curvature across the dataset. Hence, we can indeed conclude that the peak is contributed to more by the memorized samples than the rest of the dataset.

Reviewer comment (Evaluation):
Figure 8: Why is the curvature of test points much higher than memorized points in CIFAR-100? And why don't we see that in ImageNet?

Author response:
We believe this is due to ReNet18 being significantly overparameterized for CIFAR100 compared to ImageNet. Thus, with CIFAR100 the model is able to overfit to the training dataset a lot more than on ImageNet, thus widening the gap between train and test examples.

Reviewer comment (Writing):
End of page 8: What does the sensitivity of test samples to perturbation mean?

Author response:
Thanks for identifying a source of confusion. We mean the sensitivity of loss to perturbation of input. We mean here, that the margin maximization we see for training samples, which causes the network to get overconfident on train samples, and hence the loss does not change much with perturbation of input, does not happen for test samples. However, this does not mean that the curvature ranking for test samples is incorrect. As shown in Figures 16 and 17 in the Appendix, the high curvature samples on the validation sets are also long-tailed and atypical. We have re-written this section.

Reviewer comment (Novelty):
Have any studies explored using the curvature of samples for inference attacks?

Author response:
As far as we are aware, there are no such studies since we are the first to show the link between curvature and memorization. However, we are in the process of exploring the use of loss curvature for membership inference attacks as part of a new work. We hope that this manuscript will encourage more experimentation on ways to use curvature as a tool for analysis of networks.