Paper: GRADSIMCORE: GRADIENT SIMILARITY BASED REPRESENTATIVE INSTANCES AS CORESET (ICLR-2024)

Reviewer comment (Experiments):
the comparisons are very limited, given that many more methods are listed in related work

Author response:
DeepCore library contains the 12 most popular and SOTA methods for coreset selection. They have conducted extensive experiments and reported the best accuracy obtained for various percentage levels of coreset selection for the CIFAR10 and ImageNet-1K datasets. We have utilized their results and proven that our method is able to beat the performance of SOTA methods in the majority of coreset selection percentage levels. We thank the reviewer for pointing this out, and we will update the draft to mention the specific algorithms for each percentage of coreset selection for different datasets.

Q1. We thank the reviewer for pointing this out; we will update the draft with the specific algorithms that perform best in all the settings.

Reviewer comment (Experiments):
which DeepCore method was GradSimCore compared to?

Author response:
DeepCore library contains the 12 most popular and SOTA methods for coreset selection. They have conducted extensive experiments and reported the best accuracy obtained for various percentage levels of coreset selection for the CIFAR10 and ImageNet-1K datasets. We have utilized their results and proven that our method is able to beat the performance of SOTA methods in the majority of coreset selection percentage levels. We thank the reviewer for pointing this out, and we will update the draft to mention the specific algorithms for each percentage of coreset selection for different datasets.

Reviewer comment (Experiments):
is this approach applicable to datasets that are gradually collected? I.e., by rejecting newly redundant collected samples that produce a low score? It seems from this version of the paper, that the sample-wise importance scores are calculated on the entire dataset first. Then afterwards, the subset of this dataset is identified. This seems to contradict the motivation of the paper of requiring smaller datasets that are more representative overall (Table 1)

Author response:
Q2. No. In the current setting, this method is used for large static datasets and to select a representative smaller dataset. This is a valuable tool in settings such as continual learning where we have a large dataset but are limited by the time available for finetuning the model (to resume catastrophic forgetting).

Reviewer comment (Theory):
There are several assumptions made about what an averaged gradient would imply or what it means for a sample to produce a gradient that is also closer to other samples, that need a more rigorous mathematical underpinning. It is not often clear or true that samples that produce noise-free gradients are the most representative. In fact, often the opposite is true and has been utilized in a lot of data sampling techniques from hard-negative mining to label softening, just to produce noisy gradients. This is also a reason why regularization techniques such as dropouts work. The authors’ assumption may only be true in cases where there are some stringent constraints on the loss manifold.

Author response:
Our work shows empirical evidence for our proposed method that **noise-free garidents can represent a majority of the samples in the training datasets**, thereby formulating a simple procedure for coreset selection. We agree with the reviewer's remarks that there is evidence for the opposite that noisy gradients being explored towards coreset selection (e.g., forgetting, hard sample mining, etc.). Our work, on the other hand, is built on the intuition that the noise-free gradients can also be effective. We have shown that our method achieves higher performance than current SOTA methods across various CNN models and datasets, especially at lower coreset sizes. To the best of our knowledge, there is no such attempt made. Please note that no reviewer has pointed existing works that explore this. We appeal to the reviewer that coreset selection is an open problem, and the proposed methodology provides a different approach to using gradients for coreset selection. Moreover, we reckon that our method is a strong baseline to compare the effectiveness of coreset selection works to compare the effectiveness.

Reviewer comment (Experiments):
The algorithm itself is tremendously computationally expensive. Quick back-of-the-envelope calculations indicate that for the complete Imagenet dataset, most modern server-class instances will not be able to even hold the gradients in memory. Computationally, it will be more expensive to identify the coreset than to run training on the entire dataset.

Author response:
The proposed method works on the similarity score between the gradients of the loss function w.r.t. the weights of the last fully connected layer of the network. Storing the gradients w.r.t. all the model parameters is not required. Moreover, please note that our algorithm doesn’t consider all the 1.2M images in the ImageNet dataset simultaneously. Rather, it compares class-wise (approximately 1200). Hence, it is not as heavy as perceived by the reviewer.

The proposed method utilizes multiprocessing and nearest-neighbor algorithms to perform this efficiently. 
 
Below, we tabulate the time taken for one epoch processing of ImageNet training: gradient calculation and similarity score computation for that epoch on an RTX-A5000 24 GB GPU. The similarity computation (based on gradient comparison) is performed only once every epoch. Based on these similarity scores, the coreset is selected at the end (of 5 epochs) by ranking the individual images for each.

----------------------------------------------------------------------------------
Process                                         |  Time taken
----------------------------------------------------------------------------------
Single epoch full dataset training  |  4.46 hours
----------------------------------------------------------------------------------
Gradient Calculation                     |  3.19 hours
----------------------------------------------------------------------------------
Ranking                                        | 225 seconds
----------------------------------------------------------------------------------

Reviewer comment (Experiments):
The experimentation is done on small datasets, in smaller models. From the results, it looks like their work only works at low-accuracy levels (or extremely low data sizes). In modern-day ML, unless there is parity in accuracy (generalization performance), it is not a practically usable model, no matter the constraints. Putting aside the pragmatism, the results show that the authors' approach does not apply at even a relatively modest size of coreset.

Author response:
We have provided the results for three different datasets with different class numbers. We have provided results for CIFAR10 ( 10 classes), CIFAR100 (100 classes), and ImageNet-1K (1000 classes). Implementation is done for ResNet-18 (~11 million parameters), VGG-16(~138 million parameters), and Inception-V3 (~24 million parameters) models, which respectfully can not be termed as smaller models). 

Coreset selection aims to select a smaller percentage of representative samples to train the deep learning models. Yes, it takes a toll on the accuracy of the corset-trained model. Still, it should be noted that it is an open research area where our method achieves higher accuracy than SOTA models, particularly at smaller sizes. 

-  DeepCore library contains the 12 most popular and SOTA methods for coreset selection. They have conducted extensive experiments and reported the best accuracy obtained for various percentage levels of coreset selection for the CIFAR10 and ImageNet-1K datasets. We have utilized their results and proven that our method is able to beat the performance of SOTA methods in the majority of coreset selection percentage levels.

We thank the reviewer for pointing this out, and we will update the draft to mention the specific algorithms for each percentage of coreset selection for different datasets.

Reviewer comment (Reproducibility):
The authors compare against deepcore library's best algorithm, but fail to mention which algorithm it is and what were the settings used. This does not convey even the basic details. Will it be possible to make a more direct comparison?

Author response:
-  DeepCore library contains the 12 most popular and SOTA methods for coreset selection. They have conducted extensive experiments and reported the best accuracy obtained for various percentage levels of coreset selection for the CIFAR10 and ImageNet-1K datasets. We have utilized their results and proven that our method is able to beat the performance of SOTA methods in the majority of coreset selection percentage levels.

We thank the reviewer for pointing this out, and we will update the draft to mention the specific algorithms for each percentage of coreset selection for different datasets.