Paper: Leveraging Hierarchical Feature Sharing for Efficient Dataset Condensation (ICLR-2024)

Reviewer comment (Novelty):
It lacks a discussion and comparison to a very relevant SOTA method, i.e., IDM [1], which also addresses the scalability of data condensation and works in a low GPU memory cost scenario. Thus, it might be incorrect to argue that the proposed method is the "first method to achieve such good performance with a low GPU memory loss" and such statements should be revised.

Author response:
**A1:** We thank Reviewer #4 for pointing out a missing related work. As suggested by the reviewer, we have already added IDM to the Related Work section and added IDM as a new baseline in Table. 2 in the revised version. We also revised related statements in the paper. We would like to note that, *after adding IDM, our proposed method still achieves better or comparable performance than other baselines.*

Reviewer comment (Novelty):
The technical novelty is a bit thin. Specifically, the second contribution is relatively weak and more similar to a trick. It is interesting to know that over-budget generation and pruning may work, but this is more similar to remedying some of the shortcomings of the main idea rather than an independent contribution.

Author response:
**A2:**  We would like to note that the major novelty in HMN’s design comes from its hierarchical architecture. HMN is the first data container that considers the hierarchical structure of the classification system, and HMN achieves a better empirical performance.  This makes us believe that *HMN provides certain novelty and contribution to the DC area.*

We agree with the reviewer that the second contribution (HMN pruning part) is more like a favorable property of HMN. However, we note that not only HMN but also other data parameterization methods, like HaBa, also have data redundancy in the condensed data containers (as discussed in Section 3.2.1). Unlike the HMN approach, it is non-trivial to prune redundant generated examples in factorization-based data containers. HMN allows better independence between different generated images, which makes HMN a different data container that allows for further improving efficacy by pruning redundant information.

Reviewer comment (Reproducibility):
What does the memory look like? How to initialize these memories, feature extractor, and decoder?

Author response:
**A2:** All memories are 3-D tensors. For CIFAR10, CIFAR100, and SVHN, memories have a shape (4, 4, Channels). For Tiny-imageNet, memories have a shape (8, 8, Channels). All values in the tensor are initialized with standard normal distribution. We included more details about the memory design of HMN in Section 3.1 and Appendix C.1 in the revised version.

Reviewer comment (Experiments):
How many generated images per class (GIPC) used in Table 2?

Author response:
**A5:** We listed GIPC for each setting in Table 5 (Appendix C). Since each instance-level memory corresponds to a generated image, the row “#Instance-level memory” indicates the GIPC for each setting. We have also further revised the writing in Section 3.1 and Appendix C.1 to reduce unnecessary confusion about GIPC of HMNs.

Reviewer comment (Evaluation):
In Table 2, the result of IDC on CIFAR-10 with 50 IPC is lower than in the paper (74.5 (IDC) vs 71.6 (this paper)). Is it a mistake? On CIFAR-100, this paper reports the accuracy of IDC with 10 IPC on CIFAR-100 is 44.8 while on other papers such as DREAM, the accuracy is 45.1. Can the author double-check? The results of IDC (at 10 and 50 IPC) in Table 2, Table 3, and Table 6 are different. Can the author explain?

Author response:
**A7:** We appreciate Reviewer #3 for pointing this out. There is a typo on the evaluation result for the IDC CIFAR10 50 IPC setting. We have corrected the number (The original number is the number of MTT.). For the IDC CIFAR100 10 IPC setting, we use the data (44.8) from a data distillation survey paper [1]. However, we noticed that this number is different from the number reported in the IDC official repo [2]. We have updated this number to 45.1 in the revised version. After those modifications, the numbers in Tables 2 and 3 are corrected and the same as the author-reported numbers. 

The numbers in Table 6 were the results that we got by rerunning the IDC evaluation.  Some differences can be caused by randomness in training. We now also update the numbers in Table 6 to author-reported numbers to guarantee consistency in the paper. We thank the reviewer for helping us correct this. 

We would like to note that **the correction of those typos does not change the conclusion of our evaluation**. HMN still achieves better results.

-------------

We thank Reviewer #3 again for the insightful reviews. We appreciate the help in pointing out unclear points and typos of evaluation data in the paper, which indeed improves the quality of our paper. We hope that our response and revisions have successfully addressed your concerns and questions. We kindly hope that you can reconsider the rating in light of these updates. We are happy to have any further discussions or answer other questions that may arise, as we greatly value your feedback in this process.


[1] Sachdeva, Noveen, and Julian McAuley. "Data distillation: A survey." arXiv preprint arXiv:2301.04272 (2023).

[2] https://github.com/snu-mllab/Efficient-Dataset-Condensation/

Reviewer comment (Experiments):
As shown in Figure 4 (right), with an instance memory size near 100, around 600 images per class are generated. How many images per class (per epoch) are used for training in the evaluation phase? 600? If this is correct, the training time with condensed images is much longer than the conventional method where the total images per class are set as the IPC. Can the author clarify this?

Author response:
**A6:** We thank the reviewer for bringing the training time for discussion. We would like first to clarify that the dashed curve in Figure 4 is the GIPC curve, not the solid line. Thus, around 250 images per class generated (dashed line) at instance memory size near 100. 

The reviewer's larger point is valid in that a larger GIPC size can impact training time. We would like to point out that this is a common feature of data parameterization methods, including IDC, HaBa, LinBa, and HMN. They will all generate more images than conventional DC methods using images as data containers. For example, LinBa generates 115 images per class with the 1 IPC setting for CIFAR10. Although data parameterization methods generate more training images, data parameterization methods also achieve better accuracy. More GIPC can be treated as a tradeoff for better accuracy of data parameterization methods.  We note that this is a feature of all high-performing data parameterization methods, not only of HMNs. We leave the challenge of reducing GPIC for data parameterization methods, while retaining accuracy, for future exploration. 

As suggested by the reviewer, we have included those clarifications and discussions in Appendix B in the revised version.

Reviewer comment (Experiments):
There are unclear parts regarding the GIPC, the memory, and fairness in evaluation comparison.

Author response:
**A5:** We listed GIPC for each setting in Table 5 (Appendix C). Since each instance-level memory corresponds to a generated image, the row “#Instance-level memory” indicates the GIPC for each setting. We have also further revised the writing in Section 3.1 and Appendix C.1 to reduce unnecessary confusion about GIPC of HMNs.

**A2:** All memories are 3-D tensors. For CIFAR10, CIFAR100, and SVHN, memories have a shape (4, 4, Channels). For Tiny-imageNet, memories have a shape (8, 8, Channels). All values in the tensor are initialized with standard normal distribution. We included more details about the memory design of HMN in Section 3.1 and Appendix C.1 in the revised version.

**A3:** We thank the reviewer for pointing this out and agree with the reviewer that the evaluation with a multistep LR scheduler can still be useful. We report below the performance of HMN with a multistep LR scheduler on the CIFAR10 dataset as follows.

|      IPC       | 1  | 10   | 50  |
|:----------------:|:----:|:-----:|-------|
|    Multi-step    | 65.7 | 73.5  | 76.8  |
| Cosine Annealing | 65.7 | 73.7  | 76.9  |

We find the difference due to the LR scheduler choice to be marginal, and the results with the multistep LR scheduler do not change the findings and conclusions of our evaluation. Our primary reason for choosing the cosine annealing LR scheduler in our evaluation is that it has fewer hyperparameters compared to the multistep LR scheduler (like milestones to reduce learning rate, and learning rate decay factor). We added more details to Appendix C.2 and D.4 regarding our hyperparameter settings and the rationale for any differences.

Reviewer comment (Reproducibility):
Some reported results of IDC in this paper are different from those in the original paper without explanation.

Author response:
**A7:** We appreciate Reviewer #3 for pointing this out. There is a typo on the evaluation result for the IDC CIFAR10 50 IPC setting. We have corrected the number (The original number is the number of MTT.). For the IDC CIFAR100 10 IPC setting, we use the data (44.8) from a data distillation survey paper [1]. However, we noticed that this number is different from the number reported in the IDC official repo [2]. We have updated this number to 45.1 in the revised version. After those modifications, the numbers in Tables 2 and 3 are corrected and the same as the author-reported numbers. 

The numbers in Table 6 were the results that we got by rerunning the IDC evaluation.  Some differences can be caused by randomness in training. We now also update the numbers in Table 6 to author-reported numbers to guarantee consistency in the paper. We thank the reviewer for helping us correct this. 

We would like to note that **the correction of those typos does not change the conclusion of our evaluation**. HMN still achieves better results.

-------------

We thank Reviewer #3 again for the insightful reviews. We appreciate the help in pointing out unclear points and typos of evaluation data in the paper, which indeed improves the quality of our paper. We hope that our response and revisions have successfully addressed your concerns and questions. We kindly hope that you can reconsider the rating in light of these updates. We are happy to have any further discussions or answer other questions that may arise, as we greatly value your feedback in this process.


[1] Sachdeva, Noveen, and Julian McAuley. "Data distillation: A survey." arXiv preprint arXiv:2301.04272 (2023).

[2] https://github.com/snu-mllab/Efficient-Dataset-Condensation/

Reviewer comment (Writing):
As a researcher not working on this field, I had a hard time understanding what "IPC" means in experimental setup. Through searching other papers I get it is "Images allocated Per Class". It would be good if the paper can be more self-contained, to introduce this setting in the paper, and provide background knowledge on why this is used as the main setting. A more intuitive setting in my mind (without knowing the literature) is control the number of parameters of the learned data generators. Is this the same as the setup used in the paper?

Author response:
**Q2:** As a researcher not working on this field, I had a hard time understanding what "IPC" means in experimental setup. Through searching other papers I get it is "Images allocated Per Class". It would be good if the paper can be more self-contained, to introduce this setting in the paper, and provide background knowledge on why this is used as the main setting. A more intuitive setting in my mind (without knowing the literature) is control the number of parameters of the learned data generators. Is this the same as the setup used in the paper?

**A2:** We are happy to provide additional clarification regarding the unclear representation in the paper.

Reviewer #2 is right that IPC stands for the number of parameters used in the data containers. The abbreviation IPC stands for “Image Per Class”. This metric is used to measure how many tensor storage budget is equivalent to in terms of the number of images (same as SOTA works [1,2]). For example, 1 IPC for CIFAR10 stands for the storage budget:  pixels of an image * IPC * class = 3 * 32 * 32 * 1 * 10 = 30720 tensors. Similarly, 10 IPC for CIFAR100 stands for the storage budget:  pixels of an image * IPC * class = 3 * 32 * 32 * 10 * 100 = 30720000 tensors. Same to the previous works, we treat those tensors as float32 tensors. We added the definition of 'IPC' the first time it was mentioned (Section 1) in the revised version.

There is a historical reason for using IPC as a metric. Before data parameterization is proposed, DC methods usually use images as data containers to store the training information, which causes IPC to naturally become a storage budget metric. In this paper, we follow the setting of the previous DC works and use IPC to measure storage budgets for a more convenient comparison.

-------------

We thank Reviewer #2 for the insightful reviews and for highlighting areas in our paper that required further clarification and discussion.  Those suggestions indeed strengthen the quality of our work. We hope that our response and revisions have successfully addressed your concerns and questions. We kindly hope that you can reconsider the rating in light of these updates. We are happy to have any further discussions or answer other questions that may arise, as we greatly value your feedback in this process.

--------

[1] Liu, Songhua, et al. "Dataset distillation via factorization." Advances in Neural Information Processing Systems 35 (2022): 1100-1113.

[2] Deng, Zhiwei, and Olga Russakovsky. "Remember the past: Distilling datasets into addressable memories for neural networks." Advances in Neural Information Processing Systems 35 (2022): 34391-34404.

Reviewer comment (Writing):
It is difficult to follow some of the ideas presented in the paper. For example, the paper didn't mention the abbreviation for IPC. Also, the paper didn't talk about whether the decoder parameters is part of the budget or not.

Author response:
**A1:** We are happy to clarify the unclear presentation in the paper. The abbreviation IPC stands for “Image Per Class”. This metric is used to measure how many tensors the storage budget is equivalent to in terms of the number of images(same as SOTA data parameterization work [1,2]).  For example, 1 IPC for CIFAR10 stands for the storage budget:  Pixels of an image * IPC * class = 3 * 32 * 32 * 1 * 10 = 30720 tensors. Similarly, 10 IPC for CIFAR100 stands for the storage budget:  Pixels of an image * IPC * class = 3 * 32 * 32 * 10 * 100 = 30720000 tensors. Same to the previous works, we treat those tensors as float32 tensors. We added the definition of 'IPC' the first time it was mentioned (Section 1) in the revised version.

> Also, the paper didn't talk about whether the decoder parameters is part of the budget or not.

Both the decoder networks and memory tensors count towards the storage budget calculation. We also mentioned this in Section 3.1 and at the end of Section 4.1. We further highlighted this point in the revised version to eliminate potential confusion for readers.

Reviewer comment (Experiments):
Table.1 What is the accuracy drop for randomly pruning 10% ?

Author response:
**A3:** We present the accuracy of random pruning on the CIFAR10 10 IPC HaBa condensed dataset as an addition to Table 1. We can see that, compared to AUM pruning, random pruning consistently has a larger accuracy drop, but 10% AUM pruning does not cause an accuracy drop, which indicates the existence of redundancy in training data generated by HaBa.

| Pruning Rate |   0  |  10% |  20% |  30% |  40% |
|:------:|:----:|:----:|:----:|:----:|:----:|
|   AUM  | 69.5 | 69.5 | 68.9 | 67.6 | 65.6 |
| Random | 69.5 | 68.7 | 67.5 | 65.9 | 63.8 |

Reviewer comment (Experiments):
Algorithm 1. are you computing the accuracy in an online fashion, or you train the model for each subset selection? How this would scale for larger datasets?

Author response:
**A4:**  In algorithm 1, we retrain the model for each subset selection for different hard pruning rate.  

However, Algorithm 1 is not a bottleneck with respect to running time. The time complexity of Algorithm 1 is proportional to the size of the condensed dataset (which Algorithm 1 prunes) rather than the original dataset. Since training on the condensed dataset is much faster than training on the original dataset, compared to the time required for data condensation, Algorithm 1 requires a relatively short computation time in practice. For example, data condensation with HMNs for CIFAR10 1 IPC needs about 15 hours on a 2080TI GPU, but the entire Algorithm 1 to prune the over-budget HMN only costs an additional 20 minutes.

Moreover, we would note that Algorithm 1 runs only one time: we first condense an over-budget HMN; then, we use Algorithm 1 to identify the best hard pruning rate and prune the HMN to fit the budget. In other words, when we use HMNs for future training, we do not need to search for the best hard pruning rate again.

Thank you for the question. We have revised Section 3.2.2 to further discuss the computational cost of Algorithm 1.