Paper: Boosting Dataset Distillation with the Assistance of Crucial Samples (ICLR-2024)

Reviewer comment (Evaluation):
**Inconsistent and Marginal Gains in Test Accuracy**: The test accuracy of the proposed method doesn't consistently outperform existing techniques. When it does show an improvement, the margin is sometimes minimal.

Author response:
**A1**: In Table 1 and 3, we have conducted experiments on $8$ datasets under $20$ settings. *Compared to our baseline*, our proposed method improves the baseline with an average relative improvement of $6.6$\% and an absolute improvement of $2.0$\% on $16$ settings, which is not marginal in dataset distillation field (can be inferred by comparisons with SOTA methods RFAD, MTT and FRePo in Table 1). *When compared to SOTA approaches*, our method has achieved the SOTA performance under $14$ settings. Besides, the performances can be further boosted with more advanced studies, as shown in Table 4 and 8. For example, when combined with RFAD, our performance on CIFAR10, IPC=1 can achieve performance up to $54.8$\%, which is a new SOTA in Table 1.

What's more, we find the proposed module only fails to show superiority on small datasets like MNIST and F-MNIST but still shows to be comparable to other methods. In contrast, it holds a superior performance on larger datasets such as TinyImageNet and ImageNet as shown in Table 1 and 3. Since the dataset distillation aims to solve the difficulty in processing large datasets, we believe performing better on large datasets is more important nowadays.

Reviewer comment (Novelty):
**Incomplete Review of Related Work**: The paper falls short in its coverage of existing literature. The need for a more comprehensive review is also detailed in the "Questions" section below.

Author response:
**A2**: Thank you for the suggestion. In this paper, we mainly discussed studies like MTT(CVPR2022), FRePo(NeuraIPS2022), RCIG(ICML2023). We have also included the discussion of the suggested work [1*][2*][3*][4*] and experimental comparison with the dataset distillation methods [3*] (IDC) [4*] (CAFE) in Section G of the revision. CAFE proposes to conduct feature alignment for a better dataset distillation performance while IDC is a novel method that analyzes the shortcomings of the existing gradient matching-based condensation methods and develops an effective optimization technique for improving the condensation of training data information. The comparisons are listed in the table below (IDC only provides its mean accuracy in its Table 10, here we only show the mean accuracy without standard deviation value). More results can be found in the revision. Besides, IDC also proposes to divide images into several parts to make full use of the storage budget (IDC+M). We also include this strategy (Ours+M) to make comparisons with IDC in table below.

Table a. Test accuracy (\%) comparison.
| Method | CIFAR10  |          |          |
|--------|----------|----------|----------|
|        | IPC=1    | IPC=10   | IPC=50   |
| CAFE   | 31.6     | 50.9     | 62.3     |
| IDC    | 36.7     | 58.3     | 69.5     |
| Ours   | **48.4** | **67.2** | **73.8** |
| IDC+M    | 50.6     | 67.5     | **74.5**     |
| Ours+M   | **58.6** | **71.1** | 74.2 |

Reviewer comment (Writing):
**Clarification on "Data Extension" Terminology**: The term "data extension" is unfamiliar and appears to be non-standard. Is it synonymous with commonly used terms like "data augmentation" or "data interpolation"? If not, what differentiates it, and why opt for this term?

Author response:
**A4**: Thank you for the question. Both data augmentation and interpolation are techniques designed for dataset extension and differ from each other regarding the processing method. In other words, 'data extension' is the goal while 'data augmentation' and 'interpolation' are methods. In this paper, we named the title of Section 2.3 as 'Crucial sample extension with semantic augmentation' since we propose a method to "implicitly enrich harder ones in the semantic space through continuously
interpolating between two target feature vectors" (the last paragraph of Introduction). We use semantic augmentation instead of data augmentation due to the reason that data augmentation typically refers to techniques that generate new training samples by applying various transformations or modifications to existing data, such as rotation, flipping, or adding noise. It is usually done before putting samples into models. In contrast, our proposed method conducts augmentation in semantic space. Experimental comparisons are also provided in Figure3.c, which indicates the superiority of the proposed semantic augmentation in the dataset distillation task.

Reviewer comment (Novelty):
**Major Revisions in Related Work Section Needed**: The section on related work requires substantial updates for completeness and context.

Author response:
**A2**: Thank you for the suggestion. In this paper, we mainly discussed studies like MTT(CVPR2022), FRePo(NeuraIPS2022), RCIG(ICML2023). We have also included the discussion of the suggested work [1*][2*][3*][4*] and experimental comparison with the dataset distillation methods [3*] (IDC) [4*] (CAFE) in Section G of the revision. CAFE proposes to conduct feature alignment for a better dataset distillation performance while IDC is a novel method that analyzes the shortcomings of the existing gradient matching-based condensation methods and develops an effective optimization technique for improving the condensation of training data information. The comparisons are listed in the table below (IDC only provides its mean accuracy in its Table 10, here we only show the mean accuracy without standard deviation value). More results can be found in the revision. Besides, IDC also proposes to divide images into several parts to make full use of the storage budget (IDC+M). We also include this strategy (Ours+M) to make comparisons with IDC in table below.

Table a. Test accuracy (\%) comparison.
| Method | CIFAR10  |          |          |
|--------|----------|----------|----------|
|        | IPC=1    | IPC=10   | IPC=50   |
| CAFE   | 31.6     | 50.9     | 62.3     |
| IDC    | 36.7     | 58.3     | 69.5     |
| Ours   | **48.4** | **67.2** | **73.8** |
| IDC+M    | 50.6     | 67.5     | **74.5**     |
| Ours+M   | **58.6** | **71.1** | 74.2 |

Reviewer comment (Novelty):
**Coresets**: The paper cited is neither the seminal work nor the most recent in the field of coresets. It would be beneficial to include at least these two papers [1*] and [2*], and consider citing earlier foundational works they mention, perhaps in an appendix.

Author response:
**A2**: Thank you for the suggestion. In this paper, we mainly discussed studies like MTT(CVPR2022), FRePo(NeuraIPS2022), RCIG(ICML2023). We have also included the discussion of the suggested work [1*][2*][3*][4*] and experimental comparison with the dataset distillation methods [3*] (IDC) [4*] (CAFE) in Section G of the revision. CAFE proposes to conduct feature alignment for a better dataset distillation performance while IDC is a novel method that analyzes the shortcomings of the existing gradient matching-based condensation methods and develops an effective optimization technique for improving the condensation of training data information. The comparisons are listed in the table below (IDC only provides its mean accuracy in its Table 10, here we only show the mean accuracy without standard deviation value). More results can be found in the revision. Besides, IDC also proposes to divide images into several parts to make full use of the storage budget (IDC+M). We also include this strategy (Ours+M) to make comparisons with IDC in table below.

Table a. Test accuracy (\%) comparison.
| Method | CIFAR10  |          |          |
|--------|----------|----------|----------|
|        | IPC=1    | IPC=10   | IPC=50   |
| CAFE   | 31.6     | 50.9     | 62.3     |
| IDC    | 36.7     | 58.3     | 69.5     |
| Ours   | **48.4** | **67.2** | **73.8** |
| IDC+M    | 50.6     | 67.5     | **74.5**     |
| Ours+M   | **58.6** | **71.1** | 74.2 |

Reviewer comment (Novelty):
**Dataset Distillation**: In addition to [2*], works like [3*] and [4*] are missing from both the discussion and comparison tables. Also, MTT, which is covered in the experiments, lacks mention in the related work section. Please include these papers in both the textual discussion and comparative evaluations.

Author response:
**A2**: Thank you for the suggestion. In this paper, we mainly discussed studies like MTT(CVPR2022), FRePo(NeuraIPS2022), RCIG(ICML2023). We have also included the discussion of the suggested work [1*][2*][3*][4*] and experimental comparison with the dataset distillation methods [3*] (IDC) [4*] (CAFE) in Section G of the revision. CAFE proposes to conduct feature alignment for a better dataset distillation performance while IDC is a novel method that analyzes the shortcomings of the existing gradient matching-based condensation methods and develops an effective optimization technique for improving the condensation of training data information. The comparisons are listed in the table below (IDC only provides its mean accuracy in its Table 10, here we only show the mean accuracy without standard deviation value). More results can be found in the revision. Besides, IDC also proposes to divide images into several parts to make full use of the storage budget (IDC+M). We also include this strategy (Ours+M) to make comparisons with IDC in table below.

Table a. Test accuracy (\%) comparison.
| Method | CIFAR10  |          |          |
|--------|----------|----------|----------|
|        | IPC=1    | IPC=10   | IPC=50   |
| CAFE   | 31.6     | 50.9     | 62.3     |
| IDC    | 36.7     | 58.3     | 69.5     |
| Ours   | **48.4** | **67.2** | **73.8** |
| IDC+M    | 50.6     | 67.5     | **74.5**     |
| Ours+M   | **58.6** | **71.1** | 74.2 |

Reviewer comment (Theory):
The technique of discarding easy samples is heuristic and seems counterfactual. Intuitively, easy samples should contain some common patterns that can reflect what a class of objects looks like in general. These samples should be more effective than hard samples to capture the major features of each class. In dataset distillation, major information is expected to be stored while other unusual patterns are discarded. It seems strange that discarding easy samples leads to better performance, especially when IPC is small.

Author response:
**A2**: Thank you for the question . Actually, we hold the same curiosity with you on "whether we should make a compromise to focus on the majority of data points around the center to ensure good performances on these data and what would happen if we discard these samples" (the last line in the first paragraph of Section 2.2). Motivated by this curiosity,  we conduct extensive experiments on discarding easy samples or hard samples in Section2.2 (both the Figure 1 and Figure 5). The results on three datasets and four settings all suggest that dropping easy samples helps dataset distillation in all cases. Here we provide some explanations below.
- The goal for dataset distillation is to condense the large dataset into a smaller one such that a model trained on it can  achieve a comparable test performance as one trained on original dataset (see Introduction). In other words, we need to depict a good decision boundary for classification based on the distilled data. However,  relying solely on easy samples may result in short-cut learning[a]. In contrast, hard samples are difficult-to-be-distinguished samples (Eq.4) that usually exist along the decision boundary thus can help to support a more explicit decision boundary. We think that's why hard samples can help to improve the dataset distillation performances. 
- We have also explored it from both the perspective of data manifold and information in the third paragraph of Section 2.2. In short, we observed an increased manifold overlap between the generated images and the original dataset after discarding some easy samples (Figure 1 middle two images), thereby depicting a better representation of the manifold. We also find that making precise predictions for simple samples with hard samples is easy but the reverse is not. This indicates that harder samples are more informative than easier ones.

The above discussion has been included in Appendix B of the revision. We think this is a really interesting finding and it is the 'counterfactual nature' of our discovery that makes our work unique and distinct. We hope it can bring more insights to scholars in dataset distillation.

Reviewer comment (Evaluation):
This strategy drops some samples, which destroys the original data distribution. However, according to Fig. 1, it makes the distilled data better follow the original distribution, which seems strange.

Author response:
**A3**: In fact, we found that compared to the distilled samples generated by the baseline, the samples distilled by our method tend to be distributed not only in high-density spaces (central regions) but also in low-density spaces. This helps to increase the data manifold overlap with the original dataset.

Reviewer comment (Novelty):
There seems to be a paper using a similar technique: "Prune then distill: Dataset distillation with importance sampling," ICASSP 2023.

Author response:
**A4**: 'Prune then distill' is a method that proposes to prune the dataset first and then conduct dataset distillation. Our method differs from it from two perspective: 
- Compared to the vallina dataset distillation methods, the offline pruning will introduce extra time costs that you mentioned in the first question. In contrast, our method is an online one which just needs to rank the losses and average the topk losses, introducing almost no time costs as illustrated in the Answer to the fisrt question.  
- The offline pruning may result in information loss by dropping samples before dataset distillation. Our online method ranks losses within batches during dataset distillation, it is after the samples can be represented by the learned samples (*a.k.a* 'NFR' loss is small) that these samples will be discarded. In this way, the information in easy samples can be maintained. The experimental comparisons are listed as follows. Note that since the 'prune then distill' adopts the distillation method MTT as their distillation method, here we provide MTT+Ours for a fair comparison in the table below, which indicates that our proposed method has a better performace. Detailed information can be found in Section G in the revision.

Table a: Test accuracy (\%) comparison
| Method             | CIFAR10      |              |              |
|--------------------|--------------|--------------|--------------|
|                    | IPC=1        | IPC=10       | IPC=50       |
| Prune then distill | 44.7±1.5     | 63.1±0.7     | 69.7±0.4     |
| Ours               | **57.9±0.6** | **65.4±0.6** | **72.9±0.2** |

Reviewer comment (Experiments):
The reviewer would like to see separate results of only using this strategy without ISA, such as in Tables 2, 3, 4, and 8.

Author response:
**A5**: We list the results of using dropping/discarding strategy of Table 2, 3, 4, 8 in the following:
-  Table 2 evaluates the cross-architecture generalization ability of the distilled data. This evaluation with only discarding operation can be found in Figure 3.d. To be short, removing easy samples can result in a small generalization ability degradation. However, this can be alleviated by our proposed ISA, as shown in Table 2 and Figure 3.d.  

- Table 3 provides the evaluation of a larger dataset. The ablation studies of each proposed module on three datasets (CIFAR10, CIFAR100, Tiny-ImageNet) can be found in Figure 3.a. All the results validate the effectiveness of the proposed discarding strategy and ISA. We have also conducted experiments to provide discarding ablations on ImageNet datasets below. The results are consistent with ablation results in Figure 3.a that discarding some easy samples can help in dataset distillation.  

Table b: Test accuracy (\%) of adding discarding only (corresponding to Table 3 in main text)
| Method      | ImageNette(128x128) |              | ImageWoof(128x128) |              | ImageNet(64x64) |              |
|-------------|---------------------|--------------|--------------------|--------------|-----------------|--------------|
|             | 1                   | 10           | 1                  | 10           | 1               | 2            |
| Baseline    | 48.1±0.7            | 66.5±0.8     | **29.7±0.6**       | 42.2±0.9     | 7.5±0.3         | 9.7±0.2      |
| +Discarding | **49.1±0.7**        | **66.6±0.4** | 28.0-0.4           | **43.0±0.7** | **8.6±0.2**     | **10.6±0.2** |

- Table 4 and  Table 8 provide experimental results of combining the proposed modules with other methods. We list the required results of adding discarding operation only to these methods below. Discarding can boost the performances in most scenarios. This is consistent with our findings. 

Table c: Test accuracy (\%) of adding discarding only (corresponding to Table 4 in main text)
| Method | CIFAR10  |              |              |              |          |              |
|--------|----------|--------------|--------------|--------------|----------|--------------|
|        | IPC=1    |              | IPC=10       |              | IPC=50   |              |
|        | Baseline | +Discarding  | Normal       | +Discarding  | Normal   | +Discarding  |
| RFAD   | 52.1±0.1 | **54.3±0.1** | 65.3±0.1     | **66.5±0.1** | 69.8±0.2 | **70.1±0.1** |
| FRePo  | 46.8±0.7 | **48.0±0.4** | 65.5±0.4     | **66.8±0.4** | 71.7±0.2 | **72.9±0.1** |
| RCIG   | 53.9±0.5 | **53.9±0.3** | 67.3±0.3     | **67.7±0.4** | 73.5±0.2 | **73.7±0.4** |

Table d: Test accuracy (\%) of adding discarding only (corresponding to Table 8 in main text)
| Method | CIFAR10  |              |              |              |          |              |
|--------|----------|--------------|--------------|--------------|----------|--------------|
|        | IPC=1    |              | IPC=10       |              | IPC=50   |              |
|        | Baseline | +Discarding  | Normal       | +Discarding  | Normal   | +Discarding  |
| DM     | 25.9±0.8 | **26.1±0.3** | **48.9±0.6** | 48.4±0.6     | 62.7±0.5 | **62.8±0.2** |
| MTT    | 46.3±0.8 | **52.9±0.7** | 65.2±0.5     | **65.2±0.5** | 71.6±0.2 | **72.1±0.2** |

Reviewer comment (Experiments):
The reviewer suggests the authors compare qualitative samples of the baseline, with selection, and with ISA together to better reflect the functionality of each part. Currently it seems that the qualitative results are not different from the original FRePo too much.

Author response:
**A6**: The ablation study of each component is provided in the first paragraph (**Ablation Studies on Each Proposed Module.**) of Section 3.2 (ABLATION STUDIES). It is carried out on three datasets (Figure 3.a) and all the results indicate the effectiveness of both the proposed modules. 

As for the performance gain, in Table 1 and 3, we have conducted experiments on $8$ datasets under $20$ settings. Compared to the baseline, our proposed method improves the baseline with an average relative improvement of $6.6$\% and an absolute improvement of $2.0$\% on $16$ settings, which is not marginal in the dataset distillation field (can be inferred by comparisons with SOTA methods RFAD, MTT and FRePo in Table 1).