Paper: Multisize Dataset Condensation (ICLR-2024)

Reviewer comment (Theory):
It's not clear why the freezing is used in MLS selection. If adaptive is good, why not just use adaptive method to choose the subset?

Author response:
Adaptive selection and Freezing serve for different purposes.
- **adaptive selection**: it is not practical to incorporate the information of all IPC at the same time due to incurring more computations or slowing learning process. We use adaptive selection to find **one** IPC that is the most learnable. By adding information of **one** IPC at a time, the mentioned problem is addressed.
- **freezing**: freezing aims to prevent the ''already learned" images from being overwritten by information of large IPC. By extracting a smaller subset, ideally speaking, the subset should not contain information of larger IPCs, otherwise the performance will be affected.


| Calculate | Compare | Freeze | 1     | 2     | 3     | 4     | 5     | 6     | 7     | 8     | 9     | 10    | Avg.  |
|-----------|---------|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| -         | -       | -      | 27.49 | 38.50 | 45.29 | 50.85 | 53.60 | 57.98 | 60.99 | 63.60 | 65.71 | 67.50 | 53.15 |
| ✓         | -       | -      | 49.35 | 48.27 | 50.00 | 52.30 | 54.20 | 58.29 | 60.90 | **63.63** | **65.90** | **67.63** | 57.08 |
| ✓         | ✓       | -      | 40.12 | **54.91** | 56.02 | 56.12 | 56.18 | 59.74 | 61.68 | 63.41 | 65.56 | 67.01 | 58.08 |
| ✓         | ✓       | ✓      | **49.55** | 53.75 | **56.39** | **59.33** | **58.13** | **60.62** | **62.06** | 63.59 | 65.25 | 66.79 | **59.55** |

This is the Tab. 2 from our paper, it shows the difference of condensation with (row 4) and without (row 3) freezing.
Row 3 delivers a poorer performance, especially IPC=1 when compared to Row 4. 
The issue is that performance of IPC1 is affected by large IPC since it is **not frozen** (keep receiving pixel updates according to larger IPC).

Reviewer comment (Experiments):
The impact of the calculation interval (∆t) on the performance of the MDC method needs to be further analyzed to determine the optimal interval size.

Author response:
Optimizing the calculate interval $\Delta t$ will further improve the performance, and we admit finding the optimal one will not be easy, especially this hyper-parameter can vary from dataset to dataset.
We will leave this problem for future exploration.
Nevertheless, Tab. 7 shows that the choice of $\Delta t$ fall a wide range, and each of them brings a considerable improvement.

Reviewer comment (Evaluation):
Can you provide the computational resource consumption and algorithmic complexity compared to Baseline-A, B, C, and other SOTA methods? It can help authors better understand the effects of algorithms in devices with limited computational resources.

Author response:
Tab. 2 (b) compares the resources for different baselines. (A simplified version is shown below.)

|      | Condense | Storage         |
| ---- | -------- | --------------- |
| A    | N        | 1 + 2 + ... + N |
| B    | N        | N               |
| C    | 1        | N               |
| Ours | 1        | N               |

Take CIFAR-10 as an example, condensing using IDC takes roughly 11 hours on average for IPC1-10. Note the time used to condense to different IPCs is different, but will not be extremely different, e.g., IPC1 ~ 10 hrs, IPC10 ~ 12 hrs.
Here are the resources needed for each baseline:
- Baseline-A: 110 hours + 550 images (storage)
- Baseline-B: ~100 hours + 100 images
- Baseline-C: 11.5 hours + 100 images
- Ours (+0.0%): 0.6 hours + 100 images
- Ours (+6.4%): 15.1 hours + 100 images

In conclusion, our method incurs 30% more training cost to attain 6.4% increase of average accuracy. To attain the same accuracy as Baseline-C, **0.6** hrs is enough. The visualization is presented in Fig. 4 in the paper. 

**Comparing to other SOTA.** We did not conduct experiments on other SOTA except for IDC-based method, but the expected additional cost is 30% - 50% (capped at 100%). The reason is that the selected IPCs are usually not large, incurring small additional cost (as shown in Fig. 5).

Reviewer comment (Writing):
The section on Visualization of MLS is currently difficult to understand. It would be helpful to provide more detailed and accessible explanations to ensure a clear understanding for readers.

Author response:
Thank you for the feedback, we will try to adjust the figure for a clearer presentation.

Reviewer comment (Experiments):
The synthetic samples within the subset seem to be fixed, which may not reflect “Multisize Dataset Condensation” correctly.

Author response:
The keyword (**multisize**) in our title means the condensed dataset can be treated as a multi-sized dataset.
That is, the dataset of $IPC=N$ can be used as dataset of $IPC={1, 2, ..., N}$ without requiring additional condensation process.
This is currently unattainable using traditional approach, since traditional one condenses a synthetic dataset to a **specific IPC**. For example, dataset with $IPC=N$ will perform poorly on smaller $IPC$ ones, and we take the rights to name the phenomenon "Subset Degradation Problem". Our method addresses the problem, turning the dataset available for multiple size of IPCs.

Reviewer comment (Reproducibility):
In basic condensation training (Sec 4.1), for each initialization the network is trained for 100 epochs. Is it the inner loop E?

Author response:
Yes. The 100 epoch is exactly the inner loop E depicted in Fig. 3.