Paper: Towards Mitigating Architecture Overfitting in Dataset Distillation (ICLR-2024)

Reviewer comment (Novelty):
The results are impressive, however, the method contributions seem relatively marginal. Though the adapted method absorbed from previous works is non-trivial, the discussion lacks enough insight but is empirical.

Author response:
Although our design is motivated by some existing techniques such as DropPath and knowledge distillation, we adapt them in the context of dataset distillation. For DropPath, we make it applicable to single-branch networks and propose improved shortcut connection and three-phase keep rate to boost the performance. For knowledge distillation, we pioneer the use of small networks as teacher models, which is different from current common practices. In the experiments, our method significantly mitigates the architecture overfitting in dataset distillation and improves the generalization of large networks on small datasets. Section 3 comprehensively demonstrates the motivation of our proposed methods. More fundamental analysis of DropPath and knowledge distillation were well analyzed in many works [1 - 3].

Reviewer comment (Experiments):
The three-phase Keep Rate looks quite complex for tuning. How is the tuning complexity and robustness if we use the proposed method for many different models?

Author response:
In the ablation study, we evaluate the performance under different hyperparameters (HPs). **Moreover, additional ablation studies on HPs of three-phase keep rate were added**. The result is reported below and in Figure 4 (b) and (c). Compared to the significant performance improvement brought by our method, if we choose different HPs in an **appropriate but wide** range, the performance fluctuation caused by different HPs is negligible. For example, when the final keep rate ranges from 0.6 to 1.0, KD weight ranges from 0.3 to 0.9 and KD temperature ranges from 1 to 10, the performance fluctuation is with 1 percentage point. In addition, although lower minimum keep rates and longer periods of decay induce better performance, they also mean a longer training phase. We set them to 0.5 and 500 to balance the efficiency and performance, respectively. Despite that, the performance fluctuation caused by these two HPs is still insignificant. Therefore, we can demonstrate that our training method is robust to different HPs. In addition, the selected HPs also perform well in different settings, including different distilled data, different IPCs and different test networks.

Reviewer comment (Presentation):
Though the bag of methods works well, the whole paper gives the readers a feeling of separation.

Author response:
The main contribution of our work is proposing a bag of methods to mitigate architecture overfitting in dataset distillation. These methods perform like regularization from different perspectives, such as **network architecture** (i.e. DropPath with three-phase keep rate), and **training scheme** (i.e. knowledge distillation, data augmentation and optimizer). Instead of directly using these methods, we adapt them to our scenario to further improve the performance. In the experiments, we demonstrate that our method not only mitigates architecture overfitting in dataset distillation but also improves the generalization of large models on small datasets. We believe our work provides practitioners with a toolbox and some intuition to solve related problems.

Reviewer comment (Experiments):
There were only discussions on the residual architecture, why? There are also other multi-branch architectures.

Author response:
For the architectures evaluated in the experiments, we follow the setting of previous works [4, 5]. The multi-branch networks adopted in the evaluation are ResNets there.

Reviewer comment (Evaluation):
"As a result, we can also expect DropPath to mitigate the architecture overfitting issue in dataset distillation." --- any more detailed analysis?

Author response:
In dataset distillation, the test accuracy of the test network trained on the distilled dataset degrades significantly when it has a different network architecture from the training network, especially when the test network is deeper than the training one. DropPath can reduce the effective depth of networks during training by stochastically removing paths, which is equivalent to training a shallow sub-network at each iteration. Through minimizing the loss, we expect the performance of each shallow sub-network to be similar to that of the training network. Similar to Dropout, we implicitly learn an ensemble of sub-networks by DropPath. During the inference phase, since each shallow sub-network can work well, the ensemble of them will be even better.

Reviewer comment (Writing):
"Architecture overfitting arises from deeper test networks" --- any citation or discussion?

Author response:
In previous works about dataset distillation [4, 5], the architecture overfitting is also observed and becomes more severe when the depth of networks increases (from 3-layer CNN to ResNet). In addition, in a more general scenario, severe overfitting also occurs when we train a large and deep model on tiny real datasets.

Reviewer comment (Evaluation):
I am not sure whether it is really the "overfitting". Why this is an overfitting? Is there any evidence to show that the performance already saturate on one backbone, but decrease on the other backbones? It could be the case that the distillation process is even "underfit" on a given architecture - and the performance drop during "transferring" (i.e., distill with one backbone and then evaluate on the others) could be just some normal "generalization error". I would suggest to use the terminology more carefully to avoid such an confusion, or provide evidence to support the terminology.

Author response:
We are sorry about the confusion. From the perspective of evaluation, we argue that the architecture overfitting is consistent with the classical definition of overfitting: a model suffers from overfitting if the gap between its performance on the training set and the test set is big. In our cases, the training accuracy on the distilled dataset is always 100\% for all network architectures, but the test accuracy on the real dataset is lower, so overfitting definitely occurs as we see generalization gaps. Moreover, the generalization gap is bigger when we train larger networks on the distilled dataset than when we use a small network with the same architecture as the training network. We call such difference *architecture overfitting*, meaning the additional overfitting arising from the architecture difference between the training network and the test network in dataset distillation. From the dataset perspective, we agree with the reviewer that the distilled dataset is "unfit" to the original large dataset given a network architecture. We believe that the distilled dataset loses some useful information from the original large dataset, leading to noticeable performance degradation when training models on the distilled dataset. Nevertheless, it is important to note that this paper primarily focuses on examining the network architecture perspective. In this context, we utilize the term *architecture overfitting* to describe the phenomenon under investigation.

Reviewer comment (Novelty):
From what I am understanding, this work actually alters the evaluation process (the process of training a model on the distilled image set). Augmenting the training process is not novel at the high level as many previous works have already done so. A lot of augmentation to architectures (like DropOut and DropPath etc.) and data (MixUp, CutMix, AutoAugment, etc.) could be applied here and I think that can serve as potential baselines.

Author response:
**Reply: We have added more experiments in the revised manuscript in Table 5 of Appendix B.1**, where we adopt DropOut, DropPath, MixUp and CutMix as baselines and evaluate their performance with the same setting in the ablation study (ResNet18, FRePO, CIFAR10, IPC=10). Note that AutoAugment only supports uint8 data, the pixel value can only be in \{0, 1, ..., 255\}. However, distilled data are float32 and their pixel values are neither bounded by [0, 1] nor [0, 255]. As a result, we do not include AutoAugment in the comparison. Similar to AutoAugment, we already adopted 2-fold data augmentation in our method, which also randomly samples 2 augmentations from a pool of augmentations. In addition, from the results shown in Table 2 (w/o DP \& KD v.s. Full), where *w/o DP \& KD* already uses better data augmentation and better optimizer, we can expect that pure data augmentation will not outperform our method. The results of the comparison with different baselines are reported in the table below. We can observe that simple DropPath and DropOut with a constant keep rate can even deteriorate the performance. In addition, MixUp and CutMix only lead to marginal performance improvement, which further demonstrates the effectiveness of our method.

Reviewer comment (Novelty):
Section 4.2 is interesting, but I think it is off-topic. The paper is trying to solve the so-called "architectural overfitting" but it will not happen when there is no actual "fitting" process. Therefore, I think Contribution 3 is do not count and the contribution bullets should be adjusted.

Author response:
**Reply:** As pointed out in the answer to Question 1, architecture overfitting accounts for the additional generalization gap when we train models of different architectures on the distilled dataset, compared with training on small datasets. In section 4.2, we demonstrate that our methods contribute more performance improvement in the case of training on distilled dataset than in the case of training on sub-sampled dataset, so it is useful to mitigate *architecture overfitting*. In addition, section 4.2 validates the effectiveness of our methods in sub-sampled dataset, which indicates the versatility of our method.

Reviewer comment (Novelty):
I do not think the cross-architecture generalization problem should be addressed this way. The authors do not modify the process of obtaining synthetic datasets during dataset distillation. Instead, they modify the training strategies **given** synthetic datasets. In dataset distillation, we should definitely focus on the former and should not make any assumptions on how users should use provided synthetic datasets for **downstream** training. What I want to see is actually a thorough algorithm for the process of dataset distillation that can improve the cross-architecture performance following the original evaluation protocols. The current form does not really enhance dataset distillation. The improvement is from the strategies of using distilled datasets.

Author response:
We agree with the reviewer that proposing novel dataset distillation methods may help mitigate architecture overfitting and is definitely a direction for exploration. However, we argue that it is also meaningful to improve the training strategies using the distilled dataset. Our argument is supported by the following points:

- The overfitting issue studied in this work is not **solely** caused by dataset distillation because a similar overfitting phenomenon is also observed when we train models on tiny real datasets. For example, the results in Section 4.2 indicate overfitting happens when we use a tiny sub-sampled training set to train deep neural networks. Our proposed methods can also improve the performance in this context.

- Following the first point, from the results in Section 4.1 and 4.2, we observe that training on distilled datasets has better performance than training on sub-sampled datasets. However, the performance gaps between the small and big networks are also larger when training on the distilled datasets. In this context, our proposed methods can induce more performance boosts, so it is more meaningful to apply our methods for distalled datasets, which is also the focus of this paper.

- In addition, our methods are orthogonal to dataset distillation since our method focuses on **how to use the distilled dataset**, and dataset distillation focuses on **how to obtain the distilled dataset**. These two categories of methods can help improve each other. For example, by adopting our method, we can more comprehensively evaluate the performance on different distilled datasets and thus compare different dataset distillation methods fairer.

Reviewer comment (Novelty):
The technical novelty is limited because the authors only provide strategies with minor designs that can empirically improve the performance. Without sufficient analysis, the principles of how the proposed methods work are unclear, which results in limited scientific value.

Author response:
Although the elements of our method are motivated by some existing techniques such as DropPath and knowledge distillation, we adapt them in the context of dataset distillation. For DropPath, we make it applicable to single-branch networks. We also propose the improved shortcut connection and the three-phase keep rate to boost the performance. For knowledge distillation, use the small network as the teacher network which is contrary to common practices. Regarding the analyses, the ablation studies in the experiments demonstrate the effectiveness and necessity of each of these components. More fundamental analysis of DropPath and knowledge distillation were well analyzed in many works [3 - 5].
    
We agree that the recipe of our solutions may look straightforward, but **1)** the method is new, and to the best of our knowledge, there is no existing work that use these technique to tackle the architecture overfitting issue of dataset distillation, a key issue of current dataset distillation methods; **2)** the method is well-motivated as demonstrated in section 3, we explained why we need DropPath to decrease the effective depth, three-phase keep rate to stabilize training and knowledge distillation to utilize small network to guide the training of large network. For practitioners, we offer a generally applicable and easy-to-tune (as demonstrated in the ablation study) solution to solve an important issue of dataset distillation and indicate the potential of applying our method to other scenarios as well, such as training on the real but tiny dataset. We believe we do not necessarily need complicated methods to solve a problem, utilizing existing techniques wisely can also generate solutions.

Reviewer comment (Experiments):
The proposed methods are somewhat complicated. For example, they assume users would apply a three-phase keep rate during training with DropPath, which introduces lots of hyper-parameters and makes the pipeline complex and less robust.

Author response:
Actually, we do not intensively tune the hyper-parameters (HPs). In the ablation study, we evaluate the performance of different HPs. **Upon the result of the reviewer, additional ablation studies on HPs of three-phase keep rate were added**. The result is reported below and in Figure 4 (b) and (c). Compared to the significant performance improvement brought by our method, if we choose HPs in an **appropriate but wide** range, the performance fluctuation caused by different HPs is negligible. For example, when the final keep rate ranges from 0.6 to 1.0, KD weight ranges from 0.3 to 0.9 and KD temperature ranges from 1 to 10, the performance fluctuation is with 1 percentage point. In addition, although lower minimum keep rates and longer periods of decay induce better performance, they also mean a longer training phase. We set them to 0.5 and 500 to balance the efficiency and performance, respectively. Despite that, the performance fluctuation caused by these two HPs is still insignificant. Therefore, we can demonstrate that our training method is robust to different HPs. In addition, our selected HPs perform well in different settings, including different distilled data, IPCs and test networks.

***Table 1: Ablation study on the keep rate in the final phase.***
|  Final Keep Rate   | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|  ----  | ----  | ----  | ----  | ----  | ----  |
| Test Accuracy  | 66.2 | 66.3 | **66.6** | 66.3 | 66.4 |
  
  
***Table 2: Ablation study on the period of decay.***
|  Period of Decay   | 100 | 200 | 300 | 400 | 500 | 600 | 700 | 800 | 900 | 1000|
|  ----  | ----  | ----  | ----  | ----  | ----  | ----  | ----  | ----  | ----  | ----  |
| Test Accuracy  | 63.5 | 65.3 | 66.5 | 66.3 | 66.6 | 66.8 | 66.9 | **67.3** | 67.2 | 67.1 |

Reviewer comment (Theory):
I’m not sure I followed the reasoning behind the scaling of the DropPath output maps. It would be nice to have a derivation in the Appendix for how this scaling matches the expectations from training to test.

Author response:
Eq. (1) on page 4 shows that $\mathtt{DropPath}(\mathbf{x}) = \frac{m}{p}\cdot \mathbf{x},\quad m=\mathtt{Bernoulli}(p)$, where $p \in [0, 1]$ denotes the keep rate, $m = \mathtt{Bernoulli}(p) \in \{0,1\}$ outputs 1 with probability $p$ and 0 with probability $1-p$. We consider the module output $\mathbf{y} = \mathtt{DropPath}(\mathbf{x})$ in the training phase, then the expectation of $\mathbf{y}$ given $\\mathbf{x}$ is $\mathbb{E}(\mathbf{y}) = p \cdot \frac{1}{p} \cdot \mathbf{x} + (1 - p) \cdot \frac{0}{p} \cdot \mathbf{x} = \mathbf{x}$. In the test phase, DropPath is disabled, so the module output is simply $\mathbf{x}$ and consistent with the expectation in the training phase. If there is no scaling factor $1 / p$ in Eq. (1) and $p < 1$, the expectation of the module outputs in the training and test phases will be different, which leads to performance degradation.

Reviewer comment (Theory):
In the Three-Phase Keep Rate section on page 4, it is said that the variance increases as p increases: isn’t the variance maximal at p = 0.5?

Author response:
We are sorry about the confusion, and we agree with your insightful comment. For a Bernoulli distribution, the variance is indeed the largest when $p=0.5$. We have updated the corresponding statement (please check the text highlighted in blue in our revised manuscript). The key point of three-phase keep rate is to control the **effective depth** of the model by the value of $p$. In the early phase of training, the model is underfitting, stochastic architecture brings optimization challenges for training, so we turn off DropPath by setting the keep rate $p = 1$ in the first few epochs to ensure that the network learns meaningful representations. We then gradually decrease $p$ to decrease the effective depth of the model and thus to narrow down the performance gap between the deep sub-network and the shallow training network until the keep rate reaches the predefined minimum value after several epochs. In the final phase of training, we decrease the architecture stochasticity by increasing the value of $p$ to a higher value to ensure training convergence.

Reviewer comment (Experiments):
There seem to be some hyper-parameters involved, such as the values shaping the shape of the three-phase cycle and the temperature in knowledge-distillation. How are these hyper-parameters tuned? The experiments suggest direct evaluation on test set performances.

Author response:
In Section 4.3, we conduct ablation study to evaluate the performance under different hyper-parameters (HPs) when using FRePo-generated distilled dataset, IPC = 10 and ResNet18 as the test network. **More ablation studies on HPs of three-phase keep rate have been added in the revised manuscript**. The results are reported below and in Figure 4 (b) and (c). We can conclude that the performance fluctuation by different HPs in an **appropriate but wide** range is negligible, compared with the significant performance improvement brought by our method. For example, when the final keep rate ranges from 0.6 to 1.0, KD weight ranges from 0.3 to 0.9 and KD temperature ranges from 1 to 10, the performance fluctuation is with 1 percentage point. In addition, although lower minimum keep rates and longer periods of decay induce better performance, they also mean a longer training phase. We set them to $0.5$ and $500$ to balance the efficiency and performance, respectively. Despite that, the performance fluctuation caused by these two HPs is still insignificant. In addition, our results in Section 4.1 demonstrate that the selected HPs by ablation studies also perform quite well under other different settings, including different distilled datasets, IPs and test networks. In summary, many experimental observations indicate that our training method is robust to different HPs.