Paper: Improving Generalization for Small Datasets with Data-Aware Dynamic Reinitialization (ICLR-2024)

Reviewer comment (Experiments):
The work did not study the computational cost comparable with KE. Moreover, I believe the empirical improvement will more convinceable if the authors may also compare the generalization under similar computational costs. Because it is known that DNNs sometimes improve generalization with longer training.

Author response:
In our experiments, we consistently maintained a fixed training duration of 200 epochs for each generation, with the number of generations set at 10 for fair comparison. The computational cost of evolutionary training methods, including both KE and SKE, scales linearly with the number of generations (T). For example, if KE is trained for 5 generations, the total computational cost becomes 5T times that of training a single generation. Similarly to ensure fairer comparison, we train a long baseline for the same number of epochs.
The additional computational cost incurred by SKE for computing data-aware dynamic masking with SNIP is minimal. For instance, on the CUB dataset with a 20% subset, it amounts to 20.3 seconds per generation. This is a one-time calculation performed at the end of each generation. This computational cost can be further reduced by using just 128 samples to estimate the importance without affecting the final performance. Notably, SKE's performance exhibits minimal sensitivity to changes in the subset size, as demonstrated in Appendix, Table 8.

Overall, SKE maintains an equivalent computational cost compared to our long baselines and KE. The slight increase in computational cost in SKE, attributed to weight reinitialization and computing connection sensitivity after each generation, is justified by the substantial improvement in generalization performance. We believe this detailed clarification adequately addresses concerns about computational costs satisfactorily.

Reviewer comment (Theory):
There is theoretical analysis at all. Theoretical understanding under some assumptions will be appreciated.

Author response:
This paper is indeed an empirical paper. While theoretical analysis is essential, our intention was to first demonstrate the practical effectiveness of our proposed method. We hope that theoretical advancements on this topic will be subjects of future study. In general, theoretical studies on why reinitialization during standard training improves generalization are not well explored. We will make a note of this in our future work section, highlighting the need for deeper theoretical analysis to complement our empirical findings.

Reviewer comment (Experiments):
It is not clear how statistically reliable the main conclusions of the paper are. This is because the number of datasets used is small (e.g. the authors evaluate ResNet18 on only four small datasets). Since the datasets are small and there are tens of vision-related datasets available, I'm curious to know why the authors used only four datasets. I don't think including more datasets should be an issue since (again) they are small and ResNet18 is also small. The same applies to the other sections. For example, the authors evaluate robustness to adversarial attacks in one dataset only!

Author response:
Thank you for highlighting the concern about the number of datasets in our study. In response, we have expanded the empirical validation of our method by including results on a new dataset. The revised paper now encompasses evaluations on a total of five small datasets and three large datasets—CIFAR-10, CIFAR-100, and Tiny ImageNet.

This allows us to present a more comprehensive view of the performance of Selective Knowledge Evolution (SKE) across a diverse set of datasets. Notably, in the majority of these datasets, SKE consistently outperforms both Knowledge Evolution (KE) and the longer baseline. These results affirm that SKE brings discernible benefits in terms of improving generalization.

It's worth mentioning that the choice of datasets aligns with the baselines established in the original paper, ensuring a fair comparison and leveraging the availability of their results. Also, we are incorporating results on adversarial perturbation for a new dataset in the revised version of our paper. This addition aims to provide a more comprehensive understanding of the performance and robustness of Selective Knowledge Evolution (SKE) across datasets under different scenarios.

If you have any further questions or if there are specific aspects you would like more clarification on, please let us know.

Reviewer comment (Experiments):
The authors only compare with old reinitialization baselines (KE, DSD, and BAN). There have been more recent layerwise methods, but the authors only compare against those in the appendix (see Table 5). There, the improvement is marginal. Is there a reason the authors chose to not include those other methods in the main paper? Also, why aren't they included in the ResNet50 evaluation? They are also not discussed in the Related Works section, and there are a few recent related works missing as well; e.g. (Sheheryar, et al. 2023) and (Jaehoon, et al. 2022). They should all be discussed in the related works section.

Author response:
The rationale behind featuring comparisons with more recent layerwise methods in Appendix, rather than the main body of the paper, stems from our strategic emphasis on elucidating the distinctive merits and innovations of Selective Knowledge Evolution (SKE). SKE stands out by addressing the entire network, in contrast to layerwise methods like LW and LLF that operate on specific layers with architecture-specific assumptions. Thus, in the main paper, our priority was to underscore the advantages over reinitialization baselines like KE, DSD, and BAN.
In Appendix, we thoughtfully presented additional comparisons, encompassing recent layerwise methods. The marginal improvement observed in these comparisons reinforces the effectiveness of SKE, demonstrating comparable or slightly enhanced performance. Importantly, this decision does not diminish the significance of these comparisons but aligns with our intent to maintain focus in the main paper. The inclusion in the appendix ensures that interested readers can delve into supplementary insights without detracting from the core contributions highlighted in the main text.

We acknowledge the importance of addressing recent related works suggested by the reviewer, including (Sheheryar, et al. 2023) and (Jaehoon, et al. 2022). In the revised manuscript, we incorporated discussions of these works in the Related Works section to ensure a comprehensive overview of the literature in the field.

Reviewer comment (Evaluation):
The reported results for CIFAR10-C seem too low to me. I would expect ResNet18 trained on CIFAR10 to have an accuracy larger than 60% when evaluated on CIFAR10-C.

Author response:
Thank you for your insightful observation. The reported results for CIFAR10-C are indeed based on a model trained on the CUB dataset and evaluated on CIFAR10-C as part of our worst-case out-of-domain robustness analysis, rather than the standard CIFAR10 dataset. This distinction has now been made explicit in the revised version of our paper to avoid any potential confusion. Additionally, we will include the results of a model trained on CIFAR10 and evaluated on CIFAR10-C in the revised version.

The Mean Corruption Accuracy (mCA) metric is calculated as follows:

$mCA = \frac{1}{{N_c \times N_s}} \sum_{c=1}^{N_c} \sum_{s=1}^{N_s} A_{c, s}$

Where $N_c$ represents the number of corruptions (in this case, 19), and $N_s$ represents the number of severity levels (in this case, 5). The variable $​A_{c, s}$  denotes the F1-score measure evaluated on CIFAR-C under the c-th corruption with the s-th severity level. The mCA metric provides an average performance measure across all corruptions and severity levels, offering a comprehensive evaluation of the model's robustness under different forms and degrees of corruption.

Reviewer comment (Novelty):
The authors argue in the appendix that transfer learning is not useful for medical applications. But that's not true. Many SotA models are pretrained on datasets, like ImageNet. See for example: https://www.nature.com/articles/s41591-020-0842-3.

Author response:
We want to clarify that our intention was not to claim that transfer learning is not useful for medical applications. Instead, we aimed to highlight a limitation often associated with transfer learning – domain shift. In situations with limited datasets, domain shifts can significantly impact performance. We acknowledge the importance of transfer learning in various contexts, including medical applications, and we adjusted the corresponding statements in the revised paper to better convey this nuanced perspective.

Reviewer comment (Writing):
There are a few places containing typos, incomplete sentences, or undefined symbols:  
- Page 2: "Finally, Our work ... " --> "Finally, our work ..."  
- Equation 4: $\pi$ is undefined.  
- Page 4: "Due to the difficulty in deciding ..." is not a complete sentence.  
- Page 4: "retained learned" should be either "retained" or "learned".

Author response:
We want to acknowledge that these issues have been duly rectified in the revised version of the manuscript. Thank you for your meticulous review.

Reviewer comment (Evaluation):
Please explain precisely how the Mean Corruption Accuracy metric is calculated?

Author response:
The Mean Corruption Accuracy (mCA) metric is calculated as follows:

$mCA = \frac{1}{{N_c \times N_s}} \sum_{c=1}^{N_c} \sum_{s=1}^{N_s} A_{c, s}$

Where $N_c$ represents the number of corruptions (in this case, 19), and $N_s$ represents the number of severity levels (in this case, 5). The variable $​A_{c, s}$  denotes the F1-score measure evaluated on CIFAR-C under the c-th corruption with the s-th severity level. The mCA metric provides an average performance measure across all corruptions and severity levels, offering a comprehensive evaluation of the model's robustness under different forms and degrees of corruption.

Reviewer comment (Evaluation):
The main issue with the paper is its presentation of the results on imbalanced data. The authors use accuracy as a metric, which can be highly misleading on imbalanced data. I cannot infer anything from figure 4 on the robustness to imbalanced data. There are many ways to show performance on imbalanced data, including confusion matrices or per-class accuracy. I suggest the authors either improve this presentation or remove entirely the class imbalance experiments (or correct the figure and text description if they do not mean overall accuracy). In its current form the results are not supporting the conclusions.

Author response:
We appreciate your insightful comment. We would like to clarify that the accuracy mentioned in the context of class imbalance is indeed the balanced accuracy. First we evaluate the Recall for each class, then we average the values in order to obtain the Balanced Accuracy score. We have taken steps to enhance the clarity of our presentation in the revised version. Moreover, we have reorganized of our paper based on your suggestions. The robustness to class imbalance experiments has been moved to the appendix (Appendix Section A5), allowing us to highlight the significance of sensitivity analysis in the main paper (Section 5.5). We trust that this adjustment will contribute to a clearer and more effective presentation of our findings.

Reviewer comment (Presentation):
The adversarial perturbation experiments are also lacking in presentation. Is the figure a single run or multiple runs? What are the errors?

Author response:
To address this concern, we modified the figure to include error bars to provide a more comprehensive representation of the experimental results.

Reviewer comment (Experiments):
Sensitivity analysis would have been a nice addition, the authors leave this for future work. However, the paper would have been notably stronger with these included and should be relatively inexpensive to run.

Author response:
Thank you for your valuable suggestion regarding the inclusion of sensitivity analysis in our paper. Table 4 shows the effect of varying the number of reinitialized parameters on the performance and generalization of the model. We train the model in evolutionary settings using the SKE framework by varying different percentages of reinitialized parameters (5%, 10%, 20%, 30%, and 40%). Experiments were carried out with ResNet18. The results show that the reinitialization of a 5% percentage of parameters has no impact on performance, while reinitialization of more than 30% has less impact on test accuracy. We find that reinitialization 20% of the parameters results in the best performance.  We agree that this would be a valuable addition to strengthen our work, and we appreciate your insight.

Reviewer comment (Writing):
In the abstract and introduction, the authors state that the KE approach is limited due to its predetermined mask. I can see that this as a valid hypothesis, but the authors state this as if it is well-established. Are there any references to support this? With the posterior knowledge of the results of SKE this statement is supported, but the hypothesis which sparked the investigation cannot be based on the results.

Author response:
Thank you for raising this point. The motivation drawn from neurogenesis indeed inspired our exploration into the impact of a fixed mask in the context of deep neural networks. The intricate process of neurogenesis, as highlighted in the literature (Shors et al., 2001; Garthe et al., 2016; Kempermann et al., 2015), plays a crucial role in learning and memory consolidation, allowing the brain to adapt to new experiences and stimuli.
The specific insight into non-random integration and synaptic refinement observed in rodents' hippocampus, where the integration of new neurons leads to the elimination of less active synaptic connections (Aimone et al., 2014; Vadodaria & Gage, 2014), further fueled our curiosity. This led us to consider whether a fixed mask, akin to a predetermined synaptic connection pattern, could be limiting the adaptability and generalization capabilities of deep neural networks.
Our exploration, guided by the emulation of selective neurogenesis, questioned the assumed fixed mask in traditional approaches such as KE. We hypothesized that a more dynamic and selective approach, akin to the characteristics of selective neurogenesis, could potentially unlock greater potential in terms of generalization in deep neural networks. Therefore, our study sought to investigate and propose a solution, Selective Knowledge Evolution (SKE), which embraces a more adaptive and dynamic approach to reinitialization, inspired by the principles of neurogenesis.

Reviewer comment (Experiments):
How is the size of subset used to evaluate connection sensitivity selected? Is there any estimate for what is sufficient?

Author response:
To study this, we vary the quantity of data used for Importance estimation to analyze its effect on performance. In our experiments, we randomly sampled 20% of the dataset to estimate the importance of the parameters after the end of each generation. Here, we analyze the impact of the number of data used to determine the important estimation on the final performance. Similarly, we used as few as 128 samples to estimate the important parameters using SNIP. Table 8 in Appendix shows that SKE is not sensitive to the variation in the input data used to estimate the importance, as the final performance remains unchanged.

Reviewer comment (Experiments):
Sensitivity analysis is not part of this paper but how do the authors interpret the sparsity constraint k? Is there anything in the literature or insight they have as to how sensitive the model is to it?

Author response:
In response to your inquiry, we have incorporated additional experiments to systematically determine an appropriate subset size for evaluating connection sensitivity and the sparsity constraint k.  The results in Table 4 show that the reinitialization of a 5% percentage of parameters has no impact on performance, while reinitialization of more than 30% has less impact on test accuracy. We find that reinitialization 20% of the parameters results in the best performance. 
In the revised manuscript, we included a detailed discussion of the experimental setup for subset size selection and sensitivity in the Sections 5.5 and A7, along with the findings from these experiments.

Reviewer comment (Evaluation):
There are instances where the word "significant" is used to describe the difference between two methods. I highly recommend that the authors save this term to describe statistical significance (it seems a significance test was not performed). There are better words to describe a great difference that are not as ambiguous.

Author response:
Regarding the use of *'significantly'* without statistical significance testing, we acknowledge the concern and recognize the importance of substantiating such language. Going forward, we will either rephrase our sentence or ensure its validation through statistical analysis in the final revisions.

Reviewer comment (Reproducibility):
Further discussion of the proposed algorithm is needed. For example, how are scores within the SNIP method computed? What is the computation complexity of estimating such scores during the training process? Would it lead to additional overhead?

Author response:
Thank you for your insightful comments. We appreciate the opportunity to address your concerns regarding the computational aspects of our proposed Selective Knowledge Evolution (SKE) algorithm. 

The SNIP method, employed in our proposed Selective Knowledge Evolution (SKE) algorithm, identifies prunable weights at initialization without the need for full network training. It assesses the relevance of each weight through the normalized gradient of the loss with respect to an implicit multiplicative factor on the weights, referred to as "sensitivity." We rank the weights based on their sensitivity and only keep the top parameters based on the sparsity constraint (k), randomly reinitializing the rest before the start of each generation. Further details on this process can be found in Section 3.2.

In our experiments, we consistently maintained a fixed training duration of 200 epochs for each generation, with the number of generations set at 10 for fair comparison. The computational cost of evolutionary training methods, including both KE and SKE, scales linearly with the number of generations (T). For example, if KE is trained for 5 generations, the total computational cost becomes 5T times that of training a single generation. Similarly to ensure fairer comparison, we train a long baseline for the same number of epochs.
The additional computational cost incurred by SKE for computing data-aware dynamic masking with SNIP is minimal. For instance, on the CUB dataset with a 20% subset, it amounts to 20.3 seconds per generation. This is a one-time calculation performed at the end of each generation. This computational cost can be further reduced by using just 128 samples to estimate the importance without affecting the final performance. Notably, SKE's performance exhibits minimal sensitivity to changes in the subset size, as demonstrated in Appendix, Table 8.

Overall, SKE maintains an equivalent computational cost compared to our long baselines and KE. The slight increase in computational cost in SKE, attributed to weight reinitialization and computing connection sensitivity after each generation, is justified by the substantial improvement in generalization performance. We believe this detailed clarification adequately addresses concerns about computational costs satisfactorily.

Reviewer comment (Experiments):
More recent baselines need to be compared. This paper conducts a comparison with previous retraining and reinitialization algorithms. However, for training on small datasets, many regularization and training algorithms are proposed, such as sharpness-aware minimization and distance-based regularization. How would the proposed method compare to such methods? Moreover, how does the method perform on transformer-based architectures?

Author response:
Thank you for your suggestions. Both methodologies, Selective Knowledge Evolution (SKE) and Sharpness-Aware Minimization (SAM), share the common objective of enhancing generalization. However, they employ distinct strategies to achieve this goal.
In SAM, the emphasis lies on simultaneously minimizing both loss value and loss sharpness. This involves encouraging parameters that lead to neighborhoods with uniformly low loss, ultimately aiming to improve overall model quality and generalization.
Contrastingly, SKE follows a unique path. It identifies the least important parameters affecting the loss through data-aware dynamic masking using SNIP. By reinitializing these parameters and subsequently retraining the model, SKE introduces a level of noise or randomness into the network. The alteration of connection sensitivity aims to enhance generalization by freeing up the network's capacity to learn generalized information.

In addition, there are distance-based regularization techniques like DELTA, which proposes a regularized transfer learning framework preserving the outer layer outputs of the target network. These methods align outer layer outputs using attention mechanisms or by regularizing weights with Euclidean distance. Although these approaches effectively address overfitting, they differ from SKE.
It's important to note that our work stands apart in that it doesn't rely on a teacher network pretrained on a large dataset. Our models are trained from scratch directly on smaller datasets, aiming to avoid overfitting. Conducting direct comparisons with distance-based methods from transfer learning or SAM isn't straightforward and may not align with the specific focus of our paper. These divergent approaches underscore the diversity of techniques employed to tackle the challenge of model generalization.

Reviewer comment (Experiments):
Further experiments can be conducted to analyze the algorithm. For example, how can one set the number of generations and the masking ratios in the algorithm? How would these parameters affect the model performance? With such a retraining algorithm, would the model converge faster than vanilla fine-tuning?

Author response:
We appreciate the insightful suggestion regarding further experiments to analyze our algorithm. Parameters such as the number of generations, sparsity constraint (k), and the percentage of data used to calculate importance play crucial roles in shaping the performance of the retraining algorithm. In practice, the choice of the number of generations is often influenced by the characteristics of the dataset, the complexity of the task, and the availability of computational resources.

To study the impact of masking ratios, we performed experiments by varying the masking ratio K and evaluate the impact on performance.  Table below (Table 4 in the revised paper) shows the effect of varying the number of reinitialized parameters on the performance and generalization of the model. We train the model in evolutionary settings using the SKE framework by varying different percentages of reinitialized parameters (5%, 10%, 20%, 30%, and 40%). Experiments were carried out with ResNet18 on two datasets. The results show that the reinitialization of a 5% percentage of parameters has no impact on performance, while the reinitialization of 30% and more has less impact on test accuracy. We find that reinitialization 20% of the parameters results in the best performance. 

|                    | Reinitialized Params (%) | Aircraft | CUB    |
|--------------------|---------------------------|----------|--------|
| **SKE**            | 5                         | 65.34    | 69.95  |
|                    | 10                        | 66.10    | 70.15  |
|                    | 20                        | **66.63**| **71.37** |
|                    | 30                        | 64.13    | 68.42  |
|                    | 40                        | 62.79    | 66.87  |


Further, we vary the quantity of data used for importance estimation to analyze its effect on performance. In our experiments, we randomly sampled 20% of the dataset to estimate the importance of the parameters after the end of each generation. Here, we analyze the impact of the number of data points used to determine the important estimation on the final performance. Similarly, we used as few as 128 samples to estimate the important parameters using SNIP. Table 8 in Appendix shows that SKE is not sensitive to the variation in the input data used to estimate the importance, as the final performance remains unchanged.

Our experimental results in Appendix Section A3 indeed reveal that the Selective Knowledge Evolution (SKE) algorithm tends to converge faster than vanilla fine-tuning across various datasets and architectures. The faster convergence observed in SKE can be attributed to its unique methodology of selectively reinitializing less important parameters. By dynamically identifying and retraining these parameters, SKE facilitates faster adaptation to the underlying data distribution, thus contributing to a more efficient learning process.