Paper: Fully Identical Initialization (ICLR-2024)

Reviewer comment (Novelty):
(clarity) The idea of dynamical isometry has been introduced in (Saxe et al., 2014).

Author response:
We acknowledge the seminal work presented in [1] regarding the concept of dynamical isometry in the context of non-residual networks. The approach outlined in [1] focuses on achieving dynamical isometry by orthogonalizing network weights to ensure that the mean squared singular value of the Jacobian matrix remains close to 1, a significant advancement for non-residual network initialization.

However, our work with IDInit diverges from [1] in its application to residual networks, which are among the most powerful and prevalent structures in current deep learning architectures. In technique, IDInit emphasizes the maintenance of identity transit through both the main and residual stems, and empirical results show the effectiveness.

In light of this, while IDInit and the method in [1] both target dynamical isometry, IDInit's contribution lies in its novel application to residual networks. This extension not only broadens the understanding of dynamical isometry but also showcases its practical utility in some of the most advanced neural network architectures today. We will include a detailed discussion of these distinctions and contributions in the revised manuscript.

> Please, include the references listed in the weaknesses section.

Thank you for your suggestion. \[1\] pioneered the integration of dynamic isometry in nonlinear networks, enhancing training efficiency. Specifically targeting Convolutional Neural Networks (CNNs), Reference \[2\] formulated a mean field theory for signal propagation adhering to dynamic isometry principles, enabling the training of networks with up to 10,000 layers. Different from these approaches, the proposed IDInit maintains identity in both the main and residual stems to preserve dynamic isometry to achieve good performance and fast convergence. Furthermore, we adapt the transformation on the identity matrix to accommodate various scenarios, such as nonsquare matrices and higher-order tensors. We will include these references in the revision.

Reviewer comment (Writing):
(clarity) I would argue that the patch-maintain convolution is not very well motivated. I believe the problem is that I do not understand how this relates to Ghostnets (Han et al., 2020).

Author response:
Thanks for the useful comment. Ghostnet \[3\] highlights the critical role of channel diversity in enhancing network performance. This insight serves as a foundational motivation for our patch-maintain scheme. In contrast to the channel-maintain approach, which tends to replicate channel features and thus might limit diversity, our patch-maintain scheme introduces a novel strategy to increase channel differentiation. By shifting features, the patch-maintain scheme aims to add variability and uniqueness across channels. This method is designed to capitalize on the principle articulated in Ghostnet – that channel diversity is beneficial for performance. This linkage between the need for channel diversity (as emphasized by Ghostnet) and our patch-maintain scheme is a key aspect of our work. We will include this discussion in the revision.

Reviewer comment (Writing):
(clarity) It also took me some time to realise that the channel-maintain convolution (as it is called in the appendix) which is described in the first paragraph of Section 3.3.1 is something different from the proposed patch-maintain setting. Note that this channel-maintain setting has also been used in (Xiao et al., 2018).

Author response:
Sorry for the confusion. In this work, we introduce the patch-maintain scheme (i.e., IDIC$_{\tau}$) as our novel method for transforming a matrix into a convolutional format.

On the other hand, the channel-maintain method, referenced from prior studies [2][4], serves as a comparison baseline in our experiments, denoted as 'w/o IDIC$_{\tau}$'. In Section 4.4, we present this comparative analysis, where we demonstrate that the patch-maintain scheme significantly enhances the performance of IDInit. To address the clarity issues raised, we will revise our manuscript for more explicit elaboration.

Reviewer comment (Evaluation):
(clarity) The ablation experiment in Section 4.4 is claimed to explain why channel-maintain is not as good as patch-maintain. However, the explanation in section 4.4 seems to indicate that this is just an ablation of the different components of the proposed solution.

Author response:
Sorry for the confusion. In this work, we introduce the patch-maintain scheme (i.e., IDIC$_{\tau}$) as our novel method for transforming a matrix into a convolutional format.

On the other hand, the channel-maintain method, referenced from prior studies [2][4], serves as a comparison baseline in our experiments, denoted as 'w/o IDIC$_{\tau}$'. In Section 4.4, we present this comparative analysis, where we demonstrate that the patch-maintain scheme significantly enhances the performance of IDInit. To address the clarity issues raised, we will revise our manuscript for more explicit elaboration.

Reviewer comment (Presentation):
(clarity) I can not find any explanation for the legend of Figure 7 (a).

Author response:
Sorry for lacking the explanation for the legend in Figure 7 (a).  The numbers following the initialization methods in the legend represent the duration of the warmup period, measured in epochs. Specifically, (i) IDInit-0 and IDInit-10 indicate that the IDInit method was used with a warmup period of 0 and 10 epochs, respectively; (ii) Kaiming-10 and Kaiming-40 refer to the Kaiming initialization method applied with warmup periods of 10 and 40 epochs, respectively. The inclusion of different warmup durations is crucial for evaluating the performance of these initialization methods under varying conditions of network adaptation at the beginning of the training process. We will add a description of the warmup epochs and their implications in the revised version.

Reviewer comment (Experiments):
(quality) The choice of hyper-parameters is not motivated properly and it is unclear how they were chosen. Moreover, it seems like the same hyper-parameter settings were used for every network. For a fair comparison, hyper-parameters should be set for each method individually.

Author response:
Investigating the influence of hyperparameters is important to validate the effectiveness of IDInit. In response, we have conducted an extensive set of experiments to analyze the effect of weight decay and learning rate on the performance of IDInit compared to baselines. These experiments are constructed on Cifar10 with ResNet-32, and detailed in Section F.1 in the uploaded revision PDF, with partial cases IDInit and Kaiming presented in RTable 1 and RTable 2, respectively.

RTable 1. Hyperparameters for IDInit on ResNet-32.

|         | lr=1e0                | lr=1e-1               | lr=1e-2               | lr=1e-3               |
| ------- | --------------------- | --------------------- | --------------------- | --------------------- |
| wd=1e-1 | 16.38$_{±1.28}$ | 21.68$_{±2.61}$ | 68.20$_{±5.79}$ | 82.17$_{±0.14}$ |
| wd=1e-2 | 22.57$_{±2.69}$ | 90.36$_{±0.06}$ | 93.28$_{±0.04}$ | 83.18$_{±0.14}$ |
| wd=1e-3 | 89.76$_{±0.18}$ | 94.08$_{±0.13}$ | 90.09$_{±0.22}$ | 81.42$_{±0.13}$ |
| wd=1e-4 | 94.08$_{±0.08}$ | 92.54$_{±0.11}$ | 88.33$_{±0.03}$ | 81.15$_{±0.26}$ |
| wd=1e-5 | 93.27$_{±0.21}$ | 91.92$_{±0.29}$ | 88.56$_{±0.13}$ | 81.65$_{±0.34}$ |
| wd=1e-6 | 92.94$_{±0.18}$ | 91.91$_{±0.12}$ | 88.47$_{±0.10}$ | 80.76$_{±0.61}$ |
| wd=1e-7 | 93.02$_{±0.20}$ | 91.80$_{±0.18}$ | 88.74$_{±0.24}$ | 81.00$_{±0.44}$ |
| wd=1e-8 | 92.99$_{±0.19}$ | 91.96$_{±0.11}$ | 88.82$_{±0.34}$ | 81.42$_{±0.78}$ |

RTable 2. Hyperparameters for Kaiming on ResNet-32.

|         | lr=1e0                | lr=1e-1               | lr=1e-2               | lr=1e-3               |
| ------- | --------------------- | --------------------- | --------------------- | --------------------- |
| wd=1e-1 | 10.38$_{±0.35}$ | 21.12$_{±2.73}$ | 51.88$_{±5.98}$ | 69.52$_{±7.60}$ |
| wd=1e-2 | 21.77$_{±3.10}$ | 90.23$_{±0.18}$ | 92.78$_{±0.14}$ | 55.26$_{±1.14}$ |
| wd=1e-3 | 89.96$_{±0.49}$ | 93.69$_{±0.12}$ | 82.02$_{±0.01}$ | 51.25$_{±1.17}$ |
| wd=1e-4 | 94.06$_{±0.11}$ | 91.24$_{±0.10}$ | 78.37$_{±0.41}$ | 50.75$_{±1.26}$ |
| wd=1e-5 | 93.06$_{±0.07}$ | 90.14$_{±0.12}$ | 77.83$_{±0.43}$ | 50.81$_{±1.48}$ |
| wd=1e-6 | 92.60$_{±0.14}$ | 90.01$_{±0.17}$ | 78.05$_{±0.33}$ | 50.91$_{±1.38}$ |
| wd=1e-7 | 92.64$_{±0.11}$ | 90.00$_{±0.14}$ | 78.32$_{±0.58}$ | 51.00$_{±1.22}$ |
| wd=1e-8 | 92.48$_{±0.25}$ | 89.93$_{±0.09}$ | 77.96$_{±0.21}$ | 50.79$_{±1.49}$ |

As shown in RTable 1 and RTable 2, across various settings of learning rate and weight decay, IDInit consistently outperforms or matches Kaiming initialization. It demonstrates that IDInit's performance superiority is not confined to a specific set of hyperparameters but is maintained across a range of settings. Moreover, at lower learning rates (1e-2 and 1e-3), where Kaiming initialization's performance drops significantly, whereas IDInit maintains high accuracy. This robustness of IDInit, even under smaller learning rates, highlights its effectiveness in achieving isometry dynamics in the residual stem.

In light of these findings, IDInit showcases the generalizability and effectiveness of IDInit across various hyperparameter configurations, indicating the promising utility of IDInit to practical application.

Reviewer comment (Experiments):
(quality) An experiment without learning rate schedule, weight decay and other extras would be interesting for a more "raw" comparison between initialisation strategies.

Author response:
Investigating the influence of hyperparameters is important to validate the effectiveness of IDInit. In response, we have conducted an extensive set of experiments to analyze the effect of weight decay and learning rate on the performance of IDInit compared to baselines. These experiments are constructed on Cifar10 with ResNet-32, and detailed in Section F.1 in the uploaded revision PDF, with partial cases IDInit and Kaiming presented in RTable 1 and RTable 2, respectively.

RTable 1. Hyperparameters for IDInit on ResNet-32.

|         | lr=1e0                | lr=1e-1               | lr=1e-2               | lr=1e-3               |
| ------- | --------------------- | --------------------- | --------------------- | --------------------- |
| wd=1e-1 | 16.38$_{±1.28}$ | 21.68$_{±2.61}$ | 68.20$_{±5.79}$ | 82.17$_{±0.14}$ |
| wd=1e-2 | 22.57$_{±2.69}$ | 90.36$_{±0.06}$ | 93.28$_{±0.04}$ | 83.18$_{±0.14}$ |
| wd=1e-3 | 89.76$_{±0.18}$ | 94.08$_{±0.13}$ | 90.09$_{±0.22}$ | 81.42$_{±0.13}$ |
| wd=1e-4 | 94.08$_{±0.08}$ | 92.54$_{±0.11}$ | 88.33$_{±0.03}$ | 81.15$_{±0.26}$ |
| wd=1e-5 | 93.27$_{±0.21}$ | 91.92$_{±0.29}$ | 88.56$_{±0.13}$ | 81.65$_{±0.34}$ |
| wd=1e-6 | 92.94$_{±0.18}$ | 91.91$_{±0.12}$ | 88.47$_{±0.10}$ | 80.76$_{±0.61}$ |
| wd=1e-7 | 93.02$_{±0.20}$ | 91.80$_{±0.18}$ | 88.74$_{±0.24}$ | 81.00$_{±0.44}$ |
| wd=1e-8 | 92.99$_{±0.19}$ | 91.96$_{±0.11}$ | 88.82$_{±0.34}$ | 81.42$_{±0.78}$ |

RTable 2. Hyperparameters for Kaiming on ResNet-32.

|         | lr=1e0                | lr=1e-1               | lr=1e-2               | lr=1e-3               |
| ------- | --------------------- | --------------------- | --------------------- | --------------------- |
| wd=1e-1 | 10.38$_{±0.35}$ | 21.12$_{±2.73}$ | 51.88$_{±5.98}$ | 69.52$_{±7.60}$ |
| wd=1e-2 | 21.77$_{±3.10}$ | 90.23$_{±0.18}$ | 92.78$_{±0.14}$ | 55.26$_{±1.14}$ |
| wd=1e-3 | 89.96$_{±0.49}$ | 93.69$_{±0.12}$ | 82.02$_{±0.01}$ | 51.25$_{±1.17}$ |
| wd=1e-4 | 94.06$_{±0.11}$ | 91.24$_{±0.10}$ | 78.37$_{±0.41}$ | 50.75$_{±1.26}$ |
| wd=1e-5 | 93.06$_{±0.07}$ | 90.14$_{±0.12}$ | 77.83$_{±0.43}$ | 50.81$_{±1.48}$ |
| wd=1e-6 | 92.60$_{±0.14}$ | 90.01$_{±0.17}$ | 78.05$_{±0.33}$ | 50.91$_{±1.38}$ |
| wd=1e-7 | 92.64$_{±0.11}$ | 90.00$_{±0.14}$ | 78.32$_{±0.58}$ | 51.00$_{±1.22}$ |
| wd=1e-8 | 92.48$_{±0.25}$ | 89.93$_{±0.09}$ | 77.96$_{±0.21}$ | 50.79$_{±1.49}$ |

As shown in RTable 1 and RTable 2, across various settings of learning rate and weight decay, IDInit consistently outperforms or matches Kaiming initialization. It demonstrates that IDInit's performance superiority is not confined to a specific set of hyperparameters but is maintained across a range of settings. Moreover, at lower learning rates (1e-2 and 1e-3), where Kaiming initialization's performance drops significantly, whereas IDInit maintains high accuracy. This robustness of IDInit, even under smaller learning rates, highlights its effectiveness in achieving isometry dynamics in the residual stem.

In light of these findings, IDInit showcases the generalizability and effectiveness of IDInit across various hyperparameter configurations, indicating the promising utility of IDInit to practical application.

Reviewer comment (Experiments):
My biggest concern is the Cifar-10 performance compared to the simple Kaiming initialization. Table 2 shows that using SGD optimizer (which gives the best performance all across), Kaiming init obtains almost the same performance (93.36 v/s 93.41 and 94.06 v/s 94.04) as IDInit, while being only slightly slower. This brings into question the practical utility of the proposed initialization.

Author response:
Thanks for the reviewer’s comment. While both IDInit and Kaiming initialization methods perform similarly in certain settings, our research indicates that IDInit offers superior stability and adaptability across a wider range of scenarios. To verify this claim, we have conducted additional experiments with varied hyperparameters, particularly focusing on weight decay and learning rate, using ResNet-56. 

RTable 1. Hyperparameters for IDInit on ResNet-56.

|         | lr=1e0                | lr=1e-1               | lr=1e-2               | lr=1e-3               |
| ------- | --------------------- | --------------------- | --------------------- | --------------------- |
| wd=1e-1 | 10.00$_{±0.00}$ | 18.27$_{±0.32}$ | 74.46$_{±3.52}$ | 88.67$_{±0.23}$ |
| wd=1e-2 | 18.10$_{±1.95}$ | 90.37$_{±0.14}$ | 94.18$_{±0.06}$ | 84.07$_{±0.53}$ | 
| wd=1e-3 | 89.18$_{±0.24}$ | 94.64$_{±0.16}$ | 89.99$_{±0.06}$ | 81.65$_{±0.69}$ |
| wd=1e-4 | 94.97$_{±0.04}$ | 92.60$_{±0.05}$ | 88.83$_{±0.30}$ | 82.47$_{±0.12}$ |

RTable 2. Hyperparameters for Kaiming on ResNet-56.

|         | lr=1e0                | lr=1e-1               | lr=1e-2               | lr=1e-3               |
| ------- | --------------------- | --------------------- | --------------------- | --------------------- |
| wd=1e-1 | 10.00$_{±0.00}$ | 17.39$_{±0.47}$ | 48.98$_{±4.11}$ | 82.36$_{±0.42}$ |
| wd=1e-2 | 14.43$_{±4.78}$ | 90.10$_{±0.11}$ | 94.19$_{±0.09}$ | 55.44$_{±1.07}$ |
| wd=1e-3 | 87.78$_{±1.20}$ | 94.64$_{±0.12}$ | 82.44$_{±0.49}$ | 49.85$_{±2.38}$ |
| wd=1e-4 | 94.89$_{±0.37}$ | 91.52$_{±0.33}$ | 78.42$_{±0.06}$ | 50.32$_{±2.71}$ |

As shown in RTable 1 and RTable 2, these experiments reveal that IDInit consistently surpasses Kaiming initialization in almost every setting tested. Most notably, in scenarios with smaller learning rates (e.g., 1e-2 and 1e-3), we observe a significant drop in the performance of models initialized with Kaiming. In contrast, IDInit maintains high accuracy levels under these conditions. This stability of IDInit is attributable to its ability to achieve isometry dynamics in the residual stem, a crucial factor for maintaining performance stability.

Further, we have extended our experimentation to include ResNet-32 on the CIFAR-10 dataset, as detailed in Section F.1 of the uploaded revision PDF. These additional results consistently demonstrate that IDInit not only matches but often exceeds the performance of Kaiming initialization, especially under varying and challenging hyperparameter settings.

In summary, while Kaiming initialization performs comparably in certain standard scenarios, IDInit exhibits greater performance consistency and stability across a broader spectrum of conditions. This enhanced stability and adaptability make IDInit a more practically valuable choice for diverse neural network applications. We will ensure that these findings are clearly presented in our revised manuscript to underscore the practical utility and advantages of IDInit over traditional initialization methods like Kaiming.

Reviewer comment (Experiments):
IDInit is compared with other initialization methods only on Cifar-10, which is very small-scale. No such comparisons have been shown on ImageNet. I think it's important to see if the proposed init is even useful when training on large-scale datasets.

Author response:
Thanks for the suggestion. We have expanded our experimental analysis to include comparisons with other initialization methods, namely Fixup and ZerO, on ImageNet. These experiments were conducted under the same settings as for ResNet-50, described in Section 4.3 of our manuscript.

RTable 3. Results on ImageNet. The value in brackets means Epochs to 60\% Acc.

| Kaiming    | Fixup      | ZerO       | IDInit     |
| ---------- | ---------- | ---------- | ---------- |
| 75.70 (38) | 76.24 (25) | 75.73 (35) | 76.72 (24) |

As presented in RTable 3, the results demonstrate that IDInit simultaneously achieves high performance and fast convergence. Specifically, IDInit achieves the highest accuracy among all the compared methods. In terms of convergence speed, IDInit matches Fixup and significantly surpasses Kaiming and ZerO by 14 and 11 epochs, respectively.

This finding is particularly noteworthy as it illustrates that IDInit's advantages are not limited to small-scale datasets. Instead, its efficacy extends to large-scale training scenarios, highlighting its consistency in performance and convergence. We will incorporate these insights and results into our revised manuscript to comprehensively demonstrate IDInit's utility across different scales of datasets, thereby addressing the concerns regarding its practicality and scalability.

Reviewer comment (Theory):
While the paper provides a theoretical analysis of IDInit, it mainly focuses on the Jacobian and gradient update equations in residual neural networks. It would be valuable to extend the theoretical analysis to other network architectures and provide a more comprehensive understanding of the underlying principles of IDInit.

Author response:
Thanks for your insightful comments regarding the theoretical analysis of IDInit in the context of residual neural networks. We acknowledge the importance of extending this analysis to other architectures for a more comprehensive understanding. Our decision to concentrate on residual networks was guided by their widespread use and relevance in current research on deep learning. While we recognize the value of broadening our theoretical framework, such a comprehensive analysis was beyond the scope of this particular study, primarily due to the complicated variations in network architectures that demand distinct analytical approaches. Nevertheless, the findings of our current research offer significant insights into the functioning of IDInit in residual networks, contributing valuable knowledge to the broader field. We are enthusiastic about the prospect of extending our work to include other architectures and consider this an important direction for our future research endeavors.

Reviewer comment (Theory):
Although the paper briefly mentions the limitations of IDInit, such as the deterministic nature and the need for momentum to handle negative eigenvalues, further discussion and analysis of these limitations would provide a more comprehensive understanding of the potential drawbacks and challenges of implementing IDInit in practical scenarios.

Author response:
Thanks for the comment on the further discussion on the limitations. Regarding the convergence ability of IDInit, we add a theoretical analysis in Appendix F.2 of the uploaded revision PDF, which shows the mechanics of SGD w/ momentum to resolve the convergent problem by breaking symmetries of each layer weight. As for the deterministic nature, we have discussed the IDInit can already be in a good initial state that can exceed random initialization, and the whole training still contains randomness (e.g., sequence of samples) in the training process. Moreover, as acknowledged by reviewer GXx1, the deterministic nature could reduce some of the randomness in training networks, which helps comparison among networks become easier. Therefore, these two limitations are not likely to cause severe damage to the convergence and performance of IDInit. We will add this elaboration to the revision for better understanding.

Reviewer comment (Experiments):
Reference to weakness W2, can you please provide more insight into the efficacy of the proposed init method on large-scale training datasets, like ImageNet?

Author response:
Thanks for the suggestion. We have expanded our experimental analysis to include comparisons with other initialization methods, namely Fixup and ZerO, on ImageNet. These experiments were conducted under the same settings as for ResNet-50, described in Section 4.3 of our manuscript.

RTable 3. Results on ImageNet. The value in brackets means Epochs to 60\% Acc.

| Kaiming    | Fixup      | ZerO       | IDInit     |
| ---------- | ---------- | ---------- | ---------- |
| 75.70 (38) | 76.24 (25) | 75.73 (35) | 76.72 (24) |

As presented in RTable 3, the results demonstrate that IDInit simultaneously achieves high performance and fast convergence. Specifically, IDInit achieves the highest accuracy among all the compared methods. In terms of convergence speed, IDInit matches Fixup and significantly surpasses Kaiming and ZerO by 14 and 11 epochs, respectively.

This finding is particularly noteworthy as it illustrates that IDInit's advantages are not limited to small-scale datasets. Instead, its efficacy extends to large-scale training scenarios, highlighting its consistency in performance and convergence. We will incorporate these insights and results into our revised manuscript to comprehensively demonstrate IDInit's utility across different scales of datasets, thereby addressing the concerns regarding its practicality and scalability.

Reviewer comment (Novelty):
Authors mention that dead neurons problem happens when batch normalization is set to 0 or downsampling operation causes 0-filled features. However, these cases are not common in practice and it's better to motivate more on why IDIZ is important.

Author response:
Thank you for the comment on IDIZ in addressing the issue of dead neurons, particularly in relation to batch normalization (BN) and downsampling operations. However, we contend that the scenarios we described are common in practice.

- Batch Normalization: The practice of setting the gamma parameter of the last BN layer in a ResNet block to zero is not only recommended for enhanced performance as indicated in references \[1\]\[2\]\[3\], but also a default setting in widely-used packages like timm \[4\]. This approach, while improving performance, can inadvertently lead to dead neurons, a problem IDIZ aims to mitigate.

- Downsampling: In common downsampling practices, there are two prevalent approaches: (i): Using pooling (e.g., avgpool) to reduce feature resolution, coupled with zero-padding to expand channel size (as shown in Line 40 of \[5\] and option 'A' in Line 66 of \[6\]). (ii): Employing convolution to simultaneously reduce resolution and expand channel size (evident in Line 241 of \[7\] and option 'B' in Line 72 of \[6\]). In particular, for the widely-used datasets like CIFAR-10, option (i) is frequently preferred.

IDIZ is designed to generalize the utilization of IDInit by solving dead neurons of above conditions which is also common settings. So that IDIZ can be used more widely for addressing the disharmonious nature of identity-control methods, which avoids potential risks in compatibility with other techniques. We believe this is important characteristic to an initialization algorithm.

We will revise our manuscript to better discuss these points, ensuring that the relevance and utility of IDIZ are clearly and concisely presented.

Reviewer comment (Theory):
Insufficient explanation on why momentum is important to solve convergence problem of IDInit. It would be great to provide some theoretical insights to support this factor.

Author response:
This is a great question. We would like to give the interpretation from the angle of theory. According to a study on the convergence problem with identity initialization [8], when layers are initialized with the identity matrix, all the layers will be the same at each step until the end, which causes layers not to converge to a ground truth whose eigenvalues contain negative values.  To address this, we delve into the mechanics of the SGD optimizer in our detailed proof presented in Appendix F.2 of the uploaded revision PDF.

In the proof, we examine how SGD facilitates differentiation between layers. Specifically, SGD updates weights through a component expressed as:

 $$
-\eta X_2^TX_2X_1^TX_1 + \eta X_2^TX_2X_1^TY_1 - X_2^TY_2,
 $$

where $\eta$ represents the learning rate, and $\{X_1, Y_1\}$ and $\{X_2, Y_2\}$  are training pairs sampled from the same dataset $\mathcal{D}$. This update mechanism inherently encourages layers to become asymmetric and distinct from each other, thereby mitigating the issue of identical layers post-training.

By amplifying the divergence between layers, momentum plays a critical role in this context . It enhances the effect of SGD's layer differentiation mechanism, thereby significantly addressing the convergence problem. Momentum essentially accelerates the process of breaking the symmetry in layer weights, which is pivotal for effective convergence in networks initialized by IDInit.

We believe this theoretical insight, backed by the proof in Appendix F.2, clearly articulates the importance of SGD with momentum in solving the convergence challenge associated with IDInit. We will revise the manuscript to explicate this point more thoroughly to provide a better understanding of the interplay between momentum, SGD, and the layer weights in the context of IDInit.

Reviewer comment (Writing):
What is the meaning of zero down-sampling in Table 2, is this a special downsampling operation compared to standard downsampling (like avgpooling) in ResNet?

Author response:
Sorry for the unclear description of the term "zero down-sampling" in Table 2 of our manuscript. To clarify, this term refers specifically to the first downsampling setting (i) mentioned in our last response. This approach is commonly used for datasets like CIFAR-10 and involves the use of pooling (e.g., avgpool) to reduce the resolution of features, coupled with zero-padding to expand the channel size. In this case, ZerO performs poorly since affected by dead neurons.

To enhance the comparative analysis, we utilize ResNet-18 as the backbone, ensuring the avoidance of dead neurons for ZerO. Specifically, we implement 1×1 convolution for downsampling in ResNet-18, as detailed in \[6\], adhering to the downsampling setting (ii). This approach effectively avoids the dead neuron issue, thereby facilitating normal functioning of ZerO. Results are shown in STable 1 below.

|Model|Accuracy|
|---|---|
|ZerO|94.81 $\pm$ 0.05|
|IDInit|95.08 $\pm$ 0.13|

The results suggest that IDInit still outperforms ZerO of which the dead neurons are solved, which showcases the consistent robustness and effectiveness of IDInit across different downsampling strategies.  We will revise the description for better understanding.

Reviewer comment (Experiments):
It would be great to compare IDI, IDI with loose condition, and IDIZ together to show the effectiveness of IDIZ.

Author response:
We appreciate the reviewer’s suggestion to provide a comparative analysis of IDI, IDI with a loose condition, and IDIZ. In Section 4.4 of our manuscript, we initially discussed some of these components individually. Based on the reviewer’s feedback, we have expanded this analysis to further exhibit the interaction of these components when combined.

STable 2. Analysis of components.

|          |       Setting 1       |       Setting 2       |       Setting 3       |       Setting 4       |       Setting 5       |      Setting 6       |       Setting 7       |       Setting 8       |
|:--------:|:---------------------:|:---------------------:|:---------------------:|:---------------------:|:---------------------:|:--------------------:|:---------------------:|:---------------------:|
|  Loose   |                       |                       |                       |       &#10004;        |                       |       &#10004;       |       &#10004;        |       &#10004;        |
|   IDIC   |                       |                       |       &#10004;        |                       |       &#10004;        |                      |       &#10004;        |       &#10004;        |
|   IDIZ   |                       |       &#10004;        |                       |                       |       &#10004;        |       &#10004;       |                       |       &#10004;        |
| Accuracy | 86.12$_{±0.52}$ | 92.68$_{±0.08}$ | 89.47$_{±0.24}$ | 87.01$_{±0.29}$ | 92.95$_{±0.21}$ | 92.9$_{±0.18}$ | 90.43$_{±0.14}$ | 93.22$_{±0.05}$ |

The extended analysis is presented in the newly added STable 2. This table clearly illustrates that while the components 'Loose', 'IDIC', and 'IDIZ' each contribute to improvements independently, their combined application yields the most substantial results. Notably, IDIZ stands out by significantly enhancing accuracies to over 92%. This improvement is primarily attributed to IDIZ’s ability to resolve the dead neuron issue, which is a critical factor in optimizing neural network performance.

This collective analysis underscores the individual and combined efficacy of the IDInit techniques. The results firmly establish that each component plays a vital role in the overall performance, with IDIZ being particularly instrumental in achieving higher accuracy rates. Our manuscript will be updated to include this comprehensive comparative analysis, ensuring a clear demonstration of the effectiveness and interplay of IDI, IDI with a loose condition, and IDIZ.