Paper: Epitopological learning and Cannistraci-Hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning (ICLR-2024)

Reviewer comment (Writing):
Simplifying the language could enhance accessibility to a wider audience.

Author response:
Because of the time limitation, we will continue to prepare a more simplified version of the language after this rebuttal. In addition, These are the actions that we implement or we intend to implement:
+ We already simplified the message in the title of the article, proposing a new title. 
+ A commented video that shows how ESML shapes and percolates the network structure across the epochs for the example of Figure 2A is provided at this link  https://shorturl.at/blGY1
+ we are planning to release in the final version of the article an Appendix with a glossary of the definition of the main new concepts such as: epitopological learning, network shape intelligence, Cannistraci-Hebb Network automata, meta-deep architecture, hyperbolic network topology, community network organization, etc… To this aim, we introduced in the new version of the Introduction section the following sentence: << To help the reader to get familiar with the concepts introduced in this article, we provide a glossary that summarizes their definitions in Appendix.>>

Reviewer comment (Evaluation):
Some results lack standard deviation reporting, raising questions about the reliability and robustness of those specific findings.

Author response:
Thanks. To address the reviewer’s concern, we added the standard deviation of all the experiments.

Reviewer comment (Presentation):
Addressing the legibility issue by enlarging text within figures would enhance the overall effectiveness of these visuals.

Author response:
Thanks, to address the reviewer’s concern, we revised the figure and enlarged text within figures in the newly uploaded version of the manuscripts.

Reviewer comment (Experiments):
The proposed method appears effective in scenarios where neural networks are overparameterized. Does it maintain its efficacy when applied to non-overparameterized neural networks characterized by limited depth and width? Particularly, how does it perform in relation to a large dataset?

Author response:
Thanks to the Reviewer for this insightful comment. The largest dataset adopted in the previous version of the article is ImageNet2012 which is now still the biggest and comprehensive dataset in the computer vision field. However, to address the Reviewer's concerns we considered larger and more difficult datasets such as TinyImageNet instead of CIFAR10 on VGG16, for a more direct comparison with a new considered CNN model that is GoogLeNet. In general, the results show that CHT gains an ultra-sparse advantage on both VGG16 and GoogLeNet when applied to TinyImageNet.

Finally, to investigate whether CHT maintains its efficacy when applied to non-overparameterized neural networks characterized by limited depth and width, we tested a scenario in which we applied CHT to CIFAR100 (for reason of time limitations we considered only this large dataset to attain results before rebuttal deadline) with ReseNet50 architecture whose intermediate layers depth was reduced from 4 to 2 (which is the minimum we can use).

The results of this test are provided in Appendix K in Table 2 and show that CHT performance is increased from around 78% to 80% of accuracy and the ultra-sparse network advantage is retained.

Reviewer comment (Experiments):
For the node structure, this paper only compares CHT with fully connected graph, the author should take other graph structure into consideration, like ER, BA, WS graph.

Author response:
We express our sincere gratitude for the reviewer’s insightful feedback. To address the reviewer’s concern, we have included a new section in Appendix L that delves into the selection of network initialization from a network science perspective. 
We clarify that the ER model initialization is the one already used in our study and this is explained now better in Section 3.1 Epitopological sparse meta-deep learning (ESML) and section 3.2 CH training strategy. Revisions are highlighted in red.

Furthermore, we have designed procedures to generate Watts–Strogatz (WS) model and the Barabási-Albert (BA) models on bipartite networks and we considered the following scenarios: static sparse training with BA, WS and ER (equivalent to WS for beta = 1); dynamic sparse training again with BA, WS and ER (equivalent to WS for beta = 1); CHT with initialization of the network using BA, WS and ER (equivalent to WS for beta = 1). We clarify that in our previous version of the study, we initialized our network employing Correlated Sparse Topological Initialization (CSTI) on layers that directly interact with input features from the dataset and apply Erdos-Renyi (ER) to intermediate layers. Now, thanks to the Reviewer suggestions, we could investigate to initialize the sparse network structure also by using the Watts–Strogatz (WS) model and the Barabási-Albert (BA) model.

The preliminary results that we gained on 3 datasets are consistent and show that in general the static sparse training via ER, BA or WS is outperformed by the respective dynamic sparse training that, in turn, is outperformed by CHT. In addition, in some cases, CHT initialized with BA and WS displays better performance than CHT initialized by ER or CSTI. We are expecting to obtain the results in the next few weeks on the other datasets and architectures, but we are quite confident that the trend obtained on the first 3 datasets is so consistent that might be confirmed also in the next datasets. 
We thank again the Reviewer for these valuable suggestions.

Reviewer comment (Evaluation):
Compared with SOTA models, the performance of CHT is not competitive.

Author response:
We thank the Reviewer because it allows us to discuss this point that certainly needs further clarification in the manuscript. To address the Reviewer's concern, we are adding this new text in the section on Discussion of Limitations and future challenges which is provided in Appendix C.

<< With respect to the dynamic sparse training SOTA (RigL and MEST), the computational evidence in Fig. 3 on 5 empirical tests obtained in different combinations of network architectures/datasets (VGG16 on TinyImageNet; GoogLeNet on CIFAR100 and TinyImageNet; ResNet50 on CIFAR100 and ImageNet), demonstrate that CHT offers a remarkable ultra-sparse (1% connectivity) advantage on the fully connected baseline, whereas the current SOTA cannot. For the sake of clarity, we acknowledge that in this study we do not compare with the specific SOTA in the respective data or architecture types, because they often include methodologies with tailored technical features that are network and data-dependent. Including these specific SOTA, would make it difficult a fair comparison across data and networks because it would largely depend on the way we adapt the ultra-sparse training strategy to the specific case.  The attempt of this study is instead to evaluate whether, and in which cases, CHT can help to achieve ultra-sparse advantage in comparison to the fully connected baseline, regardless of the specific network or data considered>>

Reviewer comment (Novelty):
What's the inspiration of current architecture?

Author response:
We are grateful for this comment that we address in the Introduction section of the revised manuscript by introducing this paragraph:

<< The inspiration behind this theory is that for many networks associated to complex systems such as the brain, the shape of their connectivity is learned during a training process which is the result of the dynamic evolution of the complex systems across the time (ref). This means that the evolution of the complex system carves the network structure forming typical features of network complexity such as clustering, small-worldness, power-lawness, hyperbolic topology and community organization (ref). In turn, given a network with a shape characterized by these recognizable features of complexity, a significant (better than random) part of its future connectivity can be predicted by local network automata which interpret the information learned in the engrams (memory traces) of the complex network topology (ref). Based on these notions, network shape intelligence is the intelligence displayed by any topological network automata to perform valid (significantly more than random) connectivity predictions without training, by only processing the input knowledge associated to the local topological network organization (ref). The network automaton training is not necessary because it performs predictions extracting information from the network topology, which can be regarded as an associative memory trained directly from the connectivity dynamics of the complex system (ref). >>

Reviewer comment (Presentation):
In Table 2 on page15. Why some accuracy result is not the best but still highlighted?

Author response:
We are sorry if this was not clear in the previous version of the manuscript. To address the reviewer’s concern, we added the below content in all the captions of the Results Table. 

<<We executed all the DST methods in an ultra-sparse scenario (1%). The best performance among the sparse methods is highlighted in bold, and values marked with “*” indicate they surpass those of the fully connected counterparts.>>

Reviewer comment (Presentation):
While I understand the space constraints, having something like Table1 or Table2 in the main text (maybe in reduced form) would be nice.

Author response:
Thanks for this value suggestion that we implemented by adding a reduced form of Table 1 directly below the Fig. 3, in order to simplify the reader in the visualizations and interpretations of the results.