Paper: Enhancing Neural Network Performance with Leader-Follower Architecture and Local Error Signals (ICLR-2024)

Reviewer comment (Theory):
Although the proposed method is claimed to be inspired from biological neural network, the rationale behind it is still unclear to me. I’m not an expert in biological neural network, but I assume any improvement of the new design should have theoretical-grounded explanation. Could the author of the paper provide such theoretical proof on the proposed method?

Author response:
We thank the reviewer for comprehensively assessing our manuscript and we are glad to see the reviewer has acknowledged our work as “very interesting” and “promising”. 

Our LFNN and LFNN-$\ell$ methods draw inspiration from collective behavior observed in biological neural networks (BNNs) found in nature, such as cells, birds, and human brains. BNNs showcase self-organizing local connections among nodes, enabling them to learn and process limited or noisy input data [1]. Unlike BNNs, artificial neural networks (ANN) employ a layered structure where all information is analyzed at the end (output layer), this design limitation arises because the layered structure cannot be parallelized, and components in the architecture cannot communicate with each other. Furthermore, in biological networks such as the human brain, synaptic weight updates can occur through local learning, independent of the activities of neurons in other brain regions [2, 3]. Local learning has been identified as effective means to reduce memory usage during training and to facilitate parallelism in deep learning architectures [4]. With local loss in our LFNN-$\ell$, we also observe speedup in the training process.

That’s said, our goal is to draw inspiration from complex collectives and integrate this concept into artificial neural networks (ANNs) by introducing local loss functions which aim to parallelize the computation process during training. We may be not able to provide formulas as a mathematical proof, but we posit that our local loss-based approaches (LFNN) share similarities with BNNs on a higher and more abstract level, contributing to the facilitation of learning and understanding of input data. As a compensation, we show extensive experimental results as practical proof of concept to demonstrate that LFNN-$\ell$ consistently outperformed both BP and other BP-free approaches. We hope this work can open up new directions in ANN design and provide inspiration to the broader NN community, and we hope this response addresses the comment from the reviewer.


[1] Henry Markram, Wulfram Gerstner, and Per Jesper Sjöström. A history of spike-timingdependent plasticity. Frontiers in synaptic neuroscience, 3:4, 2011. 

[2] Natalia Caporale and Yang Dan. Spike timing–dependent plasticity: a hebbian learning rule. Annu. Rev. Neurosci., 31:25–46, 2008

[3] Chenzhong Yin, Phoebe Imms, Mingxi Cheng, Anar Amgalan, Nahian F Chowdhury, Roy J Massett, Nikhil N Chaudhari, Xinghe Chen, Paul M Thompson, Paul Bogdan, et al. Anatomically interpretable deep learning of brain age captures domain-specific cognitive impairment. Proceedings of the National Academy of Sciences, 120(2):e2214634120, 2023

[4] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles , pp. 1–15, 2019.

Reviewer comment (Evaluation):
The proposed method can outperform other BP-free baselines on different benchmark datasets. However, the performance of proposed LFNN method seems not promising on complete ImageNet-1K, compared to BP method. It may suggest that the proposed method may not be effective in large scale datasets.

Author response:
We thank the reviewer for the correct observation that our “proposed method can outperform other BP-free baselines on different benchmark datasets”. Scaling to larger datasets is for sure a common challenge for BP-free algorithms [1, 2, 3, 4]. We would like to clarify that our BP-free method (LFNN-$\ell$) actually can outperform all BP-free and BP baselines on larger-scale problems if we increase the complexity of networks. As illustrated in Table 3 (c and d), when applying LFNN-$\ell$ on ResNet, LFNN-$\ell$ demonstrates results that are comparable to those achieved with the BP algorithm. Hence, the principal factor contributing to LFNN's relatively inferior performance shown in Table 1 compared to BP on larger-scale problems appears to be the small network size.

[1] Ren, Mengye, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. "Scaling forward gradient with local losses." arXiv preprint arXiv:2210.03310 (2022).

[2] Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1): 13276, Nov 2016. ISSN 2041-1723. doi: 10.1038/ncomms13276.

[3] Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In Advances in Neural Information Processing Systems 29, NeurIPS, 2016.

[4] Sergey Bartunov, Adam Santoro, Blake A. Richards, Luke Marris, Geoffrey E. Hinton, and Timothy P. Lillicrap. Assessing the scalability of biologically-motivated deep learning algorithms and architectures. In Advances in Neural Information Processing Systems 31, NeurIPS, 2018.

Reviewer comment (Evaluation):
The results in Fig.2a suggest that the models with approximately 50%-60% leaders would achieve lowest performance. Can authors explain why that happen?

Author response:
We appreciate the reviewer for bringing up this question. In Figure 2a, we present the test accuracy for LFNN on MNIST after one-pass of training, representing the output results after the first epoch.

Given that our LFNN draws inspiration from collective behaviors, and considering that in real-world conditions only 5%-10% of individuals are typically chosen as leaders [1], we formulate the following hypothesis to elucidate this phenomenon: Intuitively, the higher the information leadership size would result in better decision making in complex collectives in nature. However, artificial neural networks are very different from the collective behaviors observed in nature. For more challenging tasks (CIFAR-10, ImageNet), we do observe that higher leadership size tends to give higher accuracy (in Table 5 and 6). In this one-pass training of MNIST, we do not observe this pattern. Our hypothesis is that with few and considerable amounts of leadership, there is less randomness and noisy information during training. With around 50\% leadership, for this small neural network on a simple task with one-pass training, the observation could be caused by the randomness and competing information received by followers.

[1] Peter Friedl, Yael Hegerfeldt, and Miriam Tusch. Collective cell migration in morphogenesis and cancer. International Journal of Developmental Biology, 48(5-6):441–449, 2004.

Reviewer comment (Evaluation):
The proposed method requires 90% leadership on Tiny ImageNet and ImageNet subset and 100% leadership on complete ImageNet for both LFNN and LFNN-$\ell$. Does it mean that the demand for the leadership is increasing as the size of dataset grows? Would that increase the computation cost for the training?

Author response:
We agree with the reviewer's perspective that larger leadership sizes are needed for the increasing complexity of the dataset. Tables 5 and 6 provide a comprehensive view of the error rate results for both LFNN and LFNN-$\ell$ across MNIST, CIFAR-10, and ImageNet, with varying leadership sizes. Upon careful examination, it becomes evident that in both MNIST and CIFAR-10, different leadership sizes yield comparable error rates. However, when considering ImageNet, a notable trend emerges, where models with larger leadership sizes exhibit superior performance. This phenomenon can be attributed to the architecture's need for more information when analyzing complex datasets. This requirement necessitates a higher number of leader workers to effectively guide the training process.

In LFNN and LFNN-$\ell$, increasing leadership sizes results in a slight increase in the training computation cost. This is because leader workers in some cases necessitate output neurons to compute local loss, introducing a slight overhead in the number of parameters. However, a higher leadership size corresponds to enhanced performance, as evidenced in Table 5 and 6. Therefore, there exists a minor trade-off between computation cost and performance.

Reviewer comment (Evaluation):
Why does the test accuracy decline between the 40% and 60% leaderships in Figure 2a?

Author response:
In Figure 2a, we present the one-pass test accuracy for LFNN on MNIST, representing the output results of the first epoch. It is an interesting observation that test accuracy declines between 40\%-60\% leadership. Given that our LFNN draws inspiration from collective behaviors, and considering that in real-world conditions, only 5\%-10\% of individuals are typically chosen as leaders [1, 2]. We formulate the following potential hypothesis to elucidate this phenomenon: Intuitively, the higher the information leadership size would result in better decision making in complex collectives in nature. However, artificial neural networks are very different from the collective behaviors observed in nature. For challenging tasks, we do observe that higher leadership size tends to give higher accuracy (in Table 5 and 6). In this one-pass training of MNIST, we do not observe this pattern. Our hypothesis is that with few and considerable amounts of leadership, there is less randomness and noisy information during training. With around 50\% leadership, for this small neural network on a simple task with one-pass training, the sub-optimal performance could be caused by the randomness and competing information that followers receive from multiple leaders. We welcome and appreciate any suggestions or comments here.  

[1] Peter Friedl, et al. Collective cell migration in morphogenesis and cancer. International Journal of Developmental Biology  

[2] Tarcai, Norbert, et al. Patterns, transitions and the role of leaders in the collective dynamics of a simple robotic flock.

Reviewer comment (Evaluation):
In Figure 2b, what role does \(L_l^\delta\) serve in the comparison between \(L_g\) and \(L_g + L_l^\delta\)?

Author response:
Figure 2b displays the results of the ablation study involving various loss functions, namely those using global loss only, global loss with leader loss, global loss with follower loss, and global loss with both leader and follower loss. In this context, $\mathcal{L}_l^{\delta}$ denotes the local prediction loss for leaders, while $\mathcal{L}_g$ represents the global prediction loss. The comparison between $\mathcal{L}_g +\mathcal{L}_l^{\delta}$ and $\mathcal{L}_g$ aims to elucidate whether incorporating local prediction loss for leaders has an impact on the model's performance (refer to "Ablation study of loss terms" on page 5).

Reviewer comment (Theory):
In Figure 2b, despite the similarity between \(L_g + L_l^\delta\) and \(L_g + L_l^\delta + L_l^\hat{\delta}\) in the later stages of training, what is the specific function of \(L_l^\hat{\delta}\)?

Author response:
The $\mathcal{L}_l^{\bar\delta}$ represents the local error signal of followers which is denoted as $\mathcal{L}_l^{\bar\delta} = \mathcal{D}(a, b)$, where $D$ is the MSE loss function (refer to page 3 for more details).

Reviewer comment (Evaluation):
Insufficient discussion is provided regarding the experimental results.

Author response:
We would like to express our gratitude to the reviewer for the comment and we add more discussion regarding experimental results in our revised manuscript (in page 9).

Reviewer comment (Evaluation):
While LFNN outperforms BP and LG-BP on simpler datasets in Table 1, it lags behind them in terms of ImageNet results. What could be the reason behind such outcomes?

Author response:
It is essential to acknowledge that in the context of BP-free methods, the scalability to larger datasets, presents a notable challenge. In alignment with the previous approaches, although our LFNN also does not outperform BP on larger-scale problems with a small CNN architecture (see Table 1), LFNN-$\ell$ outperforms BP when we increase the complexity of our network. As illustrated in Table 3 (c-d) (applying LFNN-$\ell$ on ResNet), LFNN-$\ell$ shows results that are comparable to those achieved with the BP. Hence, the principal factor contributing to LFNN's relatively inferior performance compared to BP on larger-scale problems appears to be the network size, which is constrained by its relatively small scale. This phenomenon could be attributed to the fact that LFNN and LFNN-$\ell$ require more information provided by leaders when analyzing more complex datasets. In the case of a small CNN architecture, there might not be sufficient information gathered for leaders to effectively guide the training process. Hence, a more complex CNN will include more leaders, which can accumulate enough information to thoroughly analyze the complex data and contribute meaningfully to the training process.