Paper: Forward Gradient Training of Spiking Neural Networks (ICLR-2024)

Reviewer comment (Reproducibility):
The article claims that this method takes a step forward towards online learning on chips, but it does not delve into the implementation of FGT on a chip or the associated storage and computational costs.

Author response:
Currently, the development of neuromorphic hardware is immature and is also developing considering software-hardware co-design [1], so most algorithms are simulated on common computational devices while considering properties of neuromorphic computing, without experiments on real neuromorphic chips. The compatibility for implementation is measured by fitness to neuromorphic properties (or biological plausibility) because neuromorphic hardware will be designed following them for efficient event-driven and in-memory computation. FGT tackles BP’s problems of weight transport and separate phases for error signal propagation as it only requires unidirectional forward propagation across layers, conforming to properties of neuromorphic computing. We do not restrict the method to specific hardware, but follow the basic properties that will guide the development of hardware and algorithms. In Section 4.4, we also discussed some details on how the signal propagation of forward gradient can be made biologically plausible, proposing the variants of FGT (e.g., FGT (S), FGT (Q1, S)) which are also verified in experiments.

We have discussed storage and computational costs on potential neuromorphic computing hardware in Appendix E.1, and we further supplement a comparison table for clearer presentation in this revision. We quote the discussions and the table here: “*for memory costs, … On the other hand, if we consider neuromorphic computing with unidirectional synapses, BP (if possible) should maintain all backward synapses between successive layers, while FGT, FGT (S), and DFA keep direct feedback connections from the top layer, which can be much smaller than BP since middle layers usually have much more neurons. As shown in Table 5, the memory costs of DFA and FGT (S) can be much smaller than BP*”, “*for operation numbers, BP requires a backward propagation across $N$ layers, DFA requires direct feedback to $N$ layers, FGT requires $N$ forward propagation of forward gradients for different layers (across $N, N−1, ..., 1$ layers) and direct feedback to $N$ layers, and FGT (S) requires a forward propagation across average $N/2$ layers and direct feedback to $N$ layers which is more practical than FGT. Direct feedback to $N$ layers may need fewer operations than BP across $N$ layers as middle layers usually have more neurons (for example, propagation from 400 to 400 neurons requires much more operations than from 10 to 400 neurons). So with a rough estimation, DFA needs much fewer operations than BP, and FGT (S) needs fewer operations than BP, as shown in Table 5*”.

| Method | Memory | Operations |
| :----: | :----: | :----: |
| BP (if possible) | $O((N-1)n^2+mn)$ | $O((N-1)n^2+mn)$ |
| DFA | $O(Nmn)$ | $O(Nmn)$ |
| FGT (S) | $O(Nmn)$ | $O(\frac{1}{2}(N-1)n^2+mn+Nmn)$ |

The above table (Table 5 in Appendix E) is an estimation of training costs on potential neuromorphic hardware. For illustration, we consider neural networks with $N$ hidden layers with $n$ neurons for each layer and $m$ neurons for the output layer, where $m \ll n$. The costs mainly focus on additional synaptic costs besides the normal forward procedure. Our method FGT (S) is expected to be better than BP considering costs, and achieving much better performance than DFA.

Reviewer comment (Experiments):
The current article only validates the effectiveness of FGT on shallow and simpler network structures, showing comparable results to BPTT. However, its performance on deeper and more complex networks remains unexplored.

Author response:
First, in Appendix E.3 (Table 7), we provided results for a deeper 9-layer convolutional network and our method is still effective and has a large improvement over DFA, with comparable performance as BP. The results show the potential for deeper networks. At the same time, biological systems do not have that deep architecture, and previous works show that shallow ANNs with recurrence achieve higher functional fidelity of human brains and similarly high performance (Kubilius et al., 2019). Future work can consider advanced network architectures.

Second, to further study large-scale and complex settings, we supplement an experiment on finetuning a pretrained **ResNet-34 on ImageNet** under potential hardware noise. This task is on the ground that there can be hardware mismatch for deploying trained SNN models (Yang et al., 2022; [2]), and we may expect finetuning models directly on hardware to better deal with the problem. Our method paves paths for on-chip learning and may be combined with other works aiming at GPU training for SOTA performance in such scenario. We simulate a kind of potential hardware noise by considering the stochastic neuron setting as described in Section 3.1, which may correspond to some hardware noise such as thermal noise. We consider finetuning a pretrained ResNet-34 model released by Xiao et al. (2022), which is trained under the deterministic setting (i.e., no noise) with the original test accuracy as 65.15%, under the noise setting with different levels of noises. All methods are trained for only 1 epoch and the results are below (where the noise parameter corresponds to the hyperparameter for the sigmoid function of the logistic noise, the larger the parameter the larger the noise scale).

| Noise parameter | Direct test | DFA 1 epoch | FGT (S) 1 epoch | *BP 1 epoch* |
| :----: | :----: | :----: | :----: | :----: |
| 1/6 | 49.17 | 44.42 | **58.56** | *61.40* |
| 1/8 | 58.51 | 58.34 | **62.14** | *63.05* |

As shown in the results, our proposed FGT (S) can effectively finetune such large-scale model, while previous biologically plausible methods with random feedback (DFA) totally fail. BP is not biologically plausible and its results are only for reference. FGT (S) has lower performance than BP under the condition of 1 epoch because it may have a slower convergence rate than BP (as studied in Section 5). Our proof-of-concept results mainly verify the effectiveness of our method for such scenarios and large-scale models. Future work can consider more SOTA models.

Besides this setting, we may also expect potential on-chip training of pretrained models for new data or tasks considering real applications, as pretrain-finetune is the popular scenario considering recent advances in deep learning. These can be interesting future work.

Reviewer comment (Experiments):
The limited performance on smaller networks suggests that if this method fails to scale well to larger networks, the significance of this work may be limited.

Author response:
Our proof-of-concept results mainly verify the effectiveness of our method for large models like ResNet-34, while previous biologically plausible method totally fails. Usually, biologically more plausible algorithms do not reach BP efficiency considering convergence rate (e.g., the one epoch setting we consider). But as our brain also do not solely rely on supervised learning, it can be important future work to study better optimization objectives (e.g., local learning, self-supervision, etc.) that may be better combined with our method.

Reviewer comment (Theory):
The authors could discuss why the proposed framework provides a more feasible implementation of synchronous circuits.

Author response:
Our proposed method tackles BP’s problems of weight transport and separate phases for error signal propagation as it only requires unidirectional forward propagation across layers, conforming to properties of neuromorphic computing.

Reviewer comment (Experiments):
There are existing methods that estimate the FLOPS and energy cost of a given NN based on its memory and operation. The authors could provide such comparisons as well.

Author response:
In Appendix E.2 (Figure 4), we have provided the firing rate statistics of models trained by different methods, which is usually used for energy estimation since the energy is proportional to spike counts. As shown in the results, on DVS-CIFAR10, the model trained by FGT has more spikes than BP, while with local learning, the spike number is reduced and is smaller than BP. On CIFAR10, the model trained by FGT has similar spikes as BP, while with local learning, the spike number is largely reduced by about half, implying fewer energy costs.

Reviewer comment (Experiments):
It is necessary to provide the comparison with other training methods on CIFAR-100 as well as ImageNet.

Author response:
In Section 5 (Table 3 and Table 4), we have already included comparison results on CIFAR-100.

In the above response to the third point, we supplemented results on finetuning ResNet-34 on ImageNet under noises.

Reviewer comment (Novelty):
The main concern is the novelty of the proposed method. The key contribution will be the section 4.1, where the authors present forward surrogate gradient to make forward gradient applicable to SNN. However, forward gradient, auto differentiation, and local learning are still existing concepts in the prior works.

Author response:
Actually, our key contribution is **Section 4.2** where we propose a novel method for forward gradients, i.e., momentum feedback connections, while Section 4.1 is only our minor contribution that generalize the concept of forward gradient to the surrogate gradient condition of SNNs. The important problem of forward gradients in previous works (Silver et al., 2022; Baydin et al., 2022; Ren et al., 2023) is that the vanilla forward gradient is a projection to a random direction, so it suffers from large variance and performs poorly, which is also verified in our experiments. To tackle the problem, in Section 4.2, we propose the novel forward gradient with momentum feedback connections, which can largely reduce the variance and significantly improve the performance, also with theoretical analysis. With the proposed method, our FGT can achieve a similar performance as BP, which is **not** achieved by all previous works on forward gradients. This is one of our key contributions. Additionally, another novel contribution is presented in **Section 4.4**, where we also propose variants of our method with more biologically plausible considerations (e.g., FGT (S), FGT (Q1, S)) and we conduct experiments to verify their effectiveness, which has not been considered by previous works. Our method is novel and different from existing forward gradient methods.

Reviewer comment (Experiments):
Despite the repeated emphasis on hardware compatibility and importance of on-chip training, the relevant experimental results are not presented. Appendix E discusses the training costs, but FGT is still worse than BP.

Author response:
Currently, the development of neuromorphic hardware is immature, so most algorithms are simulated on common computational devices while considering properties of neuromorphic computing, without experiments on real neuromorphic chips. The compatibility is measured by fitness to neuromorphic properties (or biological plausibility) because neuromorphic hardware is designed following these properties for efficient event-driven and in-memory computation. Note that an important feature of neuromorphic computing is *unidirectional* synaptic connections for event-driven and *in-memory* computation to avoid frequent memory transport which accounts for the majority of energy costs in real hardware (the computation architecture is expected to be different from the commonly used hardware with von Neumann architecture such as CPU or GPU). This means that each synapse between two neurons only allows unidirectional communications, and its weight value is stored locally. This is the reason why BP can hardly be suitable for (future) neuromorphic hardware, because BP will require two different forward and backward connections between successive layers with exactly the same weight, which will need frequent memory exchange (and also two stages, etc.) that is not compatible to the basic thought of neuromorphic computing. Our proposed FGT (and variants), on the other hand, avoids BP’s problems and is thus more biologically plausible and compatible with neuromorphic hardware.

As for the discussion on training costs in Appendix E, Table 6 is only a rough comparison on GPU and we have also discussed conditions on potential neuromorphic computing hardware in the text, where our FGT (S) is better than BP (if possible) considering neuromorphic computing. We also supplement a comparison table (Table 5 in the revised version, also shown below) for clearer presentation of this. On GPU, the training costs of FGT (or FGT(S)) is (slightly) larger than BP because GPU does not follow neuromorphic architectures and we do not optimize low-level codes of our method as BP in the established PyTorch libraries (the results are only brief comparisons). It can also be further improved, e.g., with parallelization of computation for forward gradients. 

Meanwhile, our developed variant FGT (S) is more practical in real applications and also has about the same training costs and time as BP on GPUs, as shown in Table 6 in Appendix E.1.

Considering neuromorphic computing, FGT (S) can be better and we quote our discussion and comparison table in Appendix E.1 here: 
“*for memory costs, … On the other hand, if we consider neuromorphic computing with unidirectional synapses, BP (if possible) should maintain all backward synapses between successive layers, while FGT, FGT (S), and DFA keep direct feedback connections from the top layer, which can be much smaller than BP since middle layers usually have much more neurons. As shown in Table 5, the memory costs of DFA and FGT (S) can be much smaller than BP*”, “*for operation numbers, BP requires a backward propagation across $N$ layers, DFA requires direct feedback to $N$ layers, FGT requires $N$ forward propagation of forward gradients for different layers (across $N, N−1, ..., 1$ layers) and direct feedback to $N$ layers, and FGT (S) requires a forward propagation across average $N/2$ layers and direct feedback to $N$ layers which can be more practical than FGT. Direct feedback to $N$ layers may need fewer operations than BP across $N$ layers as middle layers usually have more neurons (for example, propagation from 400 to 400 neurons requires much more operations than from 10 to 400 neurons). So with a rough estimation, DFA needs much fewer operations than BP, and FGT (S) needs fewer operations than BP, as shown in Table 5*”.

| Method | Memory | Operations |
| :----: | :----: | :----: |
| BP (if possible) | $O((N-1)n^2+mn)$ | $O((N-1)n^2+mn)$ |
| DFA | $O(Nmn)$ | $O(Nmn)$ |
| FGT (S) | $O(Nmn)$ | $O(\frac{1}{2}(N-1)n^2+mn+Nmn)$ |

The above table (Table 5 in Appendix E.1) is an estimation of training costs on potential neuromorphic hardware. For illustration, we consider neural networks with $N$ hidden layers with $n$ neurons for each layer and $m$ neurons for the output layer, where $m \ll n$. The costs mainly focus on additional synaptic costs besides the normal forward procedure. Our method FGT (S) is expected to be better than BP considering costs, and achieving much better performance than DFA.

Additionally, we also would like to emphasize that our method mainly focuses on developing neuromorphic-friendly algorithms rather than only considering efficiency on GPU. We realize our targets, i.e., more neuromorphic properties and a similar level of performance as BP, with comparable costs, which could be acceptable.

Reviewer comment (Evaluation):
The higher training cost of FGT and LL makes sense to some extent as CPU/GPU are not SNN-friendly architectures. However, SNN accelerators are still being developed, and it is difficult say there is a standard architecture for neuromorphic computing. For example, if there are dedicated hardware accelerators for BP-based supervised SNN, BP will be more hardware compatible than FGT.

Author response:
While it is true that neuromorphic hardware is still being developed, BP is not a good choice because it is not compatible with the basic thought of neuromorphic computing and can have larger costs even if it is supported (e.g., frequent memory exchange, more synapses considering forward and backward between all successive layers, etc.), as discussed in the above response to the second point. Our FGT (and variants) is not specific to certain hardware, but follows the basic properties of neuromorphic computing that will guide the development of hardware, which may be more suitable. We believe that developing biologically plausible training methods is important for SNNs.

Reviewer comment (Theory):
It is not easy to make connections to well-known biological or biologically-inspired learning rules such as Hebbian learning or spike-time dependent plasticity (STDP). If the layer-wise error propagation is the main difference with BP, the proposed method is loosely related to biological learning rules.

Author response:
In Section 4.3, we have discussed the connection to three-factor Hebbian learning. Our method is based on previous online training methods of SNNs, which is connected to the three-factor rule, while our method further answers how the global modulator can be determined – in our method, it is directly propagated from the top layer, corresponding to a more plausible neuro-modulation than previous methods that should be backpropagated across layers.