Paper: S-TLLR: STDP-inspired Temporal Local Learning Rule for Spiking Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
The temporal locality property of this method is unconvincing. In Figure 1, my naive understanding is that it is possible to simply truncate both BPTT and STDP methods in the same way S-TLLR is truncated using equation (11). In other words, all methods can have temporal locality. The only way to truly claim that the proposed method does not scale with time, is by using both BPTT and STDP (and perhaps even other existing methods) with this truncation and see if S-TLLR learns faster or if other methods fail to learn the objective.

Author response:
We would like to address this question by stating that temporal locality means using only information available at the present. For example, in a sequence with multiple time steps, a temporal local operation at time step ($t$) must use only information available at that time step. By this definition, STDP expressed in equation (6) is a temporal local learning rule at any time step as it uses only information available at the present, and therefore its memory requirements are independent of the length of the sequence. In contrast, as we discuss in appendix C, BPTT does not present a temporal locality feature due to the backpropagation of errors trough time. This implies that the weight updates ($\Delta w$) at any time step depend on future information as shown in equation (23) and Figure 2a. Note that this happens due to the implicit recurrent connections (trough membrane potential accumulation) of hidden layers and not the loss function itself, so even if the loss for BPTT is computed as described in equation (11) and just for the last time step ($T_l=T$), propagation of errors is going to be similar to the one shown in Figure 2a with the small difference that the connections between the last layer and the loss function disappear, but still the weight updates at time step $t$ ($\Delta w [t]$) are going to depend on information at future time steps ($t+1, t+2, ..., T$). Therefore, BPTT is not a temporal local learning rule and requires the information of every single time step to compute the synaptic updates. In the case of S-TLLR, similar to STDP we can use trace variables to propagate the required information forward in time and use a learning signal obtained by propagation of errors trough the layers (not in time). Therefore, S-TLLR memory requirements are independent of the sequence length. An analysis on memory requirements is shown in Appendix C. Additionally, we would like to point out that equation (11) represent the instantaneous error signal obtained at the current time step ($t$), and the term $T_l$ only control at which point the targets are available and therefore the number of weight updates. So even if the learning signal is available for all time steps ($T_l=0$) the memory requirements of S-TLLR are the same.

Reviewer comment (Experiments):
The improvement from non-causal terms is highly confounded by the secondary activation functions in equations (14-17). Suggestions for fair experiments could be:  
- universally use the same secondary activation function across all tasks, or use all 4 activation functions for all tasks  
- apply the same activation functions to other methods

Author response:
In response to the ablation study suggestion, we conducted experiments, presented in Table 5 (Appendix E3), using three activation functions with varied $\alpha_{post}$ values due to resource constraints. The findings consistently demonstrate superior performance when using non-zero $\alpha_{post}$ compared to $\alpha_{post}=0$.

Reviewer comment (Presentation):
It is not clear how the method works in the recurrent neural network task. If causal recurrent gradients were incorporated in Figure 1, that would correspond to red lines being drawn from $u[t]$ to $y[t-1]$ (and others), which means most terms would have red and blue lines in parallel.

Author response:
We appreciate the feedback and have clarified that the primary text focuses on feed-forward networks, with discussions regarding recurrent models available in the supplementary material (Appendix D). This distinction aims to provide a comprehensive understanding while maintaining clarity within the main text.

Reviewer comment (Presentation):
Should blue terms in Figure 1 also extend beyond $t-2$ (with three dots) just like the red terms?

Author response:
Yes, the blue terms in Figure 1 should be extend beyond $t-2$. We have updated Figure 1 to ensure the discussion's generality.

Reviewer comment (Experiments):
Insufficient experiments: I understand that the event-based computer vision tasks are suitable for spiking neural networks, but I think the dataset reported in this paper is not comprehensive enough. In addition to the popular DVS-CIFAR10 and DVS-Gesture, N-CalTech101, and NCARs are also adopted in prior works [R1] as benchmarks. However, these results are missing in the paper.

Author response:
**Answer 1:** Thank you for your suggestion. In our revised version, we have incorporated the results for the N-Caltech101 dataset. Regrettably, integrating the Ncars dataset into our current framework wasn't feasible at this moment. Please, note that in our initial version, we already included evaluations beyond event-based vision classification tasks. Specifically, we incorporated assessments in event-based audio recognition and optical flow domains. These evaluations were deliberately included to explore longer temporal dependencies, such as in audio, and to tackle more intricate spatio-temporal regression through self-supervision, as observed in optical flow tasks.

Reviewer comment (Experiments):
Since the proposed method claims that the conventional BNTT is memory expensive, it is important to demonstrate the memory-accuracy comparison between the proposed method and BNTT (e.g., GPU Memory).

Author response:
**Answer 2:** We have taken the reviewers' feedback into careful consideration and have supplemented our paper with a more extensive analysis of the computational and memory-intensive nature of BPTT in Appendix C. Although the analysis primarily discusses fully connected models, it can be extrapolated to encompass convolutional models as well. Additionally, we have included a visual representation in Figure 3 within Appendix C4 to illustrate a simple example that delineates how GPU memory scales linearly with the number of time steps, contrasting with the consistent of S-TLLR memory consumption. These additions aim to provide a comprehensive understanding of the computational and memory complexities associated with our proposed framework.

Reviewer comment (Novelty):
Some recent papers and SoTA methods are not cited in this paper: 

[R2]: Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

[R3]: Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks, NeurIPS'21

[R4]: Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation, CVPR 2022.

Author response:
**Answer 3:** We sincerely appreciate the reviewers' valuable input regarding the additional papers. To ensure completeness and accuracy in our comparative analysis, we have incorporated the suggested papers in Table 3. It's essential to note that while these prior works focused on achieving high performance in image classification on static datasets, our emphasis lies in addressing temporal locality and memory-efficient training for Spiking Neural Networks (SNNs). The highlighted papers, while significant, do not encompass the pivotal aspects of our proposed approach, which emphasizes temporal dynamics and memory efficiency in SNNs

Reviewer comment (Theory):
The methodology section should be elaborated more. Based on Figure 1, S-TLLR introduces the incoming gradient $\partial L / \partial y$ on top of discrete STDP. What is the theoretical advantage (or intuition) of doing that?

Author response:
**Answer 4:** STDP, an unsupervised learning algorithm, operates by adjusting synaptic connections based on the relative timing of spiking activity but lacks a global supervisory signal to guide learning toward specific goals. So, we designed S-TLLR to incorporate a mechanism akin to STDP to identify neurons for update based on spiking activity. Then, we use a third factor—the learning signal—to attribute credit or assign blame to the neurons identified for update. This additional supervisory mechanism augments the synaptic plasticity process, enabling goal-oriented learning within the SNN framework.

Reviewer comment (Experiments):
The complexity was estimated, but the real energy consumption/efficiency haven't been calculated/tested.

Author response:
We have expanded our discussion on the number of multiply-accumulate (MAC) operations required by both S-TLLR and BPTT in Appendix C. While the estimation of dynamic energy consumption is typically proportional to the number of MAC operations, during training, the energy consumption associated with memory read and write operations might significantly contribute, particularly for memory-intensive methods like BPTT. Accurately modeling such energy consumption is a complex task beyond the current scope of our research. It's worth noting that S-TLLR was specifically designed as a three-factor learning rule, tailored for potential utilization on neuromorphic hardware like Loihi, which is an avenue we plan to explore in our future research.

Reviewer comment (Experiments):
While BPTT could work on much deeper SNNs, how about S-TLLR? Could it be extended to larger models/datasets?

Author response:
Primarily, our focus lies on event-based tasks, as we believe these tasks inherently benefit from utilizing SNNs due to their temporal information, unlike static image classification tasks. While our exploration into SNNs hasn't delved extensively into deeper architectures, this limitation is largely due to the absence of neuromorphic datasets comparable in size to more mainstream vision tasks such as CIFAR100 or ImageNet. However, it's important to note our experiments on N-Caltech101, which contains a similar number of classes as CIFAR100, demonstrating that S-TLLR performs comparably to BPTT. Additionally, our experiments on optical flow using MVSEC, a challenging spatiotemporal regression task, show that S-TLLR can achieve performance similar to BPTT. These results instill confidence that S-TLLR can indeed be extended to deeper models and a broader spectrum of tasks.

Reviewer comment (Evaluation):
Could the energy consumption/efficiency be calculated/tested.

Author response:
We have expanded our discussion on the number of multiply-accumulate (MAC) operations required by both S-TLLR and BPTT in Appendix C. While the estimation of dynamic energy consumption is typically proportional to the number of MAC operations, during training, the energy consumption associated with memory read and write operations might significantly contribute, particularly for memory-intensive methods like BPTT. Accurately modeling such energy consumption is a complex task beyond the current scope of our research. It's worth noting that S-TLLR was specifically designed as a three-factor learning rule, tailored for potential utilization on neuromorphic hardware like Loihi, which is an avenue we plan to explore in our future research.

Reviewer comment (Experiments):
While BPTT could work on much deeper SNNs, how about S-TLLR? Could it be extended to larger models/datasets, such as CIFAR100?

Author response:
Primarily, our focus lies on event-based tasks, as we believe these tasks inherently benefit from utilizing SNNs due to their temporal information, unlike static image classification tasks. While our exploration into SNNs hasn't delved extensively into deeper architectures, this limitation is largely due to the absence of neuromorphic datasets comparable in size to more mainstream vision tasks such as CIFAR100 or ImageNet. However, it's important to note our experiments on N-Caltech101, which contains a similar number of classes as CIFAR100, demonstrating that S-TLLR performs comparably to BPTT. Additionally, our experiments on optical flow using MVSEC, a challenging spatiotemporal regression task, show that S-TLLR can achieve performance similar to BPTT. These results instill confidence that S-TLLR can indeed be extended to deeper models and a broader spectrum of tasks.

Reviewer comment (Novelty):
OTTT has the same learning rule as S-TLLR except that additionally S-TLLR leverages non-causality and few other minor differences. But this non-causal term doesn’t help S-TLLR consistently based on Fig. 2.

Author response:
Thank you for pointing out the similarities with OTTT [1], which allows for a more in-depth discussion. It's essential to note that while both algorithms share certain aspects, their fundamental motivations differ significantly. OTTT strives to approximate BPTT and is derived from BPTT equations by detaching all recurrent connections from the auto-differentiation graph. In contrast, S-TLLR is based on the theory of three-factor learning rules, utilizing the STDP mechanism to compute eligibility traces and a learning signal derived from backpropagation through layers or random feedback connections (DFA).

Furthermore, the selection of STDP in S-TLLR stems from the observation that the exponential decay of the causal term in STDP is akin to the computation of gradients over time with a decay factor equal to the leak factor of spiking neurons. However, while STDP considers both causal and non-causal terms for synaptic plasticity, focusing solely on causal information might result in a loss of valuable learning information. Hence, S-TLLR proposes a three-factor learning rule that models an STDP-like mechanism for neuron eligibility.

Another significant distinction lies in the target applications; OTTT, as stated by its authors, targets static image classification, while S-TLLR is specifically designed for temporal dependency tasks, deemed more suitable for SNNs. Notably, S-TLLR encompasses a pool of learning rules defined by the selection of hyperparameters (STDP parameters), where OTTT represents a specific case with hyperparameters $\alpha_{post} = 0$ and $\lambda_{pre} = \gamma$.

In our evaluations (Table 2, 3, 4, 5), using non-causal terms consistently yields better performance than using only causal terms (S-TLLR with $\alpha_{post}=0$ equivalent to OTTT). Even though we attempted to replicate OTTT's results for DVS CIFAR10 using S-TLLR ($\alpha_{post}=0$) in Table 3, our batch size discrepancy—using 48 compared to [1]'s 128 due to GPU memory limitations—may have influenced the outcome. Our comparison across various datasets supports the superiority of S-TLLR using non-causal terms over OTTT.

Reviewer comment (Evaluation):
S-TLLR doesn’t outperform OTTT.

Author response:
As previously discussed, OTTT can be viewed as a specific instance of S-TLLR when $\alpha_{post}=0$ and $\lambda_{pre}=\gamma$ (the leak factor in LIF models). Consequently, the ablation studies comparing OTTT and S-TLLR are encompassed across the work in Table 2, 3, 4 and 5. Our results consistently demonstrate that S-TLLR outperforms OTTT (S-TLLR with $\alpha_{post}=0$) under same conditions.

Reviewer comment (Novelty):
Can the authors list all the differences between OTTT with S-TLLR methods?

Author response:
Thank you for pointing out the similarities with OTTT [1], which allows for a more in-depth discussion. It's essential to note that while both algorithms share certain aspects, their fundamental motivations differ significantly. OTTT strives to approximate BPTT and is derived from BPTT equations by detaching all recurrent connections from the auto-differentiation graph. In contrast, S-TLLR is based on the theory of three-factor learning rules, utilizing the STDP mechanism to compute eligibility traces and a learning signal derived from backpropagation through layers or random feedback connections (DFA).

Furthermore, the selection of STDP in S-TLLR stems from the observation that the exponential decay of the causal term in STDP is akin to the computation of gradients over time with a decay factor equal to the leak factor of spiking neurons. However, while STDP considers both causal and non-causal terms for synaptic plasticity, focusing solely on causal information might result in a loss of valuable learning information. Hence, S-TLLR proposes a three-factor learning rule that models an STDP-like mechanism for neuron eligibility.

Another significant distinction lies in the target applications; OTTT, as stated by its authors, targets static image classification, while S-TLLR is specifically designed for temporal dependency tasks, deemed more suitable for SNNs. Notably, S-TLLR encompasses a pool of learning rules defined by the selection of hyperparameters (STDP parameters), where OTTT represents a specific case with hyperparameters $\alpha_{post} = 0$ and $\lambda_{pre} = \gamma$.

In our evaluations (Table 2, 3, 4, 5), using non-causal terms consistently yields better performance than using only causal terms (S-TLLR with $\alpha_{post}=0$ equivalent to OTTT). Even though we attempted to replicate OTTT's results for DVS CIFAR10 using S-TLLR ($\alpha_{post}=0$) in Table 3, our batch size discrepancy—using 48 compared to [1]'s 128 due to GPU memory limitations—may have influenced the outcome. Our comparison across various datasets supports the superiority of S-TLLR using non-causal terms over OTTT.

Reviewer comment (Experiments):
In the paper, it is mentioned that OTTT applies learning rules at each forward pass, whereas S-TLLR enforces the learning rule at every fourth forward step. Could the authors test the performance if S-TLLR's learning rule was applied at each forward pass, similar to OTTT?

Author response:
To clarify, we did not enforce the learning rule every fourth forward step but during the last $T-T_l$ timesteps of the sequence. For N-CALTECH101, DVS Gesture and DVS CIFAR10, it applies to the last 5 time steps, and for Optical Flow, it occurs at the final timestep. Following the reviewer's suggestion, we conducted simulations for DVS CIFAR20, presenting the learning signal during every timestep of the sequence, as indicated in Table 3. Consistently, S-TLLR with non-causal terms outperforms OTTT (S-TLLR with $\alpha_{post} = 0$) for the same batch size 48.

Reviewer comment (Experiments):
Can the authors do ablation study taking OTTT model as a reference starting point? The study would systematically integrate modifications that transition the model towards the S-TLLR approach.

Author response:
As previously discussed, OTTT can be viewed as a specific instance of S-TLLR when $\alpha_{post}=0$ and $\lambda_{pre}=\gamma$ (the leak factor in LIF models). Consequently, the ablation studies comparing OTTT and S-TLLR are encompassed across the work in Table 2, 3, 4 and 5. Our results consistently demonstrate that S-TLLR outperforms OTTT (S-TLLR with $\alpha_{post}=0$) under same conditions.