Paper: DISTA: DENOISING SPIKING TRANSFORMER WITH INTRINSIC PLASTICITY AND SPATIOTEMPORAL ATTENTION (ICLR-2024)

Reviewer comment (Novelty):
The main weakness of the paper is the novelty, the SpatioTemporal Attention has been proposed in prior work [1].

Author response:
We apologize for our lack of knowledge of the reference paper the reviewer provided during the development of our work.  We will reference it in the paper. Nevertheless,  we would like to highlight the key contributions of our paper. 
 
1. Two-level spatiotemporal attention based on neuron-level intrinsic plasticity and network-level spatiotemporal attention: DISTA uniquely integrates the network-level spatiotemporal attention with the intrinsic plasticity mechanism, allowing neurons within the spiking neural network to adaptively adjust their firing thresholds. This feature significantly enhances the model's ability to process and learn from spatiotemporal data, setting it apart from previous models that do not include such a mechanism. As the reviewer has pointed out, the spirit of our network-level spatiotemporal attention is similar to that of the reference paper [1]. Nevertheless, we want to present the difference between our approach and [1]. In [1], spatiotemporal attention is introduced when computing the attention map using keys and queries. Our approach, however, considers spatiotemporal attention when computing keys, queries, and values. As such, these two approaches are complementary. The benefit of the spatiotemporal attention scheme of [1] can be leveraged by our work, which may further improve accuracy. 

2. Denoising Approach: DISTA is distinctively designed with a denoising approach tailored for spiking neural networks. This aspect of the architecture aids in reducing the impact of spiking noise, thereby enhancing the robustness and performance of the model, especially in scenarios with varying data quality. In addition, we enable the attention map non-linearity by our proposed denoising method, in which we truncate the values that are smaller than the noise threshold to zero. Furthermore, denoising significantly reduces hardware overhead in the attention layers as shown in the general responses.

Reviewer comment (Experiments):
The imagenet dataset is widely used in the SNN field, but the work misses it.

Author response:
We conducted an experiment on ImageNet-100 and presented the results in the general responses at the very beginning.

Reviewer comment (Experiments):
Other results in the SPIKFORMER should be compared in the work.

Author response:
We added a detailed hardware energy analysis  in the general responses at the very beginning.

Reviewer comment (Novelty):
Novelty: I may have missed something, but most of the approach is taken from the spikeformer paper. This work mainly taken the temporal attention window which results in an increase in number of parameters and that may be the reason why the authors get accuracy improvement. I am not sure if the temporal attention has anything to do with it. If yes, it ll be good if the authors can elaborate. Further, there is no result on imagenet dataset. I feel the reason why authors didn't do it was the temporal attention ll explode the overall parameter requirement which becomes unmanageable in a large dataset. On the other hand, the original spikeformer work has imagenet results. Kindly clarify the scalability of your approach or discuss its limitations.

Author response:
We conducted an experiment on ImageNet-100 and presented the results in the general responses.

Reviewer comment (Experiments):
Overfitting Risk: There's a potential risk of overfitting as the model complexity increases with multiple attention mechanisms, though this is not explicitly discussed in the paper. Can the authors provide their opinion on this?

Author response:
Although the extra parameters we added in DISTA brought the risk of overfitting, it did not actually happen. Theoretically, neuron-level spatiotemporal attention enhances the nonlinear processing capabilities of neurons, network-level spatiotemporal attention provides short-term memory of past spiking information, and spatiotemporal attention denoising provides noise reduction capabilities as an alternative to softmax. Experimentally, the decent test accuracy and learning curve also proved that overfitting did not occur.

Reviewer comment (Evaluation):
Can the authors do some off the shelf energy estimations maybe by using some spiking hardware simulator tools like [1] (its understandable if the authors are not able to do it fully, a qualitative discussion on the efficiency benefits - as in whether they expect sparsity advantage at compute level or memory storage reduction from a hardware point of view will be useful)? I will be interested to see if their DISTA is actually energy efficient (specifically in terms of memory energy).  

[1] https://github.com/RuokaiYin/SATA_Sim

Author response:
We provided a detailed hardware energy analysis in the general responses at the very beginning.

Reviewer comment (Experiments):
Spiking networks by nature use less energy to function since they take advantage of the temporal coding. The authors have discussed this and it is one of the core reasons the community is interested in this; however, it would be interesting to perform a comparative analysis on hardware to test this approach. The authors have only discussed the accuracy of the approach.

Author response:
A1: We provided a detailed hardware energy analysis in the general responses at the very beginning.

Reviewer comment (Evaluation):
There should also be a comparative analysis of the space complexity of the approach, as this aspect has not been fully addressed.

Author response:
A2: We provided a detailed space complexity analysis in the general responses.

Reviewer comment (Experiments):
To get a better idea of the significance of the approach, a comparative analysis on large benchmarks such as ImageNet would also be interesting to observe.

Author response:
A3: We conducted an experiment on ImageNet-100 and presented the results in the general responses.

Reviewer comment (Evaluation):
The paper lacks an important comparative analysis of energy-efficiency, which should be considered as a measure of performance since it is one of the most crucial properties of spiking neural networks (SNNs).

Author response:
A1: We provided a detailed hardware energy analysis in the general responses at the very beginning.