Paper: Temporal Spiking Generative Adversarial Networks for Heading Direction Decoding (ICLR-2024)

Reviewer comment (Evaluation):
One of the interesting results of this paper is that different decoding architectures give different performances, which could shed light on the inherent complexity of the neural representation underlying the distributed representation of head orientation in the population of neurons studied, but this analysis is lacking.

Author response:
**W2:** "Inherent complexity of the neural representation."  **A:** Thank you for your valuable suggestion. The revised Figure 3 illustrates that the generated synthetic data exhibit distinct neural representations of head orientation when compared to the real collected data. Specifically, the figure demonstrates that the generated neural data predominantly captures the neuronal responses of specific neurons, while other neurons remain relatively inactive throughout the entire time window. This observation underscores the varying spatial dynamics within the population of neurons. Additionally, the generated neuronal responses, as indicated by spike firing frequency, manifest different intensities across various time points when contrasted with the real data. This discrepancy highlights the differences in temporal dynamics between the generated and real data.

Reviewer comment (Evaluation):
The statement "With the sparse spike-form Query, Key, and Value, its computation becomes more efficient" should be justified.

Author response:
**Q1**" Sparse spike-form Q, K, V is more efficient." **A:** Thank you for bringing up this point. In the traditional Transformer model, the computation involves dense matrices for Query, Key, and Value, wherein each element is a continuous float number, contributing to the dense computation. In contrast, the spike-based Transformer introduces a novel approach by utilizing a sparse spike form for the self-attention computation. This sparsity is strategically harnessed during the computation of Query, Key, and Value matrices, leading to a reduction in the total number of Multiply-and-Accumulate (MAC) calculations required during the attention computation [1, 2]. This optimization not only ensures a more efficient computation process but also proves particularly advantageous for spike-based models. As shown in the section D of APPENDIX in the revision about the energy consumtion estimation, the spike-based model consumes less energy than the dense none-spike computation. Therefore, exploiting the inherent sparsity of neural events allows us to enhance computational efficiency without compromising the overall performance of the model.  

[1] Yao, M., et al. Spike-driven Transformer. In Thirty-seventh Conference on NeurIPS 2023.
[2]Zhou, Z., et al. Spikformer: When Spiking Neural Network Meets Transformer. ICLR 2023.

Reviewer comment (Theory):
In the model, it is unclear why firing rates are used when the claim is that the representation is spiking.

Author response:
**Q2:**"Why claim spiking representation?" **A:** Thank you for your insightful question. The use of firing rates in our model, despite claiming a spiking representation, is a design choice driven by practical considerations. The real and generated spike firing data exhibit sparsity in the temporal dimension, making processing the data at each time point inefficient. By employing a relatively small time binning, the binned data also presents the spike form with 0 or 1 in each bin, as well as enhancing data generation efficiency. The firing rate representation, which reduces the length of the neuronal data, is also particularly beneficial for the Transformer-based structure, as demonstrated in Table 3. This approach addresses challenges associated with handling long-term dependencies and reduces computational complexity. The temporal dimension segmentation in our paper further supports these design considerations.

Reviewer comment (Presentation):
Figure 1 is too small, and the text contains a typographical error: "sythetic" should be "synthetic".

Author response:
**Q3:**"Figure 1"  **A:** The figure 1 has been modified to be clear and correct in the revision.

Reviewer comment (Evaluation):
In Figure 3, the description of the t-SNE analysis is vague and the results seem rather negative; this is hardly discussed in the "Visualisation of generated data" section and should be clarified.

Author response:
**Q4:**"The description of the t-SNE" **A:** Thanks for your question. Here the similarity reflects in the responses of some key neurons with high response intensity. The assessment of the generated synthetic data quality extends beyond mere similarity to the original data. Diversity in the generated data is a crucial factor, enhancing the model's generalization ability and facilitating the generation of novel instances that contribute to improved decoder training. In our paper, the visualization of generated data is discussed with a comprehensive consideration of both similarity and diversity, and the subsequent performance improvements in decoders are presented and analyzed (refer to Table 1 and 2).

Reviewer comment (Writing):
The syntax of the paper did not allow the reviewer to fully follow the arguments; although it was not taken into account in the evaluation, the authors should use a service, even an automatic one, to clarify certain points.

Author response:
**Q5**"Syntax and latex writting."  **A:** Thanks for your suggetion. We polished the whole manuscript and marked some key modified content red in the revision.

Reviewer comment (Evaluation):
It would be beneficial to discuss the limitations and potential biases associated with using synthetic data for training neural networks. Additionally, addressing the potential discrepancies between synthetic and real data would provide a more comprehensive understanding of the framework's applicability.

Author response:
**W1** "The limitations and potential biases associated with using synthetic data for training neural networks." **A:** Thank you for raising an important point. We acknowledge the limitations and potential biases associated with using synthetic data for training neural networks. While synthetic data, generated by models like T-SGAN, can enhance the diversity of training sets and contribute to improved generalization, it is essential to recognize that the synthetic data may not fully capture the complexities and nuances present in real-world datasets. Addressing potential discrepancies between synthetic and real data is crucial for a comprehensive understanding of the framework's applicability. We plan to conduct further investigations and analyses, comparing model performance on synthetic and real data, to provide a more nuanced discussion on the generalization capabilities and limitations of our proposed approach. Additionally, we will explicitly detail these considerations in the revised manuscript to ensure transparency and thorough evaluation of the framework.

Reviewer comment (Novelty):
Motivation for SNN-based approach unclear.

Author response:
**W1.1 and Q1:** "Motivation for SNN-based approach" **A:** The motivation behind employing Spiking Neural Networks (SNNs) in the proposed T-SGAN and recurrent SNN approach for neural spike data decoding lies in the unique characteristics of SNNs that align with the nature of spike-based neural signals. SNNs inherently capture temporal dynamics more effectively due to their event-driven nature, making them well-suited for processing spike sequences. This is crucial for decoding neural signals, which often involve intricate temporal patterns. Furthermore, SNNs are known for their energy-efficient computation, making them suitable for applications where power consumption is a concern, such as in neuromorphic hardware. This aligns with the goal of developing efficient models for processing neural spike data.

Reviewer comment (Novelty):
Motivation for GAN-based data augmentation unclear.

Author response:
**W1.1 and Q1:** "Motivation for GAN-based data augmentation." **A:** We appreciate the reviewer's inquiry regarding the motivation for integrating GAN-based data augmentation into our spike-data neural decoding framework. The rationale for employing GANs in this context is rooted in their unique capacity to address specific challenges associated with spike data. GANs are particularly suited for spike-data augmentation due to their ability to capture and reproduce intricate temporal patterns inherent in neural spike sequences. Spike data exhibits complex dynamics that are pivotal for decoding tasks, and GANs excel in capturing these intricate temporal dependencies. By leveraging the generative capabilities of GANs, our approach aims to enhance the diversity of synthetic spike sequences, ensuring that the generated data aligns with the authentic temporal intricacies observed in real neural recordings. Moreover, GANs are instrumental in mitigating the issue of data scarcity common in neural spike datasets. The generation of synthetic spike sequences allows us to substantially augment our dataset, providing the model with a more extensive set of examples to learn from. This is particularly advantageous in scenarios where obtaining large-scale labeled spike datasets is challenging. Additionally, the adversarial training paradigm of GANs enables the generation of realistic and contextually relevant spike patterns, contributing to a more effective and generalizable decoding model. The synthesized spike data not only serves as an augmentation strategy but also enriches the feature space, promoting the learning of discriminative features crucial for accurate decoding. In summary, the motivation behind incorporating GAN-based data augmentation lies in their ability to faithfully capture complex temporal patterns, address data scarcity issues, and provide a diverse and realistic synthetic dataset. These characteristics collectively contribute to the effectiveness and robustness of our spike-data neural decoding framework.

Reviewer comment (Experiments):
Simple baselines (e.g. linear model) missing.

Author response:
**Q3**"In terms of evaluation, the most straightforward baselines such as a (generalized) linear model are missing. "  **A:** Thank you for your valuable suggestion. We have supplemented a comprehensive evaluation by including baselines such as generalized linear models (GLM) and artificial neural networks in the comparison. The updated results, now presented in the following table (and Table 2 of the revised manuscript), demonstrate that our proposed ReSNN model outperforms the linear model (GLM) in terms of decoding accuracy. We believe this addition strengthens the evaluation and provides a more thorough understanding of the performance of our proposed model.  

| Models | Without synthetic data | With synthetic data from T-SGAN |  
|--------|-------------|----------|  
| GLM    | 50| **52.5** |  
| DNN    | 85 | **90** |  
| FCSNN  | 88.45 | **94.75** |  
| LSTM   | **87.25** | 86.1 |  
| ReSNN  | 93.55  | **95.2**  |

Reviewer comment (Novelty):
While the overall objective (decode heading direction) is somewhat understood, it is unclear why the authors use an SNN as opposed to standard neural networks or even simpler methods such as (generalized) linear decoding; the reasoning is not evident despite multiple reads of abstract, introduction, etc.

Author response:
**W1.1 and Q1:** "Motivation for SNN-based approach" **A:** The motivation behind employing Spiking Neural Networks (SNNs) in the proposed T-SGAN and recurrent SNN approach for neural spike data decoding lies in the unique characteristics of SNNs that align with the nature of spike-based neural signals. SNNs inherently capture temporal dynamics more effectively due to their event-driven nature, making them well-suited for processing spike sequences. This is crucial for decoding neural signals, which often involve intricate temporal patterns. Furthermore, SNNs are known for their energy-efficient computation, making them suitable for applications where power consumption is a concern, such as in neuromorphic hardware. This aligns with the goal of developing efficient models for processing neural spike data.

Reviewer comment (Experiments):
In terms of evaluation, the most straightforward baselines such as a (generalized) linear model are missing, making it unclear what any of the numbers in the paper mean.

Author response:
**Q3**"In terms of evaluation, the most straightforward baselines such as a (generalized) linear model are missing. "  **A:** Thank you for your valuable suggestion. We have supplemented a comprehensive evaluation by including baselines such as generalized linear models (GLM) and artificial neural networks in the comparison. The updated results, now presented in the following table (and Table 2 of the revised manuscript), demonstrate that our proposed ReSNN model outperforms the linear model (GLM) in terms of decoding accuracy. We believe this addition strengthens the evaluation and provides a more thorough understanding of the performance of our proposed model.  

| Models | Without synthetic data | With synthetic data from T-SGAN |  
|--------|-------------|----------|  
| GLM    | 50| **52.5** |  
| DNN    | 85 | **90** |  
| FCSNN  | 88.45 | **94.75** |  
| LSTM   | **87.25** | 86.1 |  
| ReSNN  | 93.55  | **95.2**  |

Reviewer comment (Evaluation):
The effect of the GAN is questionable, as the differences reported in Table 1 are small and there are no error bars reported.

Author response:
**Q4** "The effect of the GAN is questionable, as the differences reported in Table 1 are small and there are no error bars reported. " **A:** Thank you for your feedback. We have included error bars in Table 1 to provide a more comprehensive representation of the results. Regarding the effectiveness of T-SGAN, we conducted a  comparative analysis with TTS-GAN, a prominent time-series data generation model, under the settings with various classifiers, including GLM, deep neural networks, FCSNN, and ReSNN. Across most of these classifiers, the decoding model using synthetic data generated by T-SGAN consistently outperforms TTS-GAN in terms of accuracy. Notably, in the case of LSTM, the accuracy without generated data surpasses both GANs. This discrepancy suggests a potential incompatibility of LSTM with spike data, which further emphasizes the significance of designing the SNN-based T-SGAN model to improve the decoding performance for spike-based neural data.

Reviewer comment (Experiments):
Contribution of the augmentation (synthetical generation) vs. SNN decoding to increase in accuracy is unclear.

Author response:
**Q1 && W1**"How one would discern the contribution of the augmentation vs. the special decoder" **A:**  Thanks for your question. Firstly, the contribution of the augmentation with the proposed T-SGAN is evaluated in Table 2, by comparing the decoding performance of T-SGAN with TTS-GAN (one of the leading time-series data generating models) under the same decoders of FCSNN, LSTM, and ReSNN. The experimental results show that the decoding model with T-SGAN achives superior accuracy compared to TTS-GAN, affirming the effectiveness of the augmentation contribution. Secondly, we assess the contribution of the ReSNN decoder by comparing its decoding accuracy with other decoders, including fully-connected SNNs (FCSNN), linear models, and LSTMs, under the same data augmentation model. As shown in both Table 1 and Table 2, the proposed ReSNN decoder outperforms other decoders, providing evidence for the effectiveness of the proposed decoding architecture.

Reviewer comment (Evaluation):
Data visualization shows differences in synthetically generated spike trains data vs. real data. Also t-SNE embedding shows very different embedded points. These differences are not explained. On the contrary, spike trains are claimed to be similar.

Author response:
**Q2 && W3**"Data augmentation." **A:** Thanks for pointing out this. The contribution of the data augmentation could be measured from similarity and diversity [1]. Here the similarity reflects in the responses of some key neurons with high response intensity (figure 3 (a) and (b)). The assessment of the generated synthetic data quality extends beyond mere similarity to the original data. Diversity in the generated data is a crucial factor, enhancing the model's generalization ability and facilitating the generation of novel instances that contribute to improved decoder training. In our paper, the visualization of generated data is discussed with a comprehensive consideration of both similarity and diversity, and the subsequent performance improvements in decoders are presented and analyzed (refer to Table 1 and 2).

[1] Lucic, M., et al. Are GANs Created Equal? A Large-Scale Study. In Advances in Neural Information Processing Systems (NeurIPS), 2018.

Reviewer comment (Experiments):
There is no full description of experimental setup of data acquisition, how data was preprocessed, validated and whether this will part of this work.

Author response:
**W4**"The description of experimental setup of data acquisition, how data was preprocessed, validated and whether this will part of this work." **A:** We would explain more details about the data acquisition, however, the data collection is not the part of this work. In detail, three adult rhesus monkeys weighing 6-9kg were used. Surgically, MRI compatible T-bolts were implanted in the head to fix a headcap, through this headcap to immobilize the monkey relative to a customized chair during experiments. Scleral coils were implanted in eyes for tracking eye positions as described previously. The recording sites in VIP were precisely targeted by lower electrodes through predetermined holes in a recording grid fixed on top of the skull of monkeys. Only neurons with clear response modulations to the vestibular stimulus were further tested with the protocol described above. Data were collected from 210 neurons in six hemispheres of the three monkeys.