Paper: A Graph is Worth 1-bit Spikes: When Graph Contrastive Learning Meets Spiking Neural Networks (ICLR-2024)

Reviewer comment (Presentation):
The detailed background does not provide in background section, and there are too many existing work descriptions in the method sections.

Author response:
We appreciate the reviewer for detailed comments and suggestive feedback. We want to clarify some misunderstandings that caused some of your concerns.



>  **W1, Q1, and Q2:** (1) Background section; (2) In Sec3, does the problem formulation special for this work, or it is general for all GCL application? Please make it clearly. (3) Author should formulate the GCL problem in background (instead of giving a brief introduction). Also, it is better to highlight which part is optimized by the proposed methods.

We respectfully disagree with the point on our presentation. The current manuscript is carefully polished, as also recognized by Reviewers n3GK, jihE, and HWYD with high scores 3, 4, and 3 on the **Presentation** part. We would like to address your main concerns w.r.t. the background part below:

+ The problem addressed in this work is specifically formulated for binarized graph contrastive learning, with a slight modification from the general formulation of GCL. In this work, the output representations of GCL are required to be binarized.

+ Given the prevalence of GCL in current research, we don't think it is necessary to put a large body of context to provide a detailed formulation of the GCL problem in the paper. We believe that providing a brief introduction to GCL is sufficient to establish the context for our work on binarized graph contrastive learning, as the readers in this field are likely to be familiar with the fundamental concepts and principles of GCL.
+ In SpikeGCL, our primary focus is on optimizing the encoder part of GCL using SNNs to achieve the binarized node representation.

Reviewer comment (Reproducibility):
Some design choices do not present.

Author response:
We believe there are some misunderstanding on our work. We want to clarify some misunderstandings that caused some of your concerns.

+ **Reason to use T encoders.** The use of $T$ encoders are well-motivated in response to the increasing scale of graph data. As explained in Sec. 4.1, previous approaches for handling sequential inputs from non-temporal graphs often involve repeating the graph $T$ times with augmentation strategies.  However, this increases the computational complexity of the encoder, resulting in $T$ times the forward pass, with each step having a d-dimensional input. To address this issue, SpikeGCL utilizes $T$ encoders applied to T groups of inputs, where each group has a $\frac{d}{T}$-dimensional input. In other words, the computational complexity is reduced to approximately $\frac{1}{T}$ compared to previous methods that repeat the GNN T times. This reduction significantly decreases the computational and memory overheads associated with applying SNNs on non-temporal graphs.

+ **Complexity of $T$ encoders in SpikeGCL.** The use of $T$ encoders in SpikeGCL **does not** introduce additional overheads because the learnable parameters of the T encoders, **except for the first layer**, are shared across layers. Moreover, the first layer of each encoder handles only $\frac{d}{T}$ features, allowing the complexity of SpikeGCL to remain certainly low.

Reviewer comment (Writing):
In Sec3, does the problem formulation special for this work, or it is general for all GCL application? Please make it clearly.

Author response:
We appreciate the reviewer for detailed comments and suggestive feedback. We want to clarify some misunderstandings that caused some of your concerns.



>  **W1, Q1, and Q2:** (1) Background section; (2) In Sec3, does the problem formulation special for this work, or it is general for all GCL application? Please make it clearly. (3) Author should formulate the GCL problem in background (instead of giving a brief introduction). Also, it is better to highlight which part is optimized by the proposed methods.

We respectfully disagree with the point on our presentation. The current manuscript is carefully polished, as also recognized by Reviewers n3GK, jihE, and HWYD with high scores 3, 4, and 3 on the **Presentation** part. We would like to address your main concerns w.r.t. the background part below:

+ The problem addressed in this work is specifically formulated for binarized graph contrastive learning, with a slight modification from the general formulation of GCL. In this work, the output representations of GCL are required to be binarized.

+ Given the prevalence of GCL in current research, we don't think it is necessary to put a large body of context to provide a detailed formulation of the GCL problem in the paper. We believe that providing a brief introduction to GCL is sufficient to establish the context for our work on binarized graph contrastive learning, as the readers in this field are likely to be familiar with the fundamental concepts and principles of GCL.
+ In SpikeGCL, our primary focus is on optimizing the encoder part of GCL using SNNs to achieve the binarized node representation.

Reviewer comment (Theory):
Why T encoders? Usually, neurons adopt the same weight among T time-steps, this design may increase the model size. Also, it is not clear how computation complex relates to time step size, since the author claim that they partition the feature dimension into T blocks. In my opinion, the computation complexity would not change when modifying T, which is not consist to Fig4.

Author response:
We believe there are some misunderstanding on our work. We want to clarify some misunderstandings that caused some of your concerns.

+ **Reason to use T encoders.** The use of $T$ encoders are well-motivated in response to the increasing scale of graph data. As explained in Sec. 4.1, previous approaches for handling sequential inputs from non-temporal graphs often involve repeating the graph $T$ times with augmentation strategies.  However, this increases the computational complexity of the encoder, resulting in $T$ times the forward pass, with each step having a d-dimensional input. To address this issue, SpikeGCL utilizes $T$ encoders applied to T groups of inputs, where each group has a $\frac{d}{T}$-dimensional input. In other words, the computational complexity is reduced to approximately $\frac{1}{T}$ compared to previous methods that repeat the GNN T times. This reduction significantly decreases the computational and memory overheads associated with applying SNNs on non-temporal graphs.

+ **Complexity of $T$ encoders in SpikeGCL.** The use of $T$ encoders in SpikeGCL **does not** introduce additional overheads because the learnable parameters of the T encoders, **except for the first layer**, are shared across layers. Moreover, the first layer of each encoder handles only $\frac{d}{T}$ features, allowing the complexity of SpikeGCL to remain certainly low.

Reviewer comment (Theory):
Fig3(b) ‘y1->y2’. Also, it is not clear how the entire backpropagation work. whether all y1…yn can directly receive gradient from the loss? Sec 4.4 introduces too much previous studies, it is better to clarify the gradient diagram in revision, i.e. for different blocks, where the gradient come from.

Author response:
We apologize for any confusion caused. The gradient of each block is derived from the contrastive loss computed on the output spikes at each time step. In other words, **we calculate the contrastive loss for the outputs of each time step and perform backward propagation for each individual block.** To prevent the gradients from flowing through the blocks, we utilize stop-gradient. For a clearer illustration, we kindly refer you to Algorithm 2 in Appendix C.

Reviewer comment (Evaluation):
How SPIKEGCL can reduce parameter size? Usually, SNN can reduce the activation size but keep parameter size unchanged. Author should provide a diagram of how to compute the parameter size.

Author response:
+ **Parameter-shared Encoders**. As discussed in Appendix D.1, the learnable parameters of the $T$ encoders are shared across layers **except the first layer** to meet the diverse input dimensions of group of features.  The matrix dimension of learnable weight of the $T$ encoders are: layer 1 ($T \times \frac{d_1}{T}\sim d_1$), layer 2 ($d_2$), ..., layer $L$ ($d_L$), which is equivalent to a simple GNN with $d_1$, $d_2$,..., $d_L$ dimensions.

+ **Many-to-one Decoder**. We introduce a simple single-layer MLP as our decoder rather than other complicated architectures. The decoder is peformed on the output spikes of $T$ encoders.
+ **Small Embedding Size**: As SpikeGCL can accumulate the output spikes across $T$ time steps, it is sufficient to use a small embedding size (e.g., 16) to achieve a decent performance.

In conclusion, the learnable parameters in SpikeGCL consist of a single GCN and a single MLP, both with small embedding sizes. Compared to other SNNs and GCLs that utilize complex encoders/decoders and large embedding sizes (e.g., 1024), SpikeGCL demonstrates much higher parameter efficiency.

Reviewer comment (Experiments):
How much does the result rely on detailed implementation of the SNN (such as reset to 0/reset by subtraction/IF or LIF)?

Author response:
Thank you for your feedback. We appreciate your constructive suggestion. In the submitted version, we have included the ablation results of different spiking neurons (IF, LIF, and PLIF) in Table 7, Appendix F. Additionally, following your suggestion, we have now added the ablation results on the reset mechanism for four datasets. The results are presented below:

|                               | Computers                         | Photo                             | CS                                | Physics                           |
| ----------------------------- | --------------------------------- | --------------------------------- | --------------------------------- | --------------------------------- |
| zero-reset + IF               | 88.0±0.2                   | 92.7±0.3                   | 91.2±0.1                   | 95.0±0.2                   |
| subtract-reset + IF           | 88.1±0.1                   | 92.9±0.2                   | 91.2±0.1                   | 95.0±0.1                   |
| zero-reset + LIF              | 88.5±0.1                   | 92.9±0.3                   | 92.0±0.2                   | 95.2±0.1                   |
| subtract-reset + LIF          | 88.6±0.1                   | 92.9±0.1                   | 92.1±0.1                   | **95.3±0.2**              |
| zero-reset + PLIF             | **89.0±0.3**               | 92.5±0.3                   | 92.1±0.1                   | 95.2±0.1                   |
| ours (subtract-reset + PLIF) | 88.9±0.3                   | **93.0±0.1**               | **92.8±0.1**               | 95.2±0.6                   |

Based on our findings, the performance does not necessarily depend on the specific implementation details of the SNN architectures. The performance gap between the best architecture and the worst is not significant. Even a simple IF neuron or a reset to 0 mechanism can achieve satisfactory performance.

Reviewer comment (Theory):
The author used the SNN method to compress the original representation. One problem is that SNN considers the accumulation in time and does not take into account the distribution characteristics in time. What I mean is, if the characteristics of time T-1 and time T-2 are exchanged, it seems that the value of time T will not be affected, but they will become two completely different vectors. In this way, will there be a many-to-one situation during the compression process?

Author response:
There seems to be a misunderstanding regarding the mechanism of SpikeGCL. While a strict temporal relationship between inputs is not necessary, the input order does matter for SpikeGCL because the SNN neuron determines whether to fire-and-reset based on the input at the current time step. Therefore, if the characteristics of time T-1 and time T-2 are exchanged, it will affect the value at time T.

Technically, SNNs are a many-to-many architecture, where each time step is associated with a fired output spike. Hence, there won't be a many-to-one situation during the compression process.

Reviewer comment (Evaluation):
I think the article lacks some quantitative analysis, such as what is the connection between the compressed binary vector and the original vector, what is the distribution of the conventional

Author response:
Thank you for the insightful suggestion. We have included the experiments regarding the similarity between the compressed binary vector and the original input vector in **Appendix F, page 23 of the revised manuscript.** The results show that each group of features exhibits a strong correlation with the corresponding output spikes while demonstrating minimal correlation with spikes in other time steps. This suggests that the learned binary representations are disentangled from each other, thereby providing improved expressiveness in representing the input features.

Reviewer comment (Experiments):
In Section 4.1, to reduce the complexity of SNNs by sampling from each node, the authors uniformly partition the node features into T groups, which is unreasonable. Features of different dimensions may represent different meanings, and operations after grouping these features may lead to inconsistencies in the feature space between different groups. On the contrary, the traditional mask method retains most features by randomly masking some features, ensuring the consistency of feature distribution. Therefore, in this section, the author can consider using random masks to reduce computational complexity while ensuring the consistency of data distribution.

Author response:
We believe there are some misunderstandings on our work. To clarify the situation, here we take a simple MLP (without bias terms) as the encoder. Given an input d-dimensional vector $X$, the output $y$ is defined as: $y= X W$, where $W \in \mathbb{R}^{d \times f}$ and $f$ is the output dimension. Typically, it is equivalent to:
$$
\begin{align}
y = y1 || y2 || ...||yT, \quad
y_t= X_t W_t
\end{align}
$$

Where $X_t$ is the $t$-th group of features with dimension $\frac{d}{T}$ and $W_t \in \mathbb{R}^{\frac{d}{T} \times f}$ is the learnable weight matrix of the $t$-th encoder. Each group of features is independently learned and this does not lead to any inconsistence in the feature space across different groups. 

We appreciate the reviewer‘s suggestion. While using random masks can be a valid approach to create a sequential input for SNNs, **it is not beneficial for reducing computational complexity.** In fact, it is worth noting that SpikingGCN has employed this technique to generate $T$ graphs, each with $d$-dimensional feature inputs. However, as discussed in Sec. 4.1, this method remains a significant bottleneck for SNNs when scaling to large graphs.

Reviewer comment (Experiments):
In Table 2, the authors compare the parameter size and energy consumption between the proposed method with traditional unsupervised/self-supervised methods. However, from Table 1, the performance of SpikeGCL is worse than the spike-based methods in most cases, there’s no evidence that SpikeGCL is better than other methods. The authors should add the comparison between SpikeGCL with spike-based methods in Table 2.

Author response:
First of all, we would like to point out that the comparision between SpikeGCL with spike-based mehtods has been presented in **Table 5 of Appendix F.** In Table 5,  SpikeGCL demonstrates an average energy consumption that is approximately 50x lower, while using only 1/3 of the parameters compared to other spike-based methods. 

Secondly,  it is important to note that SpikeGCL is an **unsupervised (U) and binarized (B)** method, which is expected to underperform **supervised and full-precision** spike-based baselines. We understand that there is a tendency in the deep learning community to place a strong emphasis on achieving state-of-the-art performance on benchmark datasets, often measured in terms of specific performance metrics. In our work, SpikeGCL, which is an energy and memory-efficient method, demonstrates only a minor performance decrease (less than 3%) and even surpasses most supervised and full-precision spike-based baselines in 4 out of 9 datasets, as indicated in Table 1 and Table 4.

While outperforming state-of-the-art methods can be an important and exciting goal for many researchers, it is also important to consider the broader context such as efficiency on the wider deep learning community. **SpikeGCL is not proposed as a new state-of-the-art SNN method. It is designed to achieve satisfied performance in low-power and resource-constrained settings, addressing the challenges posed by the increasing scale of real-world graph data.**

Reviewer comment (Novelty):
Generally speaking, combining Spiking and GCL is a good idea, but the novelty is not enough. Compared with traditional methods, SpikeGCL only groups features and then uses the traditional GCL method for learning, which does not present the special nature of contrastive learning in the scenario where spike and graph are combined.

Author response:
We respectfully disagree with the reviewer that the novelty of our paper is limited. Here we would like to reiterate the three research contributions of our work:

- **[Novelty 1]** The first-ever binary GCL method leveraging the sparse and energy-efficient characteristics of SNNs.
- **[Novelty 2]** The theoretical analysis about why SpikeGCL can approximiate the performance of full-precision GCLs.
- **[Novelty 3]** The block-wise training scheme on SNNs prevent them from suffering from the notorious problem of vanishing gradients with large time steps $T$.

Particularly, **Novelty 2 and 3 are built upon the special nature of SNNs and are proposed to solve the unique challenges in the context of GCL.** Our work digs deeper into the SNNs in GCL and offers more insights that would potentially benefit the whole community, which could motive more interesting research in this area. We believe this is an interesting and insightful work and hope you could reevaluate the contributions of our work. If there are any remaining concerns about the novelty of our work, please let us know. We are more than happy to provide additional information to help clarify the situation.