Paper: Rethinking Spectral Graph Neural Networks with Spatially Adaptive Filtering (ICLR-2024)

Reviewer comment (Theory):
In my opinion, the most salient limitation of this work is the necessity to compute spectral decomposition. Although there have been many acceleration methods to conquer the O($N^3$) complexity (as the authors have also mentioned), such complexity will make SFA inapplicable to even moderate-sized graphs such as ogbn-products. Besides, there have been many more theoretical yet less practical graph filtering methods, where what limits their applications is mainly the necessity of spectral decomposition.

Author response:
> Q.1  Limitations regarding spectral decomposition in this work.

Thank you for your feedback on our use of spectral decomposition. We have addressed this shared concern in our "global" response.


We thank you for going through our rebuttal and acknowledging the strengths of our work by raising the contribution score. We are happy that our responses have addressed your previous concerns. While we are encouraged by your positive feedback, we would like to seek some clarification regarding your decision to keep the current overall rating, particularly in light of your recognization that our usage of spectral decomposition is not a significant issue for a research paper.

In response to the perspective you kindly shared:

> This is not a severe issue for a research paper in my opinion, but there would be a lot of interesting design/variants for spectral GNNs if such a computation is allowed.

We would like to offer some additional insights:

**(1)** We fully agree with your opinion that allowing spectral decomposition could open avenues for interesting designs and variants in spectral GNNs. Indeed, our paper already references over a dozen  works published in ICLR, NeurIPS, and ICML that have advanced GNNs, both theoretically and practically, by employing explicit eigendecomposition (see Appendix C.3). 

**(2)** In particular, we discuss two of these works (FE-GNN and Specformer) using spectral decomposition, showcasing their relevance to our study, respectively in the “Unified Viewpoints for GNNs” and “Long-range Dependencies” subsections of Section 2. We also include these two models in our empirical evaluations (see Tables 1 and 2).

**(3)** We mainly compare our SAF with SoTA polynomial spectral GNNs that avoid decomposition operations (such as GPR-GNN, BernNet, ChebNetII, JacobiConv, LON-GNN, and OptBasisGNN). Our empirical findings demonstrate that SAF outperforms these models across multiple datasets, combined with enhanced interpretability, suggesting the computational requirement of spectral decomposition are well-compensated.

**(4)** As for the computational aspects, spectral decomposition in our approach is a one-time computation per graph, which can be efficiently reused. Its preprocessing cost is often lower than the forward-pass cost in model training, as confirmed by our experiments in the global response. For larger graphs, while the decomposition process may be slower than training some local GNN models, it remains competitively efficient compared to global GNN models, both in terms of time and space overheads.

We sincerely hope that these points address your concerns. Your insights are invaluable to us, and we remain open and eager for any further discussion or feedback you might have.

Reviewer comment (Experiments):
I am a little confused with the so-called supervised setting. It is obvious that the split ratio is different from that of a semi-supervised setting. However, it seems that the node features of test nodes are also accessible to the compared methods during the training stage, which is the same as the semi-supervised setting.

Author response:
> Q.2 I am a little confused with the so-called supervised setting.

Thank you for your inquiry about the "supervised setting" in our experiments. 

**(1)** In line with established experimental protocols in transductive node classification, as referenced in prior works [1,2,3], our approach utilizes all node features (train, validation, and test) while only incorporating training node labels during the training phase. This method is a standard practice in the field, aimed at evaluating GNNs in node-level tasks. 

**(2)** To further evaluate models comprehensively under varying levels of supervision, we implemented two distinct data splitting strategies: a sparse splitting (2.5%/2.5%/95%) and a dense splitting (60%/20%/20%) for the training, validation, and testing sets, respectively. Consistent with the terminology used in [3], we refer to these tasks as "semi-supervised" and "full-supervised" node classification. This distinction is based on the degree of supervision each setting entails.

[1] Chien, Eli, et al. "Adaptive universal generalized pagerank graph neural network." ICLR, 2021.

[2] He, Mingguo, et al. "Bernnet: Learning arbitrary graph spectral filters via bernstein approximation." NeurIPS, 2021.

[3] He, Mingguo, et al. "Convolutional neural networks on graphs with chebyshev approximation, revisited." NeurIPS, 2022.

Reviewer comment (Presentation):
Could you explain figure 1 for me? I cannot figure out what the x-axis means as well as how these figures imply your conclusion.

Author response:
> Q.4 Could you explain figure 1 for me?

Thanks for your question regarding Figure 1 in our paper. We are pleased to provide further clarification on its content and how it supports our conclusions.

**(1)** In Figure 1, the x-axis represents the geodesic distance between node pairs in the original graph. This distance is calculated as the number of hops in the shortest path between the vertices. Concurrently, the y-axis quantifies the frequency of node pairs (of various types) being connected in the new graph.

**(2)** Using the Minesweeper dataset in Figure 1, where the longest geodesic distance is 99, as an example, we observe that node pairs from the original graph, regardless of their distance, are often directly connected in the new graph (indicated by a frequency greater than 0). This finding is crucial as it demonstrates the capability of the new graph to establish direct links among nodes that originally require multiple hops to do so.

**(3)** To enhance clarity, we have revised the caption of Figure 1 as "Distributions of adjacent nodes in the new graph based on their geodesic distance in the original graph. It is seen that nodes, distant in the original graph, can be connected in the new graph.".

Reviewer comment (Experiments):
SAF is only compatible with BernNet due to its assumption of non-negative filtering. As a result, it raises questions about whether the observed good performance is attributable to BernNet or SAF?

Author response:
Thank you for your thoughtful feedback regarding the compatibility of our SAF framework with BernNet. We appreciate the opportunity to clarify and expand on this important aspect of our work.

**(1)** As you rightly pointed out, SAF indeed requires non-negative graph filters, which we discuss in Section 5 of our paper. However, we would like to highlight that SAF's compatibility extends beyond BernNet. **Theoretically, SAF is also applicable to other models like ChebNetII [1], a recent Spectral GNN variant based on Chebyshev interpolation, by reparameterizing its learnable parameters to be non-negative.** To substantiate this theoretical proposition, we have conducted additional experiments with ChebNetII, incorporating our SAF framework. These experiments yielded encouraging results as shown in the following table, proving the potential of SAF in enhancing models other than BernNet. 

Full-supervised node classification accuracies (\%)
|              |   Cham.            |   Squi.            |   Texas            |   Corn.            |   Actor            |   Cora             |   Cite.            |   Pubm.            |
|--------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
|   ChebNetII  |   71.37$\pm$1.01   |   57.72$\pm$0.59   |   93.28$\pm$1.47   |   92.30$\pm$1.48   |   41.75$\pm$1.07   |   88.71$\pm$0.93   |   80.53$\pm$0.79   |   88.93$\pm$0.29   |
|   SAF-Cheb   |   74.97$\pm$0.66   |   64.06$\pm$0.59   |   94.43$\pm$1.81   |   92.62$\pm$2.13   |   42.65$\pm$1.01   |   89.56$\pm$0.64   |   80.68$\pm$0.68   |   91.27$\pm$0.34   |
|   Improv.    |   3.60             |   6.34             |   1.15             |   0.32             |   0.90             |   0.85             |   0.15             |   2.34             |

**(2)** While SAF, as detailed in our paper, does build upon the foundational structure of BernNet, **its distinct feature is the inclusion of non-local aggregation, aimed at capturing long-range dependencies and addressing graph heterophily.** This aspect is crucial for advanced graph learning and is not a native feature of BernNet. Our experimental results, as detailed in Tables 1, 2, and 4, clearly show that the enhancements brought by SAF are distinct and significant, with performance improvements of up to 15.37\%. These results underscore the unique contributions of SAF, reaffirming its significance beyond the inherent capabilities of BernNet.

[1] He, Mingguo, et al. "Convolutional neural networks on graphs with chebyshev approximation, revisited." NeurIPS, 2022.

Reviewer comment (Theory):
My main concern is the use of explicit eigendecomposition, which I believe is unacceptable in spectral GNNs.

Author response:
> Q.1  Computational Concerns about the use of explicit eigndecomposition.

Thank you for raising this important question. We acknowledge that this query was shared among multiple reviewers. To provide a comprehensive response, we have elaborated on this topic in our "global" response section.

Reviewer comment (Theory):
The time and space complexity of eigendecomposition for an $n$-dimensional square matrix is $O(n^3)$ and $O(n^2)$, respectively, severely limiting scalability in practice. Even as a preprocessing step, it becomes challenging to execute on graphs with millions of nodes and edges.

Author response:
> Q.1  Computational Concerns about the use of explicit eigndecomposition.

Thank you for raising this important question. We acknowledge that this query was shared among multiple reviewers. To provide a comprehensive response, we have elaborated on this topic in our "global" response section.

Reviewer comment (Novelty):
The primary advantage of spectral GNNs based on polynomials is their ability to avoid the need for eigendecomposition. If eigenvectors were readily available, the design of spectral GNNs would become trivial, rendering polynomial-based approaches unnecessary.

Author response:
> Q.1  Computational Concerns about the use of explicit eigndecomposition.

Thank you for raising this important question. We acknowledge that this query was shared among multiple reviewers. To provide a comprehensive response, we have elaborated on this topic in our "global" response section.

Reviewer comment (Novelty):
While certain works, such as Specformer, have used eigendecomposition, I believe their practical significance is quite limited.

Author response:
> Q.1  Computational Concerns about the use of explicit eigndecomposition.

Thank you for raising this important question. We acknowledge that this query was shared among multiple reviewers. To provide a comprehensive response, we have elaborated on this topic in our "global" response section.

Reviewer comment (Novelty):
Opting for the Bernstein basis as the backbone may not represent the optimal choice. On the one hand, ChebNetII can also guarantee filter non-negativity, and on the other hand, Chebyshev interpolation boasts lower complexity in comparison to Bernstein approximation.

Author response:
> Q.2  Opting for the Bernstein basis as the backbone may not represent the optimal choice. On the one hand, ChebNetII can also guarantee filter non-negativity, and its Chebyshev interpolation boasts lower complexity in comparison to Bernstein approximation.

Thank you for your insightful comments regarding our choice of the Bernstein basis in our SAF framework. Your comment has prompted a valuable extension of our analysis, which can be found in the Appendix G of our revised manuscript.

**(1) Experimenting with ChebNetII as Base Model:** We sincerely apologize for the initial omission in not including experiments with ChebNetII as a base model for SAF. We have now conducted additional experiments with ChebNetII as a base model for SAF (SAF-Cheb), alongside our original implementation using BernNet (SAF-Bern). The results, which we present in the subsequent table, illustrate that SAF indeed enhances ChebNetII’s performance.

Full-supervised node classification accuracies (\%)
|              |   Cham.            |   Squi.            |   Texas            |   Corn.            |   Actor            |   Cora             |   Cite.            |   Pubm.            |
|--------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
|   BernNet    |   68.53$\pm$1.68   |   51.39$\pm$0.92   |   92.62$\pm$1.37   |   92.13$\pm$1.64   |   41.71$\pm$1.12   |   88.51$\pm$0.92   |   80.08$\pm$0.75   |   88.51$\pm$0.39   |
|   SAF-Bern   |   75.30$\pm$0.96   |   63.63$\pm$0.81   |   94.10$\pm$1.48   |   92.95$\pm$1.97   |   42.93$\pm$0.79   |   89.80$\pm$0.69   |   80.61$\pm$0.81   |   91.49$\pm$0.29   |
|   Improv.    |   6.77             |   12.24            |   1.48             |   0.82             |   1.22             |   1.29             |   0.53             |   2.98             |
|   ChebNetII  |   71.37$\pm$1.01   |   57.72$\pm$0.59   |   93.28$\pm$1.47   |   92.30$\pm$1.48   |   41.75$\pm$1.07   |   88.71$\pm$0.93   |   80.53$\pm$0.79   |   88.93$\pm$0.29   |
|   SAF-Cheb   |   74.97$\pm$0.66   |   64.06$\pm$0.59   |   94.43$\pm$1.81   |   92.62$\pm$2.13   |   42.65$\pm$1.01   |   89.56$\pm$0.64   |   80.68$\pm$0.68   |   91.27$\pm$0.34   |
|   Improv.    |   3.60             |   6.34             |   1.15             |   0.32             |   0.90             |   0.85             |   0.15             |   2.34             |

**(2) Performance Comparison:** Interestingly, SAF-Bern slightly outperforms SAF-Cheb in most datasets, and the improvement margin with SAF-Bern is notably larger. This may be due to the $g_ \phi(\lambda) \leq 1$ constraint in our SAF (see Section 5), where filter functions need to be rescaled by their maximum values on the graph spectrum. For Bernstein polynomials, this maximum is simply the maximum polynomial coefficient ($\max \{\phi_k\}_ {k=0}^K$), as per our Proposition 3. In contrast, the best theoretical upper bound for Chebyshev polynomials is $\sum_ {k=0}^K |\phi_ k|$, which is less precise. This distinction potentially affects the new graph's construction, thereby limiting model performance. We plan to address this issue in future work.

**(3) Empirical Complexity Studies:** We acknowledge that Chebyshev interpolation boasts greater efficiency over Bernstein approximation in theory. However, our empirical studies indicate that BernNet and SAF-Bern generally demand less training time compared to ChebNetII and SAF-Cheb, as evidenced in the table below. This could be due to ChebNetII requiring additional computations for the values w.r.t. $(K+1)^2$ Chebyshev nodes.

Average total running time (s).
|               |   Cham.  |   Squi.  |   Texas  |   Corn.  |   Actor  |   Cora   |   Cite.  |   Pubm.  |
|---------------|----------|----------|----------|----------|----------|----------|----------|----------|
|   BernNet     |   8.36   |   13.74  |   3.92   |   4.16   |   4.88   |   5.24   |   5.52   |   6.06   |
|   ChebNetII   |   22.82  |   30.73  |   11.47  |   9.64   |   14.88  |   19.96  |   16.14  |   36.91  |
|   SAF-Bern    |   11.55  |   18.78  |   4.38   |   4.70   |   5.36   |   6.04   |   6.12   |   18.43  |
|   SAF-Cheb    |   19.29  |   19.87  |   8.31   |   7.97   |   13.27  |   10.65  |   19.22  |   39.94  |

Reviewer comment (Experiments):
My main concern is the practicality of explicit eigendecomposition, and I would welcome a discussion between the author and other reviewers on this issue.

Author response:
> Q.1  Computational Concerns about the use of explicit eigndecomposition.

Thank you for raising this important question. We acknowledge that this query was shared among multiple reviewers. To provide a comprehensive response, we have elaborated on this topic in our "global" response section.

Reviewer comment (Experiments):
The ablation study should exclude each component.

Author response:
> Q.3 The ablation study should exclude each component.

Thank you for your valuable suggestion to augment our ablation analysis. Your guidance has been instrumental in enhancing the comprehensiveness of our study, and we have added detailed analysis in the revision (see Appendix F.3).

**(1)** Our SAF framework primarily comprises three modules: "Non-negative Spectral Filtering" (Spec.), "Non-local Spatial Aggregation" (Spat.), and "Node-wise Prediction Amalgamation" (Amal.). We have already tested our SAF without the Spec. module in the "Ablation Study" subsection of Section 6. Additionally, the configuration of SAF w/o Spat. is effectively equivalent to the BernNet model, of which the performance can be observed in Tables 1 and 2.

**(2)** Following your kind suggestion, we have now included experiments that ablate the Amal. module. We implemented this by removing the attention mechanism and equally blending predictions from different domains. To provide a comprehensive view, all ablation results are presented in the following table. These results illustrate that the omission of any module generally results in a noticeable decrease in performance, affirming the effectiveness of our design. Interestingly, for the Amal. module, some datasets like Cham. and Squi. exhibit only a modest performance decline. This observation aligns with our observation that their optimal attention values are close to an even split, as suggested in Figures 3(b) and 8(a).

Ablation study of SAF framework regarding “Node-wise Prediction Amalgamation” (Amal.), “Non-local Spatial Aggregation” (Spat.) and “Non-negative Spectral Filtering” (Spec.) modules.
|                  |   Cham.  |   Squi.  |   Texas  |   Corn.  |   Actor  |   Cora   |   Cite.  |   Pubm.  |
|------------------|----------|----------|----------|----------|----------|----------|----------|----------|
|   SAF-$\epsilon$  |   74.84  |   64.00  |   94.75  |   93.28  |   42.98  |   89.87  |   81.45  |   91.52  |
|   SAF            |   75.30  |   63.63  |   94.10  |   92.95  |   42.93  |   89.80  |   80.61  |   91.49  |
|   SAF w/o Amal.  |   75.01  |   62.62  |   89.18  |   86.07  |   41.53  |   88.80  |   80.37  |   91.24  |
|   SAF w/o Spec.  |   73.55  |   55.70  |   90.49  |   88.20  |   41.06  |   88.03  |   78.87  |   90.12  |
|   SAF w/o Spat.  |   68.53  |   51.39  |   92.62  |   92.13  |   41.71  |   88.51  |   80.08  |   88.51  |