Paper: Large-Scale Spectral Graph Neural Networks via Laplacian Sparsification (ICLR-2024)

Reviewer comment (Theory):
The theory is a rather simple application of the results of Daniel A. Spielman et al.’s theory. The theory only shows some relations between the original graph and the sparsified graph. However, it does not give any results about the performance of GNNs. The theory is disconnected from the GNN theme of this paper.

Author response:
Our model applies a spectral sparsification method to random-matrix polynomials and extends the theorem to include arbitrary (dynamic and learnable) polynomial coefficients. 
Additionally, we introduce a more intuitive method for enhancing efficiency in semi-supervised tasks, which functions as spectral sparsification. 
We acknowledge that the performance of GNN models can be empirical, particularly with the inclusion of a linear layer. 
Previous works like GraphSAINT and Cluster-GNN have provided substantial analysis on estimation and approximation, though not directly on the GNNs' performance. 
Our model's approximation capability has been demonstrated both theoretically and empirically.

Reviewer comment (Novelty):
Spectral sparsification for GNNs has been used widely in GNNs; the authors seem to ignore all related works that use spectral sparsification in the context of GNNs.

Author response:
We appreciate your input regarding the focus of most spectral sparsification methods on the original graph. 
Our work, however, concentrates on approximating the equivalent propagation matrix of spectral methods, $L_K=\sum_{k=0}^K w_k\tilde{L}^k$.
We initially consider that our model differs from these past methods since our target is not the same.
Recognizing the distinction, we will promptly include these methods in our study for relevant comparisons.

Reviewer comment (Experiments):
Instead of using spectral sparsification, the paper “Johannes Gasteiger, Stefan Weißenberger, Stephan Günnemann, Diffusion Improves Graph Learning, NeurIPS 2019” uses a thresholding approach for spectral approaches. Can the authors comment on this and provide some comparisons?

Author response:
We appreciate your reminding us of the model GDC. 
The design of GDC is based on the homophily assumption, which may not perform as well on heterophilous datasets compared to popular spectral methods. 
We will incorporate GDC into our paper for a more comprehensive discussion and applicable comparisons.

Reviewer comment (Theory):
The authors should include the computational complexity analysis for both memory and computational time. When taking the spectral sparsification step into account, the proposed approach seems also to require a very large memory footprint.

Author response:
We will instantly complement and clarify statements about the time and memory complexity.
If processed with a single thread, the time complexity of running SLSGC is $O(K n\log n)$, and for GLSGC, it is $O(K^2n\log n)$.
Besides, our method can be highly parallel processed with GPU.
The memory consumption, dependent on the number of edges sampled, is $O(n \log n)$ for SLSGC and $O(Kn\log n)$ for GLSGC.
Our experimental results show that the training time and memory consumption are lower than the base models, and here are some preliminary results for preview.
The detailed version will be soon updated in our paper.

| time per epoch (ms) | Cora | Computers | Twitch-gamer | Penn94 |
| ------------------- | ---- | --------- | ------------ | ------ |
| APPNP               | 6.50 | 7.59      | 19.99        | 13.75  |
| APPNP-LS            | 5.58 | 5.83      | 16.73        | 12.42  |

| GPU memory consumption (GB) | Cora  | Computers | Twitch-gamer | Penn94 |
| --------------------------- | ----- | --------- | ------------ | ------ |
| APPNP                       | 0.034 | 0.144     | 0.976        | 1.725  |
| APPNP-LS                    | 0.034 | 0.127     | 0.845        | 1.821  |

Reviewer comment (Experiments):
Numerical comparisons with the detached approach are missing.

Author response:
Detaching models like SGC often show lower performance due to their simpler architecture, in comparison with GCN. 
Another defect of detaching models is that the graph propagation of raw features is inapplicable when the dimension of raw features is high.
We will conduct and complete related experiments to provide a thorough comparison, and here are some experimental results for preview.

|          | Cora           | Cite.          | PubM.          | Actor          | Wisc.          | Corn.          | Texas          | 
| -------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- | 
| GPR      | 88.80$\pm$1.17 | 81.57$\pm$0.82 | 90.98$\pm$0.25 | 40.55$\pm$0.96 | 91.88$\pm$2.00 | 89.84$\pm$1.80 | 92.78$\pm$2.30 | 
| GPR-LS   | 89.31$\pm$1.07 | 81.65$\pm$0.53 | 90.95$\pm$0.37 | 41.32$\pm$0.83 | 93.63$\pm$2.88 | 91.14$\pm$1.15 | 92.62$\pm$1.48 |
| GPRdet   | 88.13$\pm$0.85 | 80.96$\pm$1.05 | 90.96$\pm$0.29 | 40.10$\pm$0.61 | 88.88$\pm$2.75 | 89.83$\pm$2.46 | 85.08$\pm$2.62 |

Reviewer comment (Experiments):
The authors may consider comparing against other approaches for scalable GNNs, e.g. Clustered GCNs.

Author response:
We will add scalable methods like Cluster-GCN and FastGCN to our paper for a more comprehensive comparison with relevant works.
Here are some experimental results for preview.

|            | Cora      | Cite.     | PubM.     | Actor     | Wisc.     | Corn.     | Texas     | Photo    | Comp.     | Twit.     |
| ---------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | -------- | --------- | --------- |
| GPR        | 88.80     | 81.57     | **90.98** | 40.55     | 91.88     | 89.84     | **92.78** | 95.10    | 89.69     | 73.90     |
| GPR-LS     | **89.31** | **81.65** | 90.95     | **41.82** | **93.63** | **91.14** | 92.62     | **95.3** | **90.47** | **73.49** |
| $\Delta$   | +0.51     | +0.08     | -0.03     | +1.27     | +1.75     | +1.30     | -0.16     | +0.20    | +0.78     | -0.42     |
| ClusterGCN | 87.45     | 79.66     | 86.52     | 29.66     | 61.88     | 56.72     | 65.08     | 93.17    | 87.11     | 68.69     |
| GraphSAGE  | 87.95     | 80.06     | 88.82     | 39.27     | 90.37     | 85.57     | 86.39     | 94.44    | 89.29     | 67.28     |

Reviewer comment (Novelty):
The method is mostly an application of a well-known Laplacian polynomial sparsification method.

Author response:
Our model employs a spectral sparsification method for the random-matrix polynomial and extends the theorem to include arbitrary polynomial coefficients, both static and dynamic. 
Furthermore, we propose an intuitive method to enhance efficiency in semi-supervised tasks, functioning as spectral sparsification.

Reviewer comment (Novelty):
Important related methods are not cited and compared against, making it difficult to judge the paper and understand where the proposed method fits relative to other methods.

Author response:
We are thankful for the comprehensive recommendations regarding related works. 
While we recognize the importance of these scalability-motivated subsampling efforts, we note that most existing scalable GNN frameworks, are based on message-passing principles, such as GCN, GCNII, and GAT.
These methods require only one-hop neighbor information at each propagation layer, differing from spectral methods. 
Scalable methods like GraphSAGE, LADIES, and GNNAutoScale are not compatible with spectral approaches. 
Similarly, subgraph sampling methods like Cluster-GCN and GraphSAINT do not adequately preserve the spectral properties of the original graph, making them unsuitable for spectral methods. 
We initially perceived a gap between methods designed for GCN-like networks and our work, potentially leading to unfair comparisons. 
Despite these, a more extensive survey and comparison with existing scalable GNNs based on subsampling will be conducted.
Here are some experimental results for preview.

|            | Cora      | Cite.     | PubM.     | Actor     | Wisc.     | Corn.     | Texas     | Photo    | Comp.     | Twit.     |
| ---------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | -------- | --------- | --------- |
| GPR        | 88.80     | 81.57     | **90.98** | 40.55     | 91.88     | 89.84     | **92.78** | 95.10    | 89.69     | 73.90     |
| GPR-LS     | **89.31** | **81.65** | 90.95     | **41.82** | **93.63** | **91.14** | 92.62     | **95.3** | **90.47** | **73.49** |
| $\Delta$   | +0.51     | +0.08     | -0.03     | +1.27     | +1.75     | +1.30     | -0.16     | +0.20    | +0.78     | -0.42     |
| ClusterGCN | 87.45     | 79.66     | 86.52     | 29.66     | 61.88     | 56.72     | 65.08     | 93.17    | 87.11     | 68.69     |
| GraphSAGE  | 87.95     | 80.06     | 88.82     | 39.27     | 90.37     | 85.57     | 86.39     | 94.44    | 89.29     | 67.28     |

Reviewer comment (Novelty):
There is no discussion or comparison with key theoretical works on transferability and stability of GNNs with respect to graph subsampling, including:  
- N. Keriven, A. Bietti, S. Vaiter. Convergence and Stability of Graph Convolutional Networks on Large Random Graphs.  
- R. Levie, W. Huang, L. Bucci, M. Bronstein, G. Kutyniok. Transferability of spectral graph convolutional neural networks.  
- L. Ruiz, L. Chamon, A. Ribeiro. Graphon neural networks and the transferability of graph neural networks.  
Subsequent related works by these authors and others should also be surveyed.

Author response:
Thank you all for engaging in the discussion regarding the relevance of the paper "N Keriven, A Bietti, S Vaiter. Convergence and Stability of Graph Convolutional Networks on Large Random Graphs" to our paper. 

The mentioned paper highlights the practicality of node-based subsampling methods in scaling GNNs to a larger scenario. Our proposed methods focus on preserving the spectral properties of the equivalent dense propagation matrix, $L_K=\sum_{k=0}^K w_k\tilde{L}^k$ (i.e. spectral filtered Laplacian), through edge sampling. 

Node-based sampling methods are prevalent in practical use. However, node-based methods like subgraph sampling methods explicitly lose the information of graph structure. The paper of N Keriven delves into the convergence and stability of GNNs with sampling under the strong assumption of random graphs, while our paper focuses on the graph structure connection between the sampled graph and the desired graph (here the $L_K$). 

We acknowledge that the paper mentioned by the reviewer FyWK indeed intersects with our work in the realm of scalability. It is noteworthy that many node-based scalability approaches do not work well in practice or have difficulty in the application when entangled with spectral GNNs. We plan to reference this paper and make some applicable comparisons in our work later.

Reviewer comment (Novelty):
The paper does not discuss relevant literature from the field of matrix sketching, where methods such as random row sampling are used to reduce complexity.

Author response:
Matrix sketching methods are effective with real-world graphs, typically sparse. 
However, sampling rows of $L_K$ results in a dense submatrix, still unsuitable for our purposes.
We plan to conduct an extensive review of these techniques and incorporate comparisons in our upcoming paper.

Reviewer comment (Novelty):
The authors should compare their method against the above-mentioned approaches, explain the novelty and improvements over existing methods, and also state any shortcomings compared with prior work. If the comparison is lengthy, a short version could be in the main paper with a longer version in the appendix.

Author response:
Our sparsification targets the equivalent propagation matrix of spectral methods, $L_K=\sum_{k=0}^K w_kL^K$, a dense matrix. 
Unlike previous works that focus on making the original graph sparser, our method aims to transform a dense matrix into a sparse one suitable for GNN training. 
The superiority of spectral methods in handling complex benchmarks, like heterophilous graphs, is evident both theoretically and empirically. 
While our methods are primarily designed for spectral approaches, a thorough comparison with classic scalable methods will be included.

Reviewer comment (Novelty):
The paper needs to better explain its improvement over the method in "Spectral sparsification of random-walk matrix polynomials," noting that this prior work approximates the polynomial of the matrix itself, while the current work applies the polynomial filter to the signal. Since sparse graphs allow simple and efficient direct implementations, the authors should clarify how their approach improves upon such direct methods.

Author response:
In our paper, one of our main ideas is to enhance the theorem proposed in [Spectral sparsification of random-walk matrix polynomials], and then sparsify the equivalent propagation matrix of spectral methods, i.e. $L_K=\sum_{k=0}^K w_k\tilde{L}^k$. 
We turn the polynomial filter operating on the signal into the corresponding format, i.e. the polynomial of the random-walk matrix, and thus make the training of spectral methods on large graphs applicable. 
The mentioned direct method has two defects, which are all reflected in our empirical results: 1) high-dimensional original features can cause GPU memory overload during initial graph propagation (e.g., 
$LX$ in dataset Penn94 on SGC), and 2) extremely large graphs like Ogbn-papers100M exceed GPU capacity, preventing graph propagation on hidden layers.

Reviewer comment (Theory):
On Page 2, first paragraph, a limitation of precomputing L^k X is noted — the network can only have one layer, as powers of the Laplacian cannot be precomputed for hidden layers.

Author response:
Yes, and that is how most spectral methods like BernNet improve their scalability. 
In our paper, we refer to these as "detached methods" and highlight their limitations. 
One of our contributions is enabling the spectral filtering operations applicable on hidden layers, even on large graphs.

Reviewer comment (Writing):
Clarify the term “nonzeros” in the phrase “keeping the number of non-zeros within an acceptable range” — nonzeros of what?

Author response:
We appreciate your scrutiny and will correct our paper accordingly.
The mentioned "non-zeros of the approximated $L_K$" will be more clearly defined.

Reviewer comment (Writing):
On Page 3, MOTIVATION section: Ensure consistent notation for the input signal (using either lowercase x or uppercase X), and avoid repeating material already stated, especially in the last paragraph.

Author response:
We will correct all the typos promptly. 
Additionally, we will streamline our wording to convey our motivation concisely.

Reviewer comment (Writing):
Correct “semi-definite” to “positive semi-definite” in Definition 3.1.

Author response:
The omission of "positive" will be addressed and added to our paper immediately.

Reviewer comment (Writing):
On Page 4, first line: Add missing period after “eigenvalues are in close correspondence.”

Author response:
We will add the missing period and make other necessary typographical corrections.

Reviewer comment (Theory):
In Equation (2), the first approximation is incorrect (e.g., for K=1 and w₁=1, P does not approximate L). If there is an alternate set of coefficients w’_k that make the polynomial in P approximate L, the authors should clarify. Also, powers of L should use small k.

Author response:
The error concerning the approximation (small $k$ instead of $K$) will be rectified instantly.

Reviewer comment (Writing):
Change “desiring matrix” to “desired matrix” two lines below Equation (2).

Author response:
Any additional typos identified will be corrected without delay.

Reviewer comment (Theory):
Theorem 3.2 is confusingly worded; “we can construct an ε sparsifier” sounds like an existence proof, but the intended meaning seems to be that with probability 1−K/n, Algorithm 1 returns an ε sparsifier.

Author response:
To alleviate any confusion, we will rephrase our statement to more precisely indicate that "The algorithm provides an $\varepsilon$ sparsifier with a high probability."

Reviewer comment (Theory):
Other papers achieve ε error without dependency on graph degree (e.g., Keriven et al., Theorem 1). The authors should explain the asymptotic differences, their reasons, and what benefits their method offers in light of slower asymptotics.

Author response:
As highlighted earlier, the primary distinction between our method and past subsampling approaches lies in the objective of approximation. 
While existing methods focus on approximating the original graph to maintain performance in GCN or other message-passing methods, our approach aims to approximate $L_K=\sum_{k=0}^K w_k\tilde{L}^k$, thereby enhancing the scalability of more potent spectral methods. 
The original graph could be sparse, while the number of nonzeros in $L_K$  grows exponentially when $K$ grows.
The challenge of scalability is more pronounced in spectral methods due to multi-hop neighbor aggregation, justifying the additional effort in approximation.

Reviewer comment (Theory):
The computational efficiency claim should be examined — for sparse L with O(n) edges, computing L_K already takes O(Kn) operations, while the proposed method requires O(K²n) operations. This suggests the method may only be useful for dense graphs, which can already be approximated using Monte Carlo node sampling. The authors should motivate their method in view of such cases, possibly comparing complexity for m=O(n^a) edges, with a between 1 and 2.

Author response:
Computing the random edges does not always make things faster for $L$, especially for the sparse one. 
One of our motivations is the computational difficulty of the whole graph propagation (i.e. $LX$) in many large-scale scenarios.
This issue is particularly acute in cases of precomputation with high-dimensional raw features or during propagation in hidden layers. 
**Calculating $LX$ iteratively** limits the applicability of several existing scalable methods, including GraphSAGE or GNNAutoScale. 
Hence, we propose to use laplacian sparsification to approximate $L_K$, a dense matrix.
This approximation enables us to store $L_KX$ and transform the $L_KX$ computation into one-time propagation, without calculating $LX$ iteratively. 
As the neighborhood explosion phenomenon exists in real-world graphs, Monte-Carlo sampling the nodes still involves an exponentially growing number of neighbors when $K$ grows, which is still not operatable in datasets like Ogbn-papers100M. 
GPR-GNN is the well-developed spectral method that directly applies $L$ on $X$ for constructing the polynomial. We will exert such theoretical analysis in comparing the base models and our entangled model.

Reviewer comment (Writing):
In Section 4, please define “effective resistance.”

Author response:
We acknowledge the recommendation and will provide a clear definition of relevant terms in the appropriate sections of our paper.

Reviewer comment (Novelty):
On Page 7, add the missing citation for the claim “This sparsifier can be further reduced to O(n log n/ε²) by the existing works [].”

Author response:
The missing reference will be added promptly to ensure completeness and accuracy.

Reviewer comment (Writing):
Correct “the proposed bound is much tighter than what is practically required” to “the proposed bound is much higher than what is practically required.”

Author response:
Our intent is to demonstrate that the theoretical bound of required edges is significantly higher than what is actually necessary for GNN training, without compromising performance. This claim is substantiated by our empirical results.

Reviewer comment (Evaluation):
It would be helpful if there were more detailed discussions and explanations when claiming that "our models show a slight performance advantage over the corresponding base models" on the heterophilous datasets proposed by Lim et al. (2021), and when discussing "approximating the desired complex filters tailored to the heterophilous graphs."

Author response:
We acknowledge that our explanation of the model's fitting ability on various datasets, particularly heterophilous ones, could be more comprehensive. 
Our approach, which approximates the equivalent propagation matrix of established spectral methods, generally inherits the fitting capabilities of these base models.
Models like GPRGNN, APPNP, and others, including unimplemented single-layer methods in our work like BernNet, have shown effectiveness on heterophilous datasets.
Due to space limitations in the main paper, our focus was on performance and scalability.
However, a more detailed analysis concerning heteophilous datasets and complex filters will be included in the appendix soon.

Reviewer comment (Novelty):
The theoretical analysis in this work is mostly based on the prior work of "Dehua Cheng, Yu Cheng, Yan Liu, Richard Peng, and Shang-Hua Teng. Spectral sparsification of random-walk matrix polynomials. CoRR, abs/1502.03496, 2015," and is a bit incremental.

Author response:
Our method expands upon the foundational theory established by Cheng et al., extending it to include dynamic (learnable) polynomial coefficients.
This theoretical backing has enabled us to devise two graph sparsification methods for both static and learnable coefficients, thereby broadening the applicability of spectral methods.
Additionally, we introduce a node-centric approach to improve efficiency in sparse-labeled graphs, which can be substantiated as a form of spectral sparsification independent of Cheng et al.'s work.

Reviewer comment (Experiments):
The proposed framework has many hyperparameters, which may make it impractical for use in real-world problems.

Author response:
Regarding the hyperparameter setup, our framework introduces only one additional parameter, ```--ec```, compared to the baseline spectral methods.
This controls the sampling density.
What may introduce the misunderstanding may stem from our code's simplification from a larger training framework supporting various baselines.
As clarified in Appendix A.4, only a fraction of the hyperparameters in the file ```train.py``` are employed for training a single model.

Reviewer comment (Evaluation):
The experimental results are not encouraging: spectral sparsification only produces marginal improvement for a few heterophilic graph datasets but degraded performance for well-known datasets.

Author response:
Our model is approximate, and its performance has a connection to the base method used. 
Most current scalable methods, including Cluster-GCN, GraphSAGE, GraphSAINT, LADIES, GNNAutoScale, etc., exhibit some performance loss compared to their base model, particularly as the scale of the benchmark grows.
These methods typically balance efficiency, performance and applicability.
Our model not only increases the applicability of spectral methods but also demonstrates comparable or superior performance even against robust base models like GPR-GNN, effectively addressing the issue of performance loss in scalability.

Reviewer comment (Reproducibility):
What's the percentage of edges that were retained after using the proposed spectral sparsification technique in GNN training?

Author response:
Our primary goal in sparsification is to approximate the equivalent propagation matrix of spectral methods, denoted as $L_K=\sum_{k=0}^K w_k L^k$, which is inherently a dense matrix.
Taking the dataset ```twitch-gamer``` as an example, with $K=2$, the non-zeros in $L_K$ amount to $6,827,688,588$.
In contrast, our approximated graph contains only $12,136,808(1.7\%)$ in GPR-LS and $6,068,404(0.9\%)$ in APPNP-LS with ```--ec=3```.
The number of sampled edges in our method increases linearly with $K$, while the non-zeros in $L_K$ grow exponentially, with an upper bound of $n^2$. The proportion of retained edges remains below $2\%$ in all cases.

Reviewer comment (Evaluation):
Is there any reduction in the overall GNN training time?

Author response:
Yes, our method reduces the training time of the base models.
We will provide a detailed wall-clock time comparison of GNN training in the updated version of our paper, and here is the preliminary results for preview.

| time per epoch (ms) | Cora | Computers | Twitch-gamer | Penn94 |
| ------------------- | ---- | --------- | ------------ | ------ |
| APPNP               | 6.50 | 7.59      | 19.99        | 13.75  |
| APPNP-LS            | 5.58 | 5.83      | 16.73        | 12.42  |

| GPU memory consumption (GB) | Cora  | Computers | Twitch-gamer | Penn94 |
| --------------------------- | ----- | --------- | ------------ | ------ |
| APPNP                       | 0.034 | 0.144     | 0.976        | 1.725  |
| APPNP-LS                    | 0.034 | 0.127     | 0.845        | 1.821  |

Reviewer comment (Evaluation):
What is the connection between spectral similarity for spectral sparsification and the final GNN performance (accuracy)?

Author response:
Our method employs an approximated equivalent propagation matrix for GNN training, enhancing scalability. 
Since we elevate the scalability, the performance is influenced by the imprecision introduced by the approximation. 
Generally, the closer the approximation is to the original, the more similar the performance will be to the base models. 
However, reducing the ```--ec``` value introduces greater randomness in the propagation matrix, potentially affecting performance significantly and influencing the optimal training epoch. 
GNN training performance remains an empirical aspect of current research.

Reviewer comment (Theory):
Overall, it is unclear if the proposed idea only applies to linear-in-the-parameters spectral GNNs. The definition of spectral GNNs (in the introduction) says they take the form $Y=g_w(L, f_{\theta}(X))$ where $g$ is a polynomial graph filter and $f$ is a linear layer. However, this formulation seems very restrictive and, for instance, does not encompass a simple 2-layer GCN.

Author response:
We will refine any ambiguity previously introduced regarding the application of our concept.
Our definition of polynomial filters aligns with the broader theme of **representation learning**
This is evident as the output delineates the learned representation of nodes, adaptable for various downstream tasks.
For example, in our implementation, we use $Z=f_{\theta'}(\sigma(Y))$ to denote the predicted labels, where $\sigma(\cdot)$ represents the nonlinear activation function.
The structure of $Y$ aligns with prevalent spectral methods employing a single layer of polynomial filters, such as GPR-GNN, BernNet, JacobiConv, and others.
We posit that the standard 2-layer GCN diverges from the typical framework of spectral GNNs.

Reviewer comment (Evaluation):
Although the motivation focuses on scalability, the experiments only measure predictive performance. I expected to see an extensive comparison of memory usage and wall-clock time for different methods and datasets. In addition, the paper should report error bars for assessing statistical significance.

Author response:
Our primary experimental findings focus on comparing the performance of our methods against established baselines.
We observe scalability issues, particularly regarding 1) the substantial memory requirements for graph propagation of raw features in the Penn94 dataset, and 2) the extensive storage demands of the Ogbn-papers100M dataset. 
Our methods demonstrate effective scalability in these contexts, underscoring their advantages.
A detailed analysis of time and memory consumption will be provided in our forthcoming paper, along with a demo version for preliminary feedback.
Additionally, we will soon update the error bars in our paper.

Reviewer comment (Experiments):
The paper only applies the proposed idea to APPNP and GPR GNNs. I would like to see results for other spectral GNNs (e.g., JacobiConv).

Author response:
We choose APPNP and GPRGNN for their soundness in the realm of spectral GNNs with static and learnable coefficients. 
Further experiments comparing our methods with other spectral GNNs like JacobiConv are underway and will soon be updated in our paper.

Reviewer comment (Theory):
The theory does not seem particularly useful since implementing GNNs involves non-linearities, rendering gradient estimates biased. Furthermore, results stem almost directly from previous works.

Author response:
Our approach is centered on developing an approximation of the equivalent propagation matrix in spectral GNNs, specifically $L_K = \sum_{k=0}^K w_k L^k$.
While approximation inherently involves some level of randomness and potential inaccuracy, we ensure practical performance by maintaining spectral properties as detailed in Theorem 3.2 

Previous studies often rely on subsampling, historical embedding, or some other techniques to achieve satisfactory practical performance.
However, these methods frequently fall short of providing robust support for gradient estimation.
The incorporation of linear layers and non-linear activations in these models makes their final performance challenging to analyze, often rendering the results more empirical.
Our model demonstrates impressive practical performance, thereby validating the effectiveness of our proposed methods. 

A novel aspect of our approach is the extension of the original theorem in a more generalized manner, incorporating dynamic (learnable) polynomial coefficients.
This theoretical foundation has enabled us to devise two versatile graph sparsification techniques, tailored for both static and learnable coefficients, enhancing the applicability of spectral methods.
Furthermore, we introduce an intuitive node-centric approach to improve efficiency in sparsely-labeled graphs.

Reviewer comment (Experiments):
What is the improvement in efficiency by applying the node-wise sampling method (section 3.3)? It would be useful to include some numbers in the Appendix.

Author response:
The time of our node-wise sampling methods decreases in datasets with sparse labels.
By directly sampling edges from involved labeled sets, our method avoids the wasted sampled edges of the whole-graph training.
For example, in the Ogbn-papers100M dataset, where labeled nodes account for only 1.39\%, edge sampling would result in over 99\% of edges being irrelevant as they connect unlabeled nodes.
We appreciate the recommendation on this matter and will address it in the upcoming revision of our paper.

Reviewer comment (Reproducibility):
Is the sampling (sparsification) procedure applied at each forward pass or only once before training?

Author response:
In our implementation, the sampling method precedes each forward pass.
When using learnable coefficients ($\mathbf{w}$), sampling is mandatory for each iteration.
However, for static coefficients, this step is optional.

Reviewer comment (Novelty):
I think the claim '[...] which is the first work tackling the scalability issue of spectral GNNs' is misleading since GCN can be viewed as a spectral GNN, and other works (e.g., SGC, LanczosNet) have tackled scalability issues of GCNs.

Author response:
We concur with the viewpoint that GCN embodies characteristics of both spectral and spatial GNNs, stemming from its roots in ChebNet and its neighborhood information aggregation mechanism.
We acknowledge that popular spectral filters typically operate in the spectral domain with various polynomial bases, represented as $Y=g_\mathbf{w}(L,f_\theta(X))$.
While scalability issues in GCN have been extensively explored in models like GraphSAGE, FastGCN, Cluster-GCN, GraphSAINT, and GNNAutoScale, our paper pioneers in addressing the scalability challenges specifically in spectral GNNs with polynomial filters.
We will refine our statement for greater accuracy and clarity.

Reviewer comment (Theory):
Please point out the exact Theorem in the paper (Cheng et al. 2015) when saying: 'We have extended the original theorem proposed by (Cheng et. al, 2015) ...'. Also, I suggest creating a specific subsection to prove Theorem 3.2 (as you have done for Theorem 4.3) --- I found the discussion in A.1 overloaded.

Author response:
We will pinpoint the relevant theorem and provide a more detailed elaboration on its proof.

Reviewer comment (Writing):
Some notation is introduced in Algorithm 1, such as e_u and e_v. Is e=(e_u, e_v) in step 1 of Algorithm 1?

Author response:
Yes, and we will ensure this explanation is included in the paper.

Reviewer comment (Theory):
'Some of the early classic GNNs, like GCNs, employ static Laplacian polynomials as the graph filter.' This is questionable since the coefficients of the linear layer can be viewed as multi-head spectral coefficients --- in fact, GCNs were introduced this way.

Author response:
The term 'static' implies that the coefficients $\mathbf{w}$ in $g_\mathbf{w}(\cdot)$ are not learnable and remain constant during training.
The linear layer is, however, actively trained.

Reviewer comment (Writing):
What is $\alpha$ in Section 3.2.1?

Author response:
The variable $\alpha$ represents the probability of halting at the current node during a random walk.
This will be clarified in our paper.

Reviewer comment (Writing):
$w$ has been used to denote both polynomial filter coefficients and weights in weighted graphs (Definition 3.1).

Author response:
We will rectify any symbol misuse immediately.

Reviewer comment (Writing):
The first identity of Eq. (2) should be $L^k$ instead of $L^K$.

Author response:
Thanks for the corrigendum. We will fix this instantly.

Reviewer comment (Presentation):
I would include the main algorithm (node-wise procedure) in the main text (btw, there is no appendix 8).

Author response:
All algorithms discussed in sections 3.2.1, 3.2.2 and 3.3 are our original contributions.
Due to space constraints, we have provided limited pseudo-codes, focusing on 'Static Laplacian Sparsified Graph Construction'.
We will correct the erroneous reference to the Appendix.

Reviewer comment (Theory):
Could you elaborate on the last identity in Eq. (2)? Or provide pointers?

Author response:
This comes from a natural substitution of symbol $\mathbf{P}$, leading to the following derivation:

$$
\sum_{k=0}^K w_k \mathbf{P}^K = 
\sum_{k=0}^K w_k \left(\mathbf{D}^{-1/2}\mathbf{AD}^{-1/2}\right)^k =
\sum_{k=0}^K w_k \mathbf{D}^{1/2}\left(\mathbf{D}^{-1}\mathbf{A}\right)^k \mathbf{D}^{-1/2}
$$
$$
=\mathbf{D}^{-1/2}\cdot\mathbf{D}\left(\sum_{k=0}^K w_k\left(\mathbf{D}^{-1}\mathbf{A}\right)^k \right)\mathbf{D}^{-1/2}.
$$