Paper: ResTran: A GNN Alternative To Learn Graph With Features (ICLR-2024)

Reviewer comment (Novelty):
Insufficient literature survey on the related work.

Author response:
Thank you for pointing out the literature. We first apologize that we should have been careful when we use "simple." We say simple since we separate the graph consumption and prediction; consumption is ResTran and we may use any vector based ML methods that include the existing ones such as LP, SVM, AVAE and so on. This may be phrased as "versatile." GNNs are designed to graph problems, while we can use any vector based methods, such as AVAE or SVM. In this sense, while we appreciate the work you raised [1,2,3,4], those are sophisticated GNN models, which are different from our aims.

Reviewer comment (Novelty):
One major claim of the paper is that current GNNs cannot handle heterophilic graphs without using complex architecture, but this is apparently wrong as spectral GNN research addresses this problem with simple design [1,2,3], and LINKX [4] is another simple architecture with superior performance on heterophilic graphs. Spectral GNNs can learn any graph spectral filtering beyond low-pass (homophily) and high-pass (heterophily) cases, yet this literature is completely ignored.

Author response:
Thank you for pointing out the literature. We first apologize that we should have been careful when we use "simple." We say simple since we separate the graph consumption and prediction; consumption is ResTran and we may use any vector based ML methods that include the existing ones such as LP, SVM, AVAE and so on. This may be phrased as "versatile." GNNs are designed to graph problems, while we can use any vector based methods, such as AVAE or SVM. In this sense, while we appreciate the work you raised [1,2,3,4], those are sophisticated GNN models, which are different from our aims.

Reviewer comment (Novelty):
The idea of obtaining node embedding from node features and graph topology in an unsupervised fashion has been proposed previously, for example in SIGN [5], which computes propagated features $X, AX, A^2X,\cdots,A^KX$ and concatenates them for downstream MLP tasks. This inspired works like SAGN [6] and GAMLP [7]. ResTrans is essentially using the embedding $L^{-1/2}X$, and although directly computing $L^{-1/2}$ is infeasible, the proposed Krylov subspace method is very similar to the SIGN design. This line of work is also ignored, making the novelty relatively limited.

Author response:
Thank you also for pointing out SIGN. The difference between SIGN and ResTran is as follows. SIGN learns the weight of $X, AX, \ldots, A^{r}X$ through a GNN framework, while we project $L^{+1/2}$ to Krylov subspace. In this sense, SIGN is a more established model. Also, since SIGN needs to learn the weight of  $X, AX, \ldots, A^{r}X$ and store them, SIGN needs space. In fact, $r$ for SIGN is typically smaller, like 5 or less as discussed in the SIGN paper. On the other hand, since our ResTran aims to $L^{+1/2}$, we don't have to store each weight of  $X, AX, \ldots, A^{r}X$, instead, we only need to have the summation of these. By this we have more space. Also, since the weight of $X, AX, \ldots, A^{r}X$ are learnt by GNNs, these have GNNs' advantages as well as ones' disadvantages, which we want to avoid.

Reviewer comment (Novelty):
The authors should explain the differences of their method compared to SIGN and compare them carefully in the experiments.

Author response:
Thank you also for pointing out SIGN. The difference between SIGN and ResTran is as follows. SIGN learns the weight of $X, AX, \ldots, A^{r}X$ through a GNN framework, while we project $L^{+1/2}$ to Krylov subspace. In this sense, SIGN is a more established model. Also, since SIGN needs to learn the weight of  $X, AX, \ldots, A^{r}X$ and store them, SIGN needs space. In fact, $r$ for SIGN is typically smaller, like 5 or less as discussed in the SIGN paper. On the other hand, since our ResTran aims to $L^{+1/2}$, we don't have to store each weight of  $X, AX, \ldots, A^{r}X$, instead, we only need to have the summation of these. By this we have more space. Also, since the weight of $X, AX, \ldots, A^{r}X$ are learnt by GNNs, these have GNNs' advantages as well as ones' disadvantages, which we want to avoid.

Reviewer comment (Experiments):
The authors should compare their method to spectral GNN baselines such as those in [1,2,3], otherwise it is hard to be convinced that ResTrans is a good method for heterophilic graphs.

Author response:
Thank you for pointing out the literature. We first apologize that we should have been careful when we use "simple." We say simple since we separate the graph consumption and prediction; consumption is ResTran and we may use any vector based ML methods that include the existing ones such as LP, SVM, AVAE and so on. This may be phrased as "versatile." GNNs are designed to graph problems, while we can use any vector based methods, such as AVAE or SVM. In this sense, while we appreciate the work you raised [1,2,3,4], those are sophisticated GNN models, which are different from our aims.

Reviewer comment (Evaluation):
I am not sure about the computational complexity of these methods: The authors should have included some experimental results on time complexity to indicate whether the usually expensive eigenvector computations can be justified instead of simple combinatorial message passing.

Author response:
Thank you very much for the review! We will address what you raised in weakness and question sections.

> Complexity analysis 

We provided the complexity analysis on page 4. Basically, ResTran takes O(rfm), and r is typically small, e.g., our experiments we used 15. O(fm) is equivalent to the established GNN models.

Reviewer comment (Experiments):
Spectral embeddings/methods have inherent limitations in the kind of data they can capture from a graph: They fail to capture relational aspects of data (such as node/edge-colors) and so on, unlike combinatorial message-passing algorithms. If the authors propose such a radical departure from standard GNN methods, they should investigate their method on a variety of datasets, such as molecular graphs or synthetic graphs arising from relational sources.

Author response:
Thank you very much for the review! We will address what you raised in weakness and question sections.

> Edge-coloring

Thank you for the suggestion. As stated in future work, at this stage, we are unsure how much of the expressive power our ResTran has. We speculate that at least ResTran has expressive power as two established models, GCN and GAT. However, we leave these as future work.

Reviewer comment (Experiments):
The computational cost of spectral methods typically goes to O(n^3). Can the authors comment on the running time complexity of the Krylov-subspace based embedding and compare it with the run-time costs of a standard message-passing GNN? Have the authors carried out any experiments to compare the run-time costs of the two approaches? Especially, I would like to know the status on sparse graphs, where the pseudo-inversions might be way more costly than message-passing. And how does the time complexity scale with graph size? The empirical investigation considers only medium-sized graphs.

Author response:
Thank you very much for the review! We will address what you raised in weakness and question sections.

> Complexity analysis 

We provided the complexity analysis on page 4. Basically, ResTran takes O(rfm), and r is typically small, e.g., our experiments we used 15. O(fm) is equivalent to the established GNN models.

Reviewer comment (Novelty):
How does the proposed method differ from the commonly used positional encodings based on spectral properties of the input graph?

Author response:
Thank you very much for the review! We will address what you raised in weakness and question sections.

> Difference between ResTran and positional encoding 

As written in Appendix B, our ResTran can be plugged into any vector based model, while positional encoding using spectral features aims to graph transformer architectures.

Reviewer comment (Novelty):
Not much novelty from traditional spectral clustering methods, most part of the papers are well-known results or naive extension of existing methods.

Author response:
As summarized in Appendix G.2, our theorem 8 is novel for the following points. Mainly, while the previous studies (stated in Prop 2), such as (Dhillion et al., 2004), only hold for the vector data, Thm.8 holds for the "given" graph data. Note also that the proof of Thm.8 is NOT the off-the-shelf application of Prop 2. See more in Appendix G.2

Reviewer comment (Presentation):
Most of the background or preliminaries can be distilled into shorter context or put in appendix such as propositions from previous papers, it’s currently taking more than 2 pages of the main paper.

Author response:
Thank you for the suggestion. We will endeavor to further shorten the preliminaries. We also stress that our main claims are not incremental results of preliminaries.

Reviewer comment (Theory):
No complexity analysis to support the claim that it’s less complicated than GNN.

Author response:
We provided the complexity analysis on page 4.

Reviewer comment (Experiments):
Using the shifted graph Laplacian term b to control the heterphilous information in feature map seems to require a lot of fine tuning. How to choose the hyperparameters (b, r, etc) in experiment section is not clear to me, based on the appendix the authors used a fixed value, some ablation study would be nice to see.

Author response:
Thank you for pointing out. From our preliminary experiments, the choice of b does not affect much. Also, the choice of r does not affect either if r is larger than 10. Thus, for simplicity, we used the fixed b and r. We will further clarify this point.

Reviewer comment (Novelty):
There are multiple works in GNN that already support heterophilous dataset without over-smoothing. The authors’ claim about the lack of GNN is not valid. I would suggest the authors to at least do some comparison with the recent ones.

Author response:
We are aware that multiple GNN models address the homophily bias. However, in the experiments, as written in the manuscript, we wanted to exclude sophistication for both methods as much as possible. Thus, we did not compare with the advanced models you raised. At the same time, we do not use some complicated methods for ResTran either. See more for the reply to Fupb.

Reviewer comment (Experiments):
Experiment section lack of comparison to more recent GNN works that also targeting at heterophilous datasets.

Author response:
We are aware that multiple GNN models address the homophily bias. However, in the experiments, as written in the manuscript, we wanted to exclude sophistication for both methods as much as possible. Thus, we did not compare with the advanced models you raised. At the same time, we do not use some complicated methods for ResTran either. See more for the reply to Fupb.

Reviewer comment (Novelty):
It seems like the authors didn't include most recent works in heterophilous GNNs and most of the claims against GNNs are lack of support. To name a few, JKNet [Xu et al., 2018], H2GCN [Zhu et al., 2020a], Geom-GCN [Pei et al., 2020],  GPR-GNN [Chien et al., 2020], GPNN [Yang et al., 2022] and many more are all methods that work with heterophilous graph dataset. I would recommend the authors to at least go over the literature before making the final conclusion.

Author response:
We are aware that multiple GNN models address the homophily bias. However, in the experiments, as written in the manuscript, we wanted to exclude sophistication for both methods as much as possible. Thus, we did not compare with the advanced models you raised. At the same time, we do not use some complicated methods for ResTran either. See more for the reply to Fupb.

Reviewer comment (Novelty):
It seems to me ResTran is closely related to 1-layer message-passing neural networks (MPNNs), since it utilizes $L^+$ to transform node features $X$ and it is known that $ L^+_{ij} $ represents the effective resistance between two end nodes in the graph interpreted as an electrical network. While traditional MPNNs utilize adjacent matrix $A$ to transform $X$ and have weight 0 when there is no edge between two nodes, I find the key idea is similar, which sounds like ResTran is still in some sense a GNN.

Author response:
Our contribution is to propose ResTran. Using ResTran, we obtain a vector representation of the graph-with-features. We may apply the transformed vector to any vector-based ML method. From this sense, our aim can be explained as "separating" consumption of graph and prediction. Contrary, GNNs do these at the same time. If we use NN models, the first layer "touches" $L^{+1/2}$; in some sense, we may say this is a sort of GNN. However, again, the point is that we separate consumption of graphs and prediction, which is different from most the frameworks of GNNs.

Reviewer comment (Evaluation):
I do not really see that ResTran is simpler than existing GNNs. Depending on the definition of complexity, I find the feature transformation is already non-trivial as it includes utilizing Krylov subspace method to approximate the transformed features. After that, from the experiments, it seems it still needs complex neural networks to get decent results, i.e., AVAE, and using simple methods such as SVM does not seem to work.

Author response:
We apologize for the confusion. We should have been careful when we use "simple." We say simple since we separate the graph consumption and prediction; consumption is ResTran, and we may use any ML methods that include the existing ones. To describe this separation, this may be phrased as, say, "versatile." GNNs are designed for graph problems, and therefore we apply GNNs only to some specific problems over graphs. On the other hand, we can use any vector-based methods, such as AVAE or SVM, which is more "versatile" methods than GNNs.

Reviewer comment (Experiments):
I am unsure if the comparison is fair. Different from traditional GNNs, which propagate features with $A$, use MLPs to make predictions, and train the network with a classification loss, it seems critical for ResTran to use some semi-supervised models such as VAT and AVAE to get good results. However, the node representations yielded by those GNNs are not trained in the same way, e.g., VAT involves adversarial training--it is known that adversarial training can also further improve GNNs' performance [2].

Author response:
We are aware that various GNNs address various problems. Also, there may be various vector-based methods that better work for ResTran. For example, there may be better NN layers for ResTran or some kernel functions that work better than Gaussian kernel together with SVMs. However, as written in the manuscript, in order to eliminate any sophistications for both GNNs and ResTran, we may want to keep our experimental setting as simple as possible. For this purpose, i) we use the most established models, and ii) We use these methods with the simplest settings. For GNNs, GCN and GAT are the two most established models. We used SGC since SGC is close to ResTran, as discussed in Appendix B. For ResTran, AVAE is a simpler version of VAE, and VAE is one of the most established models. VAT is one of the early established models for semi-supervised learning using NNs. Also, we use SVMs with the established kernel function, Gaussian. We use GNNs and ResTran with the simplest settings as written in the manuscript. Also, by this, we say our comparison is fair since we do not use any sophistication for both methods. We simply plug in three GNN methods and SVM, LP, NN based methods, all of which are already established models.

Reviewer comment (Experiments):
Are there any particular reasons for using a different dataset split instead of the one that has been widely used in the previous literature?

Author response:
As written in the manuscript, we avoided using public split since "overfitting" to for some classical datasets public split is reported (Shchur et al. 2018). Therefore, we use random splitting.