Paper: SEESAW: Do Graph Neural Networks Improve Node Representation Learning for All? (ICLR-2024)

Reviewer comment (Novelty):
This article lacks novelty to some extent, despite providing detailed analysis and experiments. The conclusions are relatively trivial and are already a consensus in the community.

Author response:
We agree with the reviewer that a part of the observations revealed in this paper has also been reported by existing literatures, e.g., GNNs could bear sub-optimal performances on highly heterophilic nodes. However, we would like to note that the main focus of this paper is to take an initial step to perform a comprehensive comparison between GNNs and shallow methods. Especially, to the best of our knowledge, no existing work has comprehensively explored the strengths and weaknesses of GNNs compared with shallow methods with a unified view. Therefore, a comprehensive analysis presented in our paper is still needed to have an in-depth understanding of GNNs. For example, it is valuable to point out that dimensional collapse solely happens in GNNs instead of traditional shallow methods, and such a conclusion has not been revealed by any other existing works.

Reviewer comment (Novelty):
The paper may be improved if it discusses some works that combine the advantage of network embedding and GNN, like [1,2].

Author response:
We would like to thank the reviewer for pointing this out. Specifically, we note that [1] successfully achieves optimization for the context hyper-parameters of shallow graph embedding methods. However, the proposed approach cannot take node attributes as the input and thus fails to effectively utilize the information encoded in node attributes; [2] explored to generalize the GNNs to adaptively learn high-quality node representations under both homophilic and heterophilic node label patterns. Nevertheless, it fails to avoid using the node attributes as the learning prior, and thus still follows a design that has been proved to bear dimensional collapse in our paper.  

We thank the reviewer again for bringing this up so we can make our discussion more comprehensive. We have included the discussion in Appendix C.11.  

[1] Abu-El-Haija S, Perozzi B, Al-Rfou R, et al. Watch your step: Learning node embeddings via graph attention[J]. Advances in neural information processing systems, 2018, 31.   

[2] Chien E, Peng J, Li P, et al. Adaptive Universal Generalized PageRank Graph Neural Network[C]//International Conference on Learning Representations. 2020.

Reviewer comment (Reproducibility):
The result of E2E-GCN on GCN, CiteSeer, Pubmed is lower than that reported in the paper. Can the authors explain the difference of experimal setting?

Author response:
We thank the reviewer for pointing this out. We utilized standard implementations of the GCN layer from standard packages, and the performance differences do not influence our observations and conclusions.  We agree with the reviewer that such inconsistency over these most commonly used datasets could lead to confusion, and we have changed the corresponding performances to the results obtained based on the implementation from the original papers to avoid any further confusion.

Reviewer comment (Novelty):
Is it possible to overcome the weakness of both GNNs and shallow embedding methods, and propose a new graph representation paradigm to combine their strengths?

Author response:
We agree with the reviewer that there could be ways to combine their strengths, and we believe this would be an interesting future direction to work on. However, despite the significance of this problem, we would also like to note that **handling such a problem is difficult**. In fact, most existing works fail to avoid the weaknesses characterized by our paper, and thus they are not able to properly combine the strengths of the two methods. We believe it would be exciting for the graph machine learning community to explore such a new future direction.

In fact, it is great that this paper can lead to questions like this from the readers since we believe that this is part of what we want with this work: to have researchers and practitioners not only looking at GNNs but also traditional shallow embedding methods for further advancements.  

Thank you for your prompt reply, and we truly appreciate your recognition. 

We agree that being able to propose a state-of-the-art solution to solve this problem would be great. Nonetheless, this would require huge efforts that can easily go beyond the scope of a single paper, especially considering that our paper already contains 20+ pages. We'd like to note that one of the main goals of this paper is to carefully pose this problem with a systematic analysis and draw more attention from the research community to it, as the main research interests have been shifting away from the traditional shallow methods in the past few years. We argue that these efforts make our work particularly interesting to the graph learning community, and our study is also critical for any future follow-up works that could propose a SOTA solution to it.

Reviewer comment (Novelty):
It is already shown that MLPs (which is same with GNNs without aggregation) outperforms commonly used GNNs such as GCN, GAT, and SAGE. Furthermore, several methods are proposed to perform well on both homophilous and heterophilous graphs [1, 2, 3].

Author response:
We sincerely appreciate the time and efforts you've dedicated to reviewing and providing invaluable feedback to enhance the quality of this paper. We provide a point-to-point reply below for the mentioned concerns and questions.  

---


>  **Reviewer**: W1. It is already shown that MLPs (which is same with GNNs without aggregation) outperforms commonly used GNNs such as GCN, GAT, and SAGE. Furthermore, several methods are proposed to perform well on both homophilous and heterophilous graphs [1, 2, 3].

**Authors**: We thank the reviewer for pointing this out. We have been aware that MLPs may outperform GNNs in certain cases, and our work is not tailored for the homophily level of the input graph data. We elaborate on the details below.  

We note that the main focus of this paper is to perform a comprehensive comparison between GNNs and shallow embedding methods. The observation and discussion will remain on both homophilous and heterophilous graphs. Existing works (e.g., [1, 2, 3]) do not explicitly explore the strengths and weaknesses of GNNs compared with shallow methods. We argue that the novelty of our paper mainly lies in the comprehensive comparison of (1) the levels of dimensional collapse and (2) performances over different levels of homophily between the two types of models.  

[1] Zhu, Jiong, et al. "Beyond homophily in graph neural networks: Current limitations and effective designs." Advances in neural information processing systems 33 (2020): 7793-7804.

[2] Lim, Derek, et al. "Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods." Advances in Neural Information Processing Systems 34 (2021): 20887-20902.

[3] Yang, Liang, et al. "Diverse message passing for attribute with heterophily." Advances in Neural Information Processing Systems 34 (2021): 4751-4763.

Reviewer comment (Novelty):
It seems that the circumstances where input features are limited is not common in the real-world senarios and we can augment input features with large language models even in attribute-poor settings [4].

Author response:
**Authors**: We thank the reviewer for pointing this out. We would like to note that, in fact, the circumstances where input features are limited are common. For example, online social networks have become prevalent, while it is common that anonymized social networks usually lack the identity information of nodes [1]. As another example, the attribute of a node in social networks may also be an artifact and captures no semantic meanings [1]. Therefore, the phenomenons revealed by our paper widely exist, and such a weakness would also be shown across a series of real-world applications.  

We also agree with the reviewer that input features could be augmented with large language models in attribute-poor settings, which is an exciting future direction to explore. However, augmenting features with LLMs might not scale to real-world scenarios. For example, a social network could have billions of nodes [2], and generating node features with LLMs may bear high computational costs.  

[1] Sun, Z., Zhang, W., Mou, L., Zhu, Q., Xiong, Y., & Zhang, L. (2022, June). Generalized equivariance and preferential labeling for gnn node classification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 8, pp. 8395-8403).  

[2] Przepiorka, A., & Blachnio, A. (2016). Time perspective in Internet and Facebook addiction. Computers in Human Behavior, 60, 13-18.

Reviewer comment (Novelty):
It is not surprising that GNNs suffer dimensional collapses when input features are poor since it is well-known for general neural netoworks.

Author response:
**Authors**: We agree with the reviewer that dimensional collapse has been found to happen in general neural networks by a series of existing works. However, we would like to point out that such a phenomenon does not necessarily mean that GNNs will also suffer a similar problem. Especially, to the best of our knowledge, no existing work has comprehensively explored the strengths and weaknesses of GNNs compared with shallow methods with a unified view. Therefore, a comprehensive analysis is still needed to have an in-depth analysis of GNNs. For example, it is valuable to point out that dimensional collapse solely happens in GNNs instead of traditional shallow methods, and such a conclusion has not been revealed by any other existing works.

Reviewer comment (Novelty):
There are several models combining walking-based approaches and GNNs such as APPNP [1]. I think that this kind of mechanism might alleviate the problem of GNNs due to the adoption of pagerank. Does APPNP also suffer similar problems as other GNNs?

Author response:
**Authors**: We thank the reviewer for pointing this out. First, we would like to point out that in the mentioned model APPNP, only the way of the information distribution in APPNP is similar to the random walk. However, as is revealed in our paper, **the problem of dimensional collapse is caused by utilizing the node attributes as the prior of learning**. Since APPNP still utilizes the node attributes as the prior of learning, **APPNP does not bear any significant difference in terms of the cause for dimensional collapse** compared with the GNNs adopted in this paper. Therefore, we conclude that APPNP naturally suffers similar problems as other GNNs.  

In addition, we also perform empirical experiments based on APPNP. We present the unsupervised learning performances on the Cora dataset below as an example. Here utility is measured by node classification accuracy, while the effective dimension ratio (EDR) is measured by the ratio of the value of rank (of representation matrix) to the representation dimensionality. We observe that similar to GCN, APPNP also bears severe dimensional collapse (exhibited by the significant reduction in the value of EDR).  

|       | 100%Att, Acc | 100%Att, EDR | 1%Att, Acc | 1%Att, EDR | 0.01%Att, Acc | 0.01%Att, EDR |
| ----- | ------------ | ------------ | ---------- | ---------- | ------------- | ------------- |
| GCN   | 67.8%        | 96.9%        | 36.9%      | 35.5%      | 32.3%         | 1.56%         |
| APPNP | 75.5%        | 56.3%        | 42.4%      | 13.7%      | 10.6%         | 0.78%         |

We note that the discussion above reveals that the problem pointed out by our work is non-trivial to handle. Correspondingly, this paper is particularly interesting to researchers working in this area and follow-up studies of our work remain desired.  

We thank the reviewer again for bringing this up so we can make our evaluation more comprehensive. We have included the experimental results and the corresponding discussion in Appendix C.10.

Reviewer comment (Experiments):
Several approaches such as LINKX [2] encode node topology and node attribute separately and combine two representations later. Since these approaches can learn how much to reflect node attributes on node representations, I think that these methods might not suffer dimensional collapse. Does LINKX also suffer similar problems?

Author response:
**Authors**: We agree with the reviewer that dimensional collapse could be mitigated by encoding node topology and node attribute separately and combining two representations later. However, it is also worth noting that **the adjacency matrix is naturally low-rank as well** [1, 2], and thus it could be difficult to avoid dimensional collapse with the operations above. Specifically, we perform empirical experiments based on LINKX. We present the unsupervised learning performances on the Cora dataset below as an example. Here utility is measured by node classification accuracy, while the effective dimension ratio (EDR) is measured by the ratio of the value of rank (of representation matrix) to the representation dimensionality. We observe that compared with GCN, LINKX exhibits smaller values of EDR in attribute-rich scenarios (e.g., 100% available node attributes), while it also mitigates dimensional collapse in attribute-poor scenarios (e.g., 0.01% available node attributes). This demonstrates that (1) such an approach may jeopardize the effective dimension ratio in attribute-rich scenarios and (2) such an approach effectively helps to mitigate dimensional collapse in attribute-poor scenarios.  

|       | 100%Att, Acc | 100%Att, EDR | 1%Att, Acc | 1%Att, EDR | 0.01%Att, Acc | 0.01%Att, EDR |
| ----- | ------------ | ------------ | ---------- | ---------- | ------------- | ------------- |
| GCN   | 67.8%        | 96.9%        | 36.9%      | 35.5%      | 32.3%         | 1.56%         |
| LINKX | 65.9%        | 50.4%        | 64.6%      | 49.6%      | 68.8%         | 24.2%         |

However, we would also like to point out that even if such a method successfully mitigates dimensional collapse for GNNs, it **is not ideal**, since it (1) sacrifices the capability of GNNs in inductive learning and (2) improves the computational complexity from $\mathcal{O}(n*k)$  to $\mathcal{O}(n^2)$ to perform inference ($k$ is the number of node attributes and $n$ is the number of nodes). Therefore, the problem of dimensional collapse is non-trivial to handle, and more analysis can be a great follow-up study of our work.  

We thank the reviewer again for bringing this up so we can make our evaluation more comprehensive. We have included the experimental results and the corresponding discussion in Appendix C.10.  

[1] Entezari, N., Al-Sayouri, S. A., Darvishzadeh, A., & Papalexakis, E. E. (2020, January). All you need is low (rank) defending against adversarial attacks on graphs. In *Proceedings of the 13th International Conference on Web Search and Data Mining* (pp. 169-177).  

[2] Zhuang, L., Gao, H., Lin, Z., Ma, Y., Zhang, X., & Yu, N. (2012, June). Non-negative low rank and sparse graph for semi-supervised learning. In *2012 ieee conference on computer vision and pattern recognition* (pp. 2328-2335). IEEE.

Reviewer comment (Novelty):
The novelty of the work seems limited - most of the observations are straightforward or appeared in previous research.

Author response:
We agree with the reviewer that a part of our observations has been revealed by existing works.  However, we note that the main focus of this paper is to perform a comprehensive comparison between GNNs and shallow embedding methods. Especially, to the best of our knowledge, no existing work has comprehensively explored the strengths and weaknesses of GNNs compared with shallow methods with a unified view. Therefore, a comprehensive analysis is still needed to have an in-depth analysis of GNNs. For example, it is valuable to point out that dimensional collapse solely happens in GNNs instead of traditional shallow methods, and such a conclusion has not been revealed by any other existing works.

Reviewer comment (Evaluation):
While the paper contains a guide for practitioners about which model to choose, it is not specific enough to be directly applied to a given application. For instance, in Section 5, it is written "we recommend adopting GNNs and shallow embedding methods on attribute-rich and attribute-poor networks, respectively." However, it is not clear how to decide whether the attributes are rich. For example, in both Flickr and PubMed, there are 500 features, but the results on them are completely different. So, it is not the number of features that can be used for this decision.

Author response:
We thank the reviewer for pointing this out. We would like to note that rich attributes do not necessarily mean a large number of available features. For example, no matter what the dimensionality of the input node features is, if most of the available features are linearly correlated with each other, i.e., the rank of the input feature matrix is small, then dimensional collapse is still likely to happen. This is because as shown in Figure 5.b, the non-linear operations in GNNs can hardly improve the rank of representations (compared with node features) regardless of the dimensionality. Correspondingly, we note that it is the rank of the input features that truly determines whether it is an attribute-poor scenario or not. However, since the rank of the input feature matrix cannot exceed the number of attribute dimensions, the number of attribute dimensions also plays a critical role in determining whether it is an attribute-poor scenario. In addition, we note that both Flickr and PubMed are not attribute-poor datasets, since the GNN is able to fully utilize most hidden dimensionalities when 100% attributes are available. We have improved the corresponding explanation in Section 4.3 and Section 5 accordingly to avoid further confusion.  

Furthermore, we would like to point out that it is not appropriate to compare performances across different datasets, since how much the node attributes help with the GNN prediction could vary across different datasets. Instead, we present Figure 2 to compare the performances within each dataset under different levels of available input attribute dimensions, which makes the analysis of dimensional collapse more rigorous.

Reviewer comment (Theory):
It is written that "It is difficult for shallow embedding methods to properly exploit information encoded in node attributes and make use of the homophily nature of most graphs" - why the latter is true? Classic methods have similar embeddings for nodes located close to each other in the graph. Under the homophily assumption, such nodes often have the same label.

Author response:
We would like to thank the reviewer for pointing this out. We agree that the latter is not appropriate, and the main claim here is that shallow embedding methods cannot effectively exploit node attribute information. We have revised the expression accordingly to avoid confusion.

Reviewer comment (Experiments):
The fact that GNNs strongly rely on node features and removing them leads to decreased performance is very natural. Can this problem be solved by augmenting node features with structural graph-based features? Or maybe even with random features? Both options can also increase the representation effective dimension.

Author response:
We agree with the reviewer that feature augmentation could be a potential direction to explore, such that the disadvantage of GNNs revealed in this paper can be mitigated. However, the problem of dimensional collapse cannot be properly addressed simply by augmenting node attributes by either adding structural graph-based features or adding random features. We performed experiments by (1) concatenating a random matrix with the same dimensionality as the original node attributes onto the node attribute matrix and (2) concatenating a matrix encoded with structural information following the state-of-the-art *degree+* strategy [1] onto the node attribute matrix. We present the unsupervised learning performances on the Cora dataset below as an example. Here utility is measured by node classification accuracy, while the effective dimension ratio (EDR) is measured by the ratio of the value of rank (of representation matrix) to the representation dimensionality.

|                      | 100%Att, Acc | 100%Att, EDR | 1%Att, Acc | 1%Att, EDR | 0.01%Att, Acc | 0.01%Att, EDR |
| -------------------- | ------------ | ------------ | ---------- | ---------- | ------------- | ------------- |
| Walk-GCN             | 67.8%        | 96.9%        | 36.9%      | 35.5%      | 32.3%         | 1.56%         |
| Walk-GCN, Random     | 68.0%        | 97.3%        | 29.9%      | 67.2%      | 10.6%         | 5.08%         |
| Walk-GCN, Structural | 69.0%        | 97.3%        | 37.3%      | 35.2%      | 32.3%         | 2.34%         |

We observe that: (1) concatenating a matrix with structural information slightly improves the node classification accuracy, while such a strategy does not stop the significant drop in the rank of node representations; (2) concatenating random node attributes successfully improves the rank of the node representations, however, the classification accuracy is reduced. Therefore, both strategy does not really solve the problem of dimensional collapse, and we believe handling such a problem is non-trivial. Correspondingly, this paper is particularly interesting to researchers working in this area and such a problem is also worth to be explored in future works.  

We thank the reviewer again for bringing this up so we can make our evaluation more comprehensive. We have included the experimental results and the corresponding discussion in Appendix C.9.  

[1] Cui, H., Lu, Z., Li, P., & Yang, C. (2022, October). On positional and structural node features for graph neural networks on non-attributed graphs. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (pp. 3898-3902).

Reviewer comment (Theory):
Can small representation effective dimension be explained just by the dimension of the initial feature set? This would also explain Figure 5(b) since increased rank bound cannot solve this issue.

Author response:
We thank the reviewer for pointing this out and we agree with the reviewer. Specifically, if the number of the dimensions of the initial feature set is already small, this means the initial feature set bears the problem of low rank, since its rank cannot exceed its feature dimensionality. Accordingly, the learned node representations are then more likely to bear the dimensional collapse, since the GNNs cannot effectively improve the rank again even if the rank bound increases (i.e., the number of hidden dimensionality increases). Therefore, the number of attribute dimensions also plays a critical role in determining whether it is an attribute-poor scenario, and this is the reason why we adopt the ratio of the available number of node attributes to establish different levels of attribute richness. We have also added the associated explanation in Section 4.3 and Section 5 to make the discussion of our paper more comprehensive.

Reviewer comment (Experiments):
The concatenation experiment is conducted only on one dataset (DBLPFull). Are the results on other datasets consistent with this?

Author response:
We thank the reviewer for pointing this out. We would like to note that for the results reported in this paper, we also have similar observations on other adopted datasets. In fact, we have presented the rank of representations learned by GNNs in Figure 9 (Appendix) and the rank of representations learned by shallow methods in Figure 11 (Appendix). The rank of the concatenation of two representation matrices will be at least the rank of the maximum rank of the two matrices. Noticing that in all datasets, the representations learned by shallow methods are full-rank. Therefore, the tendency curve of the concatenated representation will either reach a minimum of effective dimension ratio of 0.5 (similar to the results presented in Figure 8) or a value of effective dimension ratio better than 0.5 (i.e., greater than 0.5). We have also modified Section 5 for further clarification.

Reviewer comment (Presentation):
Do I understand correctly that GNN w/o Agg (Figure 6) does not use any graph structure?

Author response:
Yes. For GNN w/o Agg, we remove all message-passing operations.

Reviewer comment (Writing):
There are some typos throughout the text:  
- Page 2: "the most graph popular representation"  
- Page 2: footnote is placed before the punctuation mark  
- Page 3: (line 2, line 3): "output" -> "outputs"  
- Page 3: "There have been various of GNNs"  
- Section 4.2: "We hypothesis that"  
- Section 4.2: "methods preserves"  
- Section 4.2, page 6: in the definition of matrices Z, C, F, the matrix Z repeats twice (same typo in appendix, page 17)  
- Page 9: "respeectively"  
- Page 20: "the available node attributes becomes"

Author response:
We thank the reviewer for pointing this out. We have revised them throughout this paper. Thanks again for your efforts to make this paper more enjoyable to read!

Reviewer comment (Novelty):
Novelty is not good. The findings that GNNs face some challenges when we do not have enough attribute information (e.g.[1][2]) and when we have heterophilic data (e.g.[3]) are already identified by other works.

Author response:
**Authors**: We agree with the reviewer that (1) existing works such as [1, 2] have pointed out that GNNs face challenges when we lack attribute information, and (2) GNNs also face challenges on heterophilic graphs [3]. However, we note that the main focus of this paper is to perform a comprehensive comparison between GNNs and shallow embedding methods. Especially, to the best of our knowledge, no existing work has comprehensively explored the strengths and weaknesses of GNNs compared with shallow methods with a unified view. Therefore, a comprehensive analysis is still needed to have an in-depth analysis of GNNs. For example, it is valuable to point out that dimensional collapse solely happens in GNNs instead of traditional shallow methods, and such a conclusion has not been revealed by any other existing works.  

[1] Wei, J., Goyal, M., Durrett, G., & Dillig, I. (2020). Lambdanet: Probabilistic type inference using graph neural networks. arXiv preprint arXiv:2005.02161.  

[2] Sun, Z., Zhang, W., Mou, L., Zhu, Q., Xiong, Y., & Zhang, L. (2022, June). Generalized equivariance and preferential labeling for gnn node classification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 8, pp. 8395-8403).  

[3] Zhu, J., Yan, Y., Zhao, L., Heimann, M., Akoglu, L., & Koutra, D. (2020). Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in neural information processing systems, 33, 7793-7804.