Paper: Understanding Heterophily for Graph Neural Networks (ICLR-2024)

Reviewer comment (Novelty):
Two related works need to be discussed [1,2].

Author response:
**Response to Comment 1:** Thank you for your reminder about these two recent related works, i.e., [1,2]. Both of them, as well as [3] (mentioned in our paper), analyze the heterophily problems of GNNs based on the CSBM models. [3] presents a preliminary result of GCN regarding the impact of different p and q on the two-block CSBMs, while [1] delves into the results when utilizing low-pass/high-pass GNNs. [2] evaluates the performance of GNNs on graphs containing nodes with homophily and heterophily patterns simultaneously. However, these efforts only focus on the simple binary node classifications, which is not general enough to analyze diverse heterophily patterns. Our work analyzes the impact of complicated heterophily patterns in the multi-class classifications, which has not been explored to the best of our knowledge. According to your suggestion, we have compared these two related works in Section 2 (Paragraph: Contextual Stochastic Block Models) in our revised manuscript.

Reviewer comment (Novelty):
"Its primary objective is to accurately categorize samples into their respective classes while distinguishing them from samples belonging to other classes." I think this is the same as the statement about intra- and inter-class node distinguishability in [1]. What is the difference and advantage of your "ideal Bayes classifier" over the optimal Bayes classifier in [1]?

Author response:
**Response to Comment 2:** The "ideal Bayes classifier" in our paper is the same as the "optimal Bayes classifier" in [1]. It is worth noting that it is the commonly utilized classifier in analyzing the behavior of GNNs based on random graph data models, e.g., [1,2,4,5], due to its simplicity and effectiveness in predicting the performance score on the generated data. Therefore, we utilize it to construct our analytical framework and further state that our theoretical results based on the Bayes classifier can be achievable in practice, as detailed in Appendix D.

Reviewer comment (Experiments):
The authors should consider elaborating on the experimental section, providing more detailed results, methodologies, and potential pitfalls or challenges. The benchmarks used in https://arxiv.org/abs/2302.11640 are suggested.

Author response:
**Response to Comment 2:** Thank you for your comment. We note that in the initial version of the manuscript, we provided comprehensive details regarding the experimental settings, results, and analysis for both synthetic data and real-world data in Appendix B and Appendix C. Still, according to your suggestion, we have employed Amazon-ratings and Workers in [5] in the revised manuscript. Additionally, we have also employed two large-scale heterophilious benchmarks, i.e., Arxiv-year and Snap-patents in [6], in the revised manuscript, to verify our theory. Arxiv-year has 169,343 nodes and 1,166,243 edges, while Snap-patents has 2,923,922 nodes and 13,975,791 edges.

As detailed in Appendix C.3 (in the revised manuscript), the results on these datasets further validate the effectiveness of our method. Specifically, according to their separability gains, calculated by our method, the Workers dataset possesses a good heterophily pattern, while the other three possess a mixed heterophily pattern. Besides, for these datasets, the differences in the confusion matrix between GCN and MLP align with the corresponding separability gains. These observations, especially in large-scale real-world datasets, further validate the effectiveness of our theory.

*[5] A Critical Look at Evaluation of GNNs Under Heterophily: Are We Really Making Progress? In ICLR, 2023.*

*[6] New Benchmarks for Learning on Non-Homophilous Graphs. In WWW Workshop, 2021.*

Reviewer comment (Experiments):
The paper lacks comparisons with existing methods. Previous methods [1,2] have provided analysis on heterophily from the perspective of node degree and neighborhood distribution. It would be beneficial to include discussions and comparisons from the standpoint of assumptions, graph generation, and results.

Author response:
**Response to Comment 1:** Thank you for your reminder about these two related works [1,2]. Accordingly, we have discussed them in our revised manuscript.

[1] utilizes the relative node degree and homophily ratio to predict the performance of the binary node classification task. However, neither the homophily ratio nor the binary classification task is sufficient to analyze the impact of complicated heterophily patterns. Our work focuses on more general cases and analyzes the impact of complicated heterophily patterns in the multi-class node classifications, which have not been explored to the best of our knowledge. Besides, it seems that [2] is the preprint version of [3], which had already been discussed in Section 1 (Introduction) in the previous manuscript. According to your suggestion, we have added the comparison with [1] in Section 1 (Introduction) in the initial version of the manuscript.

*[1] Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks. In ICDM, 2022.*

*[2] Is Heterophily A Real Nightmare For Graph Neural Networks To Do Node Classification?*

*[3] Revisiting heterophily for graph neural networks. In NeurIPS, 2022.*

Reviewer comment (Evaluation):
The paper does not provide suggestions for learning on heterophilous graphs. The analysis of heterophilous patterns in relation to model performance is provided, but it would be helpful to offer some guidance on model design when dealing with these graphs. Alternatively, the paper could explain why existing methods succeed on these graphs.

Author response:
**Response to Comment 2:** Thank you for your suggestion. As stated in the initial version of our manuscript, our work introduces a fresh perspective on heterophily and over-smoothing, coupled with a novel GNN analytical framework, which may further boost the development of innovative methods. We did not provide specific suggestion for learning methods on heterophilious graphs in the initial version. According to your suggestion, we have bridged our theoretical results to the model design of heterophily-specific GNNs, in the revised manuscript (Appendix F). Several possible suggestions and insights are as follows.

Firstly, according to Theorem 2, different GNN aggregators may successfully classify different subsets of nodes. Therefore, when learning on graphs with heterophily, it may be beneficial to adaptively select specific aggregators for individual nodes, e.g. [3,4]. Besides, this result can also explain why the combination of ego- and neighbor-embeddings can establish a stable and competitive baseline across diverse heterophily patterns [5,6].

Secondly, according to Theorem 3, the inconsistency in neighborhood distribution poses a potential challenge when handling heterophilious graphs. Therefore, it may be beneficial to mitigate the influence of topological noise within each class, probably through the design of an attention scheme.

Thirdly, according to Theorem 4, GNN with varying numbers of layers may successfully classify different subsets of nodes. Therefore, it may be beneficial to explore information from high-order neighborhoods, e.g., [5]. 

Besides, we believe that our theoretical results may provide additional inspiration to the community for designing heterophily-specific GNNs, enabling them to successfully handle graphs with diverse heterophily patterns.

*[3] Revisiting heterophily for graph neural networks. In NeurIPS, 2022.*

*[4] Learnable graph convolutional attention networks. In ICLR, 2023.*

*[5] Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs. In NIPS, 2020.*

*[6] A critical look at the evaluation of gnns under heterophily: are we really making progress? In ICLR, 2023.*

Reviewer comment (Experiments):
The paper could benefit from evaluations on larger-scale graphs. In section 5, the evaluations on the HSBM model are too simplistic to demonstrate its effectiveness, and it would be advantageous to increase the graph size.

Author response:
**Response to Comment 3:** Thank you for your suggestion. Accordingly, we have added additional large-scale datasets, including synthetic data and real-world data, in the revised manuscript, to validate the effectiveness of our theoretical results.


For the synthetic data, we have employed a series of large-scale synthetic graphs generated from our HSBM model in the revised manuscript (Appendix B.3). Similarly, we adopt the settings in Sec. 5.1, except for assigning the number of nodes as 100,000. Since the averaged node degree for each class is still 25, the expected number of edges in these large-scale graphs is 2,500,000. The results are provided in Fig. 10 in the revised manuscript, revealing that the results of the large-scale synthetic graphs are similar to those in Fig. 2(a), where the number of nodes is only 1000. This observation further validates the effectiveness of our theory.

For the real-world data, we have utilized two multi-class large-scale heterophilious graphs, i.e., Arxiv-year and Snap-patents [7], in the revised manuscript. Arxiv-year possesses 169,343 nodes and 1,166,243 edges, while Snap-patents possesses 2,923,922 nodes and 13,975,791 edges. As detailed in Appendix C.3 (in the revised manuscript), the results on these datasets further
validate the effectiveness of our method. Specifically, according to their separability gains, calculated by our method, they both exhibit a mixed heterophily pattern. Their differences in the confusion matrix between GCN and MLP align with the corresponding separability gains. This observation in the large-scale real-world datasets further validates the effectiveness of our theory.

*[7] New Benchmarks for Learning on Non-Homophilous Graphs. In WWW Workshop, 2021.*