Paper: Rethinking the Effectiveness of Graph Classification Datasets in Benchmarks for Assessing GNNs (ICLR-2024)

Reviewer comment (Experiments):
The synthetic graph generator is based on the Erdos-Renyi (ER) graph model, which may not be practical for generating many real-world graphs. For instance, in fields like chemistry and pharmacy, the motif structure is crucial, and simple metrics such as node count and cycle count are not sufficient for graph classification.

Author response:
Our synthetic graph generator, exemplifying the application of Theorem 1, is designed to demonstrate the control of correlation between graph properties and labels. Theorem 1 is versatile and can be adapted to any graph generation process that specifies numerical graph properties, making it suitable for real-world applications.

Reviewer comment (Evaluation):
While the authors conducted a correlation analysis between graph statistics and class labels produced from the graph generator, the quality analysis for the generated graphs is not enough because the accuracy evaluations of graph classification are not conducted.

Author response:
Sorry for the confusion. We indeed provide the accuracy evaluations and performance gaps of these generated graphs in Section 4.2, specifically in Figure 6(b). This figure illustrates the relationship between property correlation and performance gaps. Furthermore, we demonstrate performance gaps ranging from 0% to 100%, associated with correlations from 0 to 1. This figure suggests the effectiveness of our controllable graph generation method.

Reviewer comment (Reproducibility):
The sources for the 14 datasets used in Section 4.3 are not provided.

Author response:
Sorry for that, this is a typo, it should be all 16 datasets, we have corrected it in the latest revison.

Reviewer comment (Novelty):
Can you provide some representative application fields and supporting literature where simple metrics like node count and cycle count are good enough to effectively estimate class labels with high accuracy?

Author response:
In our introduction, we reference studies [1,2,3] demonstrating that MLP-based baselines, which utilize metrics like average degree and clustering coefficients (cycle count=3), can perform comparably or even better than GNNs. Other research [4,5] emphasizes the importance of counting graph substructures, crucial in biology, chemistry, and social network analysis, particularly highlighting the role of cycle counts in chemical substructures. For more complex properties, such as graph biconnectivity, essential in fields like chemistry and social networks, a recent study [6] shows that popular GNN paradigms struggle with tasks efficiently handled by simpler algorithms. Node count, while basic, is implicitly included in metrics like average degree or clustering coefficients and shows high correlation with labels in datasets like REDDIT. Given that GNNs are at least as powerful as the 1-WL test, which easily distinguishes based on node count, GNNs should theoretically handle this metric effectively.

Reviewer comment (Experiments):
The reason for limiting the cycle count sequence to k = {3, 4, 5, 6} is unclear.

Author response:
This limit is informed by our finding that significant graph properties in social networks and molecular datasets, such as clustering coefficients (k=3) and common structures like benzene rings (k=6), are adequately represented within this range. Cycle counts larger than 6 are relatively rare in these datasets, hence their exclusion.

Reviewer comment (Evaluation):
Lack of recommendations to the practitioner (see below).

Author response:
Thank you for highlighting this important aspect! We have offered following disccussion in the latest revison to address your question:

While the choice of datasets should not solely depend on the effectiveness metric as proposed in our paper, this metric serves as a valuable indicator to understand the characteristics of these datasets. For instance, our methodology focuses on comparing a limited set of graph properties as baselines against GNN performance. However, there are additional graph properties that are crucial for accurate predictions.

Our study is to bridge the gap between theoretical understanding and practical application in the realm of graph classification datasets and is to provide a comprehensive empirical analysis that could inform and guide future theoretical explorations in this field.

As we have summarized in Section 5, exploring a broader range of properties and establishing a relationship between GNN expressiveness and these properties is essential to comprehending what GNNs truly learn from datasets. 

Therefore, for practitioners, our findings provide valuable insights. Specifically, we highlight that datasets with a high correlation between average degrees and labels may be too simplistic for GNNs, making it challenging to discern the additional learning and predictive capabilities of these networks.

In practical terms, this implies that when selecting datasets for training and evaluating GNNs, practitioners should consider the complexity and diversity of graph properties present. Datasets that offer a wider range of challenges will likely be more effective in distinguishing the unique strengths of GNN methodologies.

Reviewer comment (Writing):
There are many typos, which I suspect come from a rushed writing. Please have a second round of proofread before the rebuttal.

Author response:
We sincerely thank you for your thoughtful appreciation of our work. In the latest revision, we have addressed and corrected all typos as per your suggestions.

Reviewer comment (Experiments):
Do you have recommendations for GNN practitioners? e.g., which benchmarks should definitely be avoided when presenting a novel graph-based method?

Author response:
Thank you for highlighting this important aspect! We have offered following disccussion in the latest revison to address your question:

While the choice of datasets should not solely depend on the effectiveness metric as proposed in our paper, this metric serves as a valuable indicator to understand the characteristics of these datasets. For instance, our methodology focuses on comparing a limited set of graph properties as baselines against GNN performance. However, there are additional graph properties that are crucial for accurate predictions.

Our study is to bridge the gap between theoretical understanding and practical application in the realm of graph classification datasets and is to provide a comprehensive empirical analysis that could inform and guide future theoretical explorations in this field.

As we have summarized in Section 5, exploring a broader range of properties and establishing a relationship between GNN expressiveness and these properties is essential to comprehending what GNNs truly learn from datasets. 

Therefore, for practitioners, our findings provide valuable insights. Specifically, we highlight that datasets with a high correlation between average degrees and labels may be too simplistic for GNNs, making it challenging to discern the additional learning and predictive capabilities of these networks.

In practical terms, this implies that when selecting datasets for training and evaluating GNNs, practitioners should consider the complexity and diversity of graph properties present. Datasets that offer a wider range of challenges will likely be more effective in distinguishing the unique strengths of GNN methodologies.

Reviewer comment (Writing):
There are several grammatical errors in the paper. For example, "whether a GNN method has truly improved" should be "whether a GNN method has been truly improved".

Author response:
We have thoroughly reviewed the entire paper to correct these grammatical errors in the latest revison.

Reviewer comment (Presentation):
The authors do not provide a comprehensive figure to illustrate the overall assessment of the proposed strategy for graph classification methods. Although the idea is straightforward, such a figure can help readers capture the main objective for understanding.

Author response:
Thank you for pointing out this important aspect. In response to your valuable feedback, we have included a comprehensive figure in Appendix A8 of our latest revision. This figure aims to clarify our overall framework, enhancing the reader's understanding and facilitating the replication of our results. We appreciate your suggestion and hope that this addition effectively addresses your concern.

Reviewer comment (Theory):
The effectiveness definition in Eq. (2) seems not to be intuitive. The authors multiple two factors that consider both the changing proportion of the performance gap and the complexity. However, it remains unclear why we should use the product form instead of others. Also, it is unclear why the first component is normalized by |Y|-1 instead of |Y|.

Author response:
We apologize for any lack of clarity in our explanation and appreciate the opportunity to elaborate further. Here, we try to provide the intuitions, more detailed explanation and proof are provided in Appendix 5, with key properties discussed in Sec. 3.2. 

Briefly:

   1. We consider the worst-case scenario where accuracy equals random guessing, which is 1/|Y|, e.g., if |Y| = 4, then random guessing has accuracy 25%.
   2. The largest gap in performance is therefore 100% - 1/|Y|.
   3. The proportion $(R_{high} - R_{low})/ R_{low} \in [0, (1 - |Y|^{-1})/|Y|^{-1}] = [0, |Y| - 1]$.
   4. Thus, we normalize the proportion by $|Y| - 1$, such that the proportion ranges from 0 to 1.

Reviewer comment (Novelty):
Have the authors considered devising a method that can solve the limitations in existing graph classification methods?

Author response:
Our primary focus is on the effectiveness of benchmark datasets rather than directly improving GNN methods. That said, our findings can assist in assessing the fairness of new methods. Notable recent studies [1-5] have made significant strides in capturing complex structures, a crucial area where basic GCN and GIN models fall short. These works often employ synthetic datasets to test method effectiveness, underscoring the community's need for more robust real-world datasets.


References:

[1] You, Jiaxuan, Jonathan M. Gomes-Selman, Rex Ying, and Jure Leskovec. "Identity-aware graph neural networks." In Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 12, pp. 10737-10745. 2021.

[2] You, Jiaxuan, Rex Ying, and Jure Leskovec. "Position-aware graph neural networks." In International conference on machine learning, pp. 7134-7143. PMLR, 2019.

[3] Huang, Yinan, Xingang Peng, Jianzhu Ma, and Muhan Zhang. "Boosting the Cycle Counting Power of Graph Neural Networks with I$^2$-GNNs." arXiv preprint arXiv:2210.13978 (2022).

[4] Chen, Zhengdao, Lei Chen, Soledad Villar, and Joan Bruna. "Can graph neural networks count substructures?." Advances in neural information processing systems 33 (2020): 10383-10395.

[5] Zhang, Bohang, Shengjie Luo, Liwei Wang, and Di He. "Rethinking the expressive power of gnns via graph biconnectivity."  In ICLR. OpenReview.net, 2023.

Reviewer comment (Novelty):
In general, the novelty of the paper is a bit limited. The paper mainly concentrates on an empirical investigation of the effectiveness of existing benchmarking datasets for graph classification. While providing some insights on the datasets, there is a lack of theoretical understanding or support. This also makes it more suitable for a benchmark track such as NeuIPS benchmarking track.

Author response:
We first thank you for acknowledging our contributions in your comments: "Significance. This paper revisits some existing datasets or benchmarks and highlights the issue that some datasets fail to distinguish advancements in Graph-based models. This problem is critically important. Addressing it has the potential to significantly impact the broader community."

To the best of our knowledge, **our work is the first to thoroughly study and provide an explicit definition for the effectiveness of datasets in the graph learning area, which is an urgent issue that requires investigation.** Previous excellent works [1,2,3,4,5,6] have observed that the expressiveness of GNN is not fully exploited due to issues with dataset reliability, where even baseline models can achieve comparable performance. 

Therefore, we have meticulously studied this problem by introducing a novel definition of effectiveness. Moreover, Theorem 1 provides a powerful tool for controllable graph generation, facilitating future explorations in creating controllable synthetic datasets. Our extensive results on 16 common benchmarks in the field and synthetic datasets validate the effectiveness of this definition. Looking ahead, we believe our work can be further refined for greater generality and accuracy.

In addition, we acknowledge your concern about the perceived lack of theoretical support. However, it is important to emphasize that **the primary objective of our study is to bridge the gap between theoretical understanding and practical application in the realm of graph classification datasets. Our approach is to provide a comprehensive empirical analysis that could inform and guide future theoretical explorations in this field.**

Moreover, our work aligns well with the scope of ICLR, under the major area: datasets and benchmarks. Our research contributes to this by offering a detailed empirical analysis, highlighting the strengths and limitations of current benchmarking datasets, thereby setting the stage for more theoretical investigations.

We believe that our study, with its focus on empirical analysis, complements theoretical research and provides a necessary foundation for developing more robust and effective graph classification methods. The insights gained from our work can spur further theoretical research, contributing to a more comprehensive understanding of graph classification benchmarks.

Reviewer comment (Experiments):
For example, why GCN and GIN are chosen as the two methods for evaluating the benchmarks. The authors claim “GIN is spatial” and “GCN is spectral”, which is not informative enough. It would be better if the authors could provide more description and comparison between the two (or with other methods) and motivate their choices.

Author response:
As detailed in Section 2.1, we selected GIN and GCN for our study because these two Graph Neural Networks (GNNs) represent the most fundamental types in both spatial and spectral categories. They are widely used and demonstrate competitive performance in most applications [7,8]. These GNNs are also considered baseline models on the Open Graph Benchmark platform (OGB) [3]. Furthermore, GIN [9] is proven to be as powerful as the 1-WL test and is a quintessential example of message-passing neural networks. Meanwhile, GCN [10] is known for achieving competitive performance on small graphs with just 2 layers.

Reviewer comment (Experiments):
The choice of average degree as the input for the structure-dominated MLP model seems to be a bit ad-hoc without solid support. It is not very clear why the average degree is selected especially given that the authors investigate several other graph properties in later sections.

Author response:
Following the methodologies of previous works [1,2,3,4,5,6], we utilize degree information, without node attributes, as the input for MLPs, or employ the aggregation of neighboring features [3], which also implicitly includes degree information. Given that the average degree is a significant graph property in various applications, and considering that the learning paradigm of GNNs involves aggregating neighboring information followed by a pooling layer, GNNs can readily learn node degree information. Thus, the critical aspect of our study is to determine whether GNNs learn solely the average degree information or if they capture more complex structural information as well.

Reviewer comment (Theory):
It is not very clear how Theorem 1 describes or supports the graph generation process. In particular, the meaning of Eq. (3) is not clearly explained. It would be helpful if the authors could provide more details on this part. Furthermore, as generating synthesized graphs is a key component of the paper, it might be better if the authors provide more details on the graph generation model and process.

Author response:
Apologies for any confusion caused. Theorem 1 plays a crucial role in controllable graph generation, specifically with given correlation coefficients, as detailed in Appendix Algorithm 1. We hope the following explanation will help address your concerns:

Utilizing Theorem 1, we are able to efficiently generate the label sequence for the graph generation process in the following manner:


### Algorithm 1: Controllable dataset construction
**Input**: {$r_k$}$_{k=1}^{K}$, number of labels: $C$,  

 {$P_k$}$_{k=1}^K \sim N(\mu_k, \sigma_k)$ or $U(a_k, b_k)$;  

**Output**: Dataset $D$ with size $N$;

**for** k=1 to K do;

**Sample** $n_k \sim N(0, \sigma_k)$ or  $U(-\sqrt{3}, \sqrt{3})$ with size $N$;

$P_k = \mu_k + \sigma_k n_k$ or $\frac{a_k + b_k}{2} + \sqrt{\frac{b_k - a_k}{12}} n_k$;

**end**

**Calculate** $Y$ by the Equation (3);

$Y = \text{ROUND}(\text{NORM}(\mathcal{Y}) * C)$;

D = {(g_i, y_i)}$_{i=1}^N$, 

where each graph $g_i$ has properties {$P_k[i]$}$_{k=1}^K$,  and corresponding label $y_i = Y[i]$;

Reviewer comment (Evaluation):
In Observation 3, of Section 2.3, it claims “only REDDIT-B displayed a noteworthy performance gap, indicating a strong correlation between degree information and task labels”. Why this is the case, if there is a strong relation between average degree and task labels, should the performance gap be small as the baseline can also achieve good performance?

Author response:
Thank you for pointing out that issue. Indeed, there was a typographical error; it should state "weak correlation" instead of "strong correlation." In Section 4.3 and Figure 4, we illustrate the correlations between graph properties and labels. Specifically, we observed that REDDIT-B exhibits a lower correlation between the degree of nodes and labels compared to other properties. This suggests that using average degree as a baseline input is less effective than employing GNNs. In contrast, other social network datasets such as COLLAB and REDDIT-M show higher correlations, leading to better performance.

Reviewer comment (Reproducibility):
In section 4.1, how is the correlation between graph properties and class labels calculated? Can you provide any details in calculating this correlation? Also, how high should these correlation coefficients be to be considered high correlations?

Author response:
Thank you for your query regarding the calculation of the correlation. We have attempted to explain this in Section 4.2 and Theorem 1, where we mention the use of Pearson correlation for calculating the coefficients. We understand that these sections might not have been as clear as desired. To offer further clarity, we have provided a detailed step-by-step process in Algorithm 1, located in Appendix A2. This algorithm is designed to assist in applying Theorem 1 for generating graphs with specified correlation coefficients.

You raise an important point. Indeed, the absolute values of correlation coefficients differ across various datasets and graph properties. When we refer to 'high' or 'low' correlation, it is meant for comparative purposes within similar types of graph datasets, like those in social networks. The relationship between correlation coefficients and performance gaps can be verified using synthetic datasets, as demonstrated in Figure 6(b). In the context of real-world datasets, the correlation coefficient should be regarded as one of several relative indicators.