Paper: Towards Better Evaluation of GNN Expressiveness with BREC Dataset (ICLR-2024)

Reviewer comment (Theory):
Why do the authors adopt cosine similarity instead of other contrastive loss? Will this loss be possible to introduce any spurious correlation leading to biased results?

Author response:
We appreciate the reviewer's meticulous examination of our evaluation methodology.Our choice of cosine similarity in the contrastive learning setting is notably simplified compared to common scenarios where batches of samples need to be separated. In our case, the objective is to distinguish only one sample from another, eliminating the necessity for certain properties, particularly the control of differences in one type. Unlike typical contrastive learning, which aims to maintain alignment and uniformity, and often introduces additional terms to regulate the distance between samples in one batch, such considerations are rendered unnecessary in our context. Consequently, we adhere to the simplest method to streamline the evaluation process.

Reviewer comment (Evaluation):
What about the relationship about the performance on the proposed synthetic datasets and real-world datasets? In other words, is the model that shows to more expressive in the benchmark performing better in real-world tasks? It would be better to provide more fine-grained analyses.

Author response:
We thank the reviewer for underscoring the significance of practical utility concerning expressiveness. Generally, graphs in chemical areas tend to have high expressiveness demand. There exist some structures that standard MPNNs can hardly recognize. For example, the benzene ring (6-cycle) is a common and essential molecule component that most simple GNNs (including all MPNNs) cannot recognize. It is inevitable those GNNs might learn some statistical patterns correlated to benzene rings, but they fundamentally fail to understand the structure and fail to make predictions causally. In real-world datasets, expressive GNNs also perform remarkably in molecular datasets. ZINC [1] is a widely used molecular property regression benchmark. Currently, the 2 SOTA models are GRIT [2] and N$^2$-GNN [3], both focusing on expressiveness.

We collect our tested models' performance on ZINC:

| Model | BREC(all) | BREC(CFI Graph) | ZINC↓ |
| --- | --- | --- | --- |
| OSAN | 148 | 5 | 0.155 |
| Graphormer | 79 | 10 | 0.122 |
| DS-GNN | 222 | 16 | 0.116 |
| GSN | 254 | 0 | 0.115 |
| DE+NGNN | 231 | 21 | 0.111 |
| DSS-GNN | 221 | 15 | 0.097 |
| GNN-AK | 222 | 15 | 0.093 |
| KP-GNN | 275 | 11 | 0.093 |
| SUN | 223 | 13 | 0.083 |
| I$^2$-GNN | 281 | 21 | 0.083 |
| PPGN | 233 | 23 | 0.079 |
| KC-SetGNN | 211 | 1 | 0.075 |
| SSWL_P | 248 | 38 | 0.070 |
| N$^2$-GNN | 287 | 27 | 0.059 |


Observably, N$^2$-GNN and SSWL_P outperform other models on the ZINC dataset. Both models exhibit commendable performance on BREC, particularly on CFI Graph, known for its substantial size and asymmetric composition, necessitating the model to discover graph pattern differences in its layers iteratively. The ZINC dataset, with its regression target "constrained solubility," incorporating water-octanol partition coefficient, synthetic accessibility score, and the number of cycles with more than six atoms, requires structure learning from a molecular accessibility perspective. Expressiveness, therefore, plays a pivotal role in enhancing model performance in the ZINC dataset.

References

[1] Benchmarking Graph Neural Networks https://arxiv.org/abs/2003.00982

[2] Graph Inductive Biases in Transformers without Message Passing http://arxiv.org/abs/2305.17589

[3] Towards Arbitrarily Expressive GNNs in O(n2) Space by Rethinking Folklore Weisfeiler-Lehman https://arxiv.org/abs/2306.03266