Paper: GraSP: Simple yet Effective Graph Similarity Predictions (ICLR-2024)

Reviewer comment (Presentation):
The presentation of the content, particularly in Section 3, lacks clarity and cohesiveness, making it challenging for readers to follow and understand the proposed methodology.

Author response:
**Response:** Thanks for the suggestion. We have enhanced the overview paragraph at the beginning of Section 3 to make it clear and cohesive.

Reviewer comment (Evaluation):
The paper posits inefficiency in contemporary cross-graph interaction techniques as a primary catalyst for the development of GRASP. However, the narrative lacks a coherent demonstration of how GRASP mitigates these inefficiencies. The empirical section, intended to validate the method's enhanced efficiency, does not decisively support this assertion. Specifically, the performance metrics juxtaposed with existing strategies such as GREED and ERIC suggest comparable efficiencies, an outcome that muddles the purported superiority of GRASP in this domain. The authors should consider a more nuanced exposition of the method's unique efficiencies, supplemented by robust experimental evidence, to substantiate claims of its advancement over current practices.

Author response:
**Response:** The efficiency of the proposed GraSP is due to the following designs. First, GraSP does not need the expensive cross-graph node-level interactions that are usually adopted in existing methods. Second, the positional encoding used in GraSP to enhance node features can be precomputed and reused for the efficiency of online inference. Third, all the technical components in GraSP are designed to make the complicated simple and effective for graph similarity predictions, resulting to efficient performance. By following the literature, we report the inference time per 10K pairs in *Figure 4* of the paper, where GraSP is the fastest method to predict graph similarity scores, which experimentally validates the efficiency of GraSP. 
As suggested, we have added the exposition above into Section 5.3 of the revised paper.

Reviewer comment (Novelty):
The authors assert that prevailing cross-graph interaction modules contribute significantly to computational overheads. However, the delineation of how their proposed GRASP framework, which ostensibly employs similar cross-graph interactions via NTN, innovates upon or diverges from traditional methodologies is ambiguous. This lack of clarity muddles the reader's understanding of any novel contributions the paper might be making in this specific aspect of the framework. It is imperative for the authors to elucidate the nuanced operational differences, if any, introduced by GRASP that ameliorate the time-costly nature of cross-graph interactions, distinctly setting their approach apart from conventional ones. This clarification could significantly enhance the perceived value and ingenuity of their methodology.

Author response:
**Response:** We clarify that in Eq. (7), NTN is applied to two graph embeddings $\mathbf{z}_1$ and $\mathbf{z}_2$ to get an interaction value, interaction$(\mathbf{z}_1, \mathbf{z}_2)$. The interaction value in Eq. (7) is essentially a multi-headed weighted cosine similarity score, which is efficient to compute. It is *different* from the cross-graph *node-level* interactions required in existing studies as reviewed in Section 6. Specifically, the interactions in existing studies refer to the expensive node-level matching or node-graph matching processes across graphs, which take quadratic time w.r.t. the number of nodes. 
As explained in the response to **W2** above, our GraSP does not need such expensive cross-graph nodel-level interactions.

Reviewer comment (Experiments):
The statistics of the data used for training, validation, and inference are not provided in the paper, which are important to understand the quality of results.

Author response:
**Response:** We split training, validation, and testing data with a ratio of 6:2:2 for all datasets and all methods, by following the same setting as SimGNN and GraphSim.

Reviewer comment (Reproducibility):
The MSE scores of baselines reported in the paper do not align with the existing literature. For example, GREED outperforms other baselines such as H2MN, SIMGNN in the literature but it is not reflected in this paper. What is the source of this discrepancy? Were all methods trained on error over GED or over the similarity score. The authors need to release the version of code they used benchmarking the baselines and the exact loss function so that reproducibility and any source of discrepancy can be properly analyzed.

Author response:
**Response:** The original papers of existing methods usually adopt *different* data split ratios. For example, GREED paper uses "100K query-target pairs for training and 10K pairs each for validation and test". In our paper, to be consistent across all methods, we split every dataset in the ratio of 6:2:2. A method can have different results with different data split ratios, which is a main source of discrepancy. 
In the original paper of GREED, three datasets AIDS, Linux, and IMDBMulti are used for GED predictions. In Table A of the response to **W1** above, the RMSE results of GREED on AIDS, Linux, IMDBMulti are on par with the original paper of GREED, which shows our reproducibility on baselines. Moreover, in **Table 1** of our paper, GREED outperforms H2MN by 10 out of 15 metrics  and SIMGNN by 12 out of 15 metrics on the three datasets, which again illustrates the reproducibility of our experiments. At Line 170 in src/trainer.py, we just provide users with the command-line options to train by either GED or similarity scores. As suggested, we have included the codes of all methods in the anonymous code repository.

Reviewer comment (Experiments):
The heat maps are provided for the AIDS700nef dataset where graphs are of small sizes. Heatmaps on datasets with larger graphs such as PTC give a better idea of the performance of Grasp and those need to be included.

Author response:
**Response:** As suggested, we have included the heatmaps of PTC, LINUX, and IMDBMulti in Figures 9,10,11 of Appendix A.5 in the revised paper. The new heatmaps also show that our method GraSP achieves low error, indicated by a light color.

Reviewer comment (Experiments):
It is not clear how Grasp generalizes for unseen graph sizes. Given that generating ground truth for graphs with larger sizes is expensive, it is interesting to see how Grasp performs when training is done with smaller graphs and testing is done on larger unseen graphs. This aspect needs to be compared with other state-of-the-art baselines such as GREED and H2MN.

Author response:
**Response:** As suggested, we conducted new experiments by following the experimental setting in GREED to test the generalization ability of GraSP on large unseen graphs for GED predictions, compared to GREED and H2MN. Specifically, we get GraSP-25 and GraSP-50 by training GraSP on graphs with node sizes up to 25 and 50, respectively, and then test on graph pairs in which the number of nodes in the query graph is in the range [25, 50]. Table B shows the results of PTC data. Observe that (i) the performance of all methods degrades for large query sizes in [25,50], compared with the entire set in [0,50], (ii) when trained using smaller graphs from 50 to 25 size, all methods also degrade, (iii) our method GraSP-25 (resp. GraSP-50) keeps the best performance than baselines GREED and H2MN under all these generalization settings.

**Table B.** Generalization on large unseen graphs with RMSE results to predict GED directly.

|     | Query Size in [0,50] | Query Size in [25,50] |
| --- | --- | --- |
| GraSP-50 | **5.418** | **6.104** |
| GREED-50 | 5.885 | 7.470 |
| H2MN-50  | 7.041 | 9.024 |
| GraSP-25 | **7.499** | **9.841** |
| GREED-25 | 8.341 | 10.204 |
| H2MN-25  | 8.965 | 10.956 |

Reviewer comment (Theory):
The estimated GED didn’t include costs for relabeling of edges. The authors didn’t mention how to extend this work to include edge substitution costs.

Author response:
**Response:** As suggested, we have added the explanation on how to extend our method to include the cost of relabeling edges in Appendix A.8. Specifically, one possible way is that, instead of the gate $\eta_{i,j}$ in Eq. (3) of the paper, with reference to [1], we can modify Eq.(3) to adopt an edge gate $e^{(\ell)}_{i,j}$ : 

$h_i^{(\ell)} = h_i^{(\ell - 1)} + {ReLU}(W_1 h_i^{(\ell - 1)}  + {\underset{j \in \mathcal{N}(i)}{\sum}} {e_{i,j}}^{(\ell)}\odot W_2h_j^{(\ell - 1)})$, where $e_{i,j}^{(\ell)}$ is the edge gate and can be defined as: 
$e_{i,j}^{(\ell)} = \sigma(e_{i,j}^{(\ell - 1)} + ReLU(A h_i^{(\ell - 1)} + B h_j^{(\ell - 1)} + Ce_{i,j}^{(\ell - 1)}))$, where $A,B,C \in \mathbb{R} ^ {2d \times 2d}$ and $e^{(0)}_{i,j}$ represents the input edge features.


[1] Vijay Prakash Dwivedi et al. Benchmarking Graph Neural Networks. Journal of Machine Learning Research, Volumn 24. 2023.

Reviewer comment (Novelty):
The novelty of GRASP is limited apart from using positional encoding. Grasp tackles the issue of cross-graph interactions, although it's important to recognize that Greed had already addressed this limitation before Grasp.

Author response:
**Response:** We acknowledge that the issue of cross-graph interactions has been studied in the literature. Our novelty lies in the different technical designs to mitigate this issue, compared with existing methods. In the proposed GraSP, for the first time, positional encoding is introduced for the task of graph similarity prediction to enhance node features for higher expressiveness; GraSP further employs a graph neural network with gating and residual connections, and utilizes a multi-scale pooling technique to generate meaningful representation for accurate graph similarity predictions. Our theoretical analysis shows that GraSP can exceed 1-WL test. Notably, GraSP is versatile to accurately predict GED and MCS, and achieves superior performance across various datasets.

Reviewer comment (Experiments):
It would be interesting to know what would be the alternative to the proposed combination of summation and attention pooling, e.g. what if a simple non-learning combination is used, i.e. $\alpha * z_{sum} + (1 - \alpha) * z_{att}$ where $\alpha$ is a hyperaprameter scalar. This will further highlight the importance of learnable combination of the two pooling methods.

Author response:
As suggested, we vary a non-learning scalar value $\alpha$ from 0 to 1, with MSE results on GED prediction in Table A below, where the last row shows GraSP with the proposed learnable combination. Observe that (i) the proposed learnable combination in GraSP achieves the best performance on all datasets; (ii) the non-learning combination needs different $\alpha$ scalar values on different datasets, which is tedious to tune. These observations highlight the importance of the proposed learnable combination of the poolings.

**Table A.** MSE ($\times 10^{-3}$) results on learnable combination v.s. non-learning combination.
| $\alpha$    | AIDS700nef | IMDBMulti | LINUX     | PTC       |
|-------------|------------|-----------|-----------|-----------|
| 0 (w/o sum) | 1.540      | 1.621     | 0.156     | 1.831     |
| 0.2         | *1.433*      | 1.020     | 0.136     | 1.847     |
| 0.4         | 1.590      | 1.478     | *0.134*     | *1.784*     |
| 0.6         | 1.594      | 1.572     | 0.142     | 1.812     |
| 0.8         | 1.505      | *0.986*     | 0.153     | 1.811     |
| 1 (w/o att) | 1.556      | 2.272     | 0.156     | 1.813     |
| learnable   | **0.987**  | **0.789** | **0.075** | **1.641** |

Reviewer comment (Novelty):
The overall novelty is limited, given the adoption of random walk positional encoding and combination of existing graph pooling are not firstly proposed in this paper.

Author response:
Compared with existing methods for graph similarity predictions, our novelty lies in the different technical designs to achieve superior performance for predicting GED and MCS. To our knowledge, we are the first to introduce positional encoding to the graph similarity predictions to node features for higher expressiveness. Furthermore, GraSP employs a graph neural network with gating and residual connections, and utilizes a multi-scale pooling technique to generate meaningful representation for accurate graph similarity predictions. Our theoretical analysis shows that GraSP can exceed 1-WL test. Notably, GraSP achieves superior performance across various datasets.

Reviewer comment (Evaluation):
Why the time complexity for computing the random walk positional encoding can be omitted? For each new graph pair, the inference time should include such positional encoding, since at inference time, the graph pair can be new and thus unseen by the model. Unless the authors assume a database-like setting, such overhead cannot be omitted.

Author response:
Note that the positional encoding is applied per graph, instead of per graph pair. During online inference, if a graph appears in multiple graph pairs, the positional encoding of the graph is computed only once, but used many times. Therefore, given a dataset of graphs, we can pre-compute and store the positional encoding of every graph. Then, during online inference, given a graph pair, we only need to retrieve the encodings of the two graphs in the pair and reuse them as the input of GraSP.  This explains why the time complexity of positional encoding can be omitted for inference time.