Paper: Topology-Informed Graph Transformer (ICLR-2024)

Reviewer comment (Theory):
A graph is vertex-biconnected if it is connected and does not have any cut vertex. The definition of vertex biconnected in this paper is wrong.

Author response:
We thank the reviewer for pointing out the issue. We added a condition that the graph G is a connected graph. Nevertheless, we would like to note that the given definition of biconnectivity of graphs, assuming G is connected, is correct. The vertex v and edge e of the graph G such that the number of components of G \ {v} and G \ {e} is greater than 1 correspond to a cut vertex and a cut edge of the graph G.

Reviewer comment (Theory):
The case used in Appendix A.2 for the proof of Theorem 3.2 is too particularized, so that the generalization of the theory and the proof cannot be guaranteed.

Author response:
Theorem 3.2 shows that there exists a pair of graphs such that TIGT can distinguish whereas 3-WL cannot distinguish. The theorem does not claim that all graphs not distinguishable by 3-WL can be distinguished by TIGT. Hence, the current proof of Theorem 3.2 in the appendix of the manuscript is correct, because it presents a particula pair of graphs that 3-WL cannot distinguish whereas TIGT can.

Reviewer comment (Novelty):
Can you describe the main innovations of your method compared to previous graph transformer methods?

Author response:
One prominent novelty of this work comes from obtaining an optimal balance among encapsulating the merits of Cy2C-GNNs in incorporating topological invariants of graph datasets, utilizing the advantages of Graph Transformers in effectively analyzing node and edge features of graph datasets, and sustaining the expressive power of GNNs even under increases in number of hidden layers. Our newly added implementations demonstrate that TIGT successfully implements the optimal balance of the aforementioned three conditions, whereas other GNNS or transformer architectures may not effectively satisfy all three conditions simultaneously.

Reviewer comment (Novelty):
The theoretical proof of Theorem 3.1 in Appendix A.1 is highly similar to the reference [1], can you specify the innovation of the theory in this paper?

Author response:
The theoretical innovation of the paper focuses on the capability to incorporate cyclic substructures of graphs to graph transformers, and demonstrate how cyclic substructures are related to geometric properties of graphs analyzed from previous studies. We note that Cy2C-GNN does not incorporate cyclic substructure to graph transformers. As such, Theorem 3.1 and 3.2 demonstrates that cyclic substructures can be also incorporated into graph transformer architecture, the proof of which utilizes the theoretical results from Choi et al. (2023). The manuscript also proves new theoretical results which relates cyclic substructures of graphs with previously studied innate properties of graphs (graph biconnectivity), and analyzes limitations of previously studied techniques which utilize positional encoding of graph datasets. In particular, Theorem 3.3 utilizes Euler characteristics of graphs to show that biconnectivity of graphs can be detected from analyzing cyclic substructures of graphs. Theorem 3.4 utilizes geometric ergodicity of Markov chains (i.e. that the rate that a probability distribution converges to a stationary distribution of a Markov chain is comparable to the rate that a geometric sequence converges to a limit) to show potential limitations in attempting to distinguish geometric properties of graphs without directly utilizing the cyclic substructures of graphs.

Reviewer comment (Novelty):
Originality/Novelty: The main theoretical results (Theorem 3.1 and Theorem 3.2) are mainly based on previous work (Choi et al., 2023).

Author response:
While we utilized the results from Choi et al. (2023) to prove the theoretical effectiveness in incorporating cyclic structures of graphs, one prominent novelty of this work comes from obtaining an optimal balance among encapsulating the merits of Cy2C-GNNs in incorporating topological invariants of graph datasets, utilizing the advantages of Graph Transformers in effectively analyzing node and edge features of graph datasets, and sustaining the expressive power of GNNs even under increases in number of hidden layers. Our newly added implementations demonstrate that TIGT successfully implements the optimal balance of the aforementioned three conditions. Furthermore, we also added new theoretical results which relates cyclic substructures of graphs with previously studied innate properties of graphs (graph biconnectivity), and analyzes limitations of previously studied techniques which utilize positional encoding of graph datasets.

Reviewer comment (Theory):
The assumptions made in Theorem 3.3, which requires two graphs to have the same number of nodes and edges, and Theorem 3.4, which assumes that all graphs have the same number of nodes with degree $d$, may have limitations in practical applications.

Author response:
The conditions on pairs of graphs indicated in Theorems 3.3 and 3.4 correspond to pairs of graphs that conventional GNN may not be able to distinguish as non-isomorphic pairs. In fact, if it is the case that any pairs of two graphs does not satisfy the conditions in Theorems 3.3 and 3.4, then conventional GNNs are able to distinguish two graphs as non-isomorphic. As an example, suppose G and H are two graphs whose number of nodes are different but the number of edges are identical. Then there exists a pair of nodes v of G and w of H such that the degree of v and degree of w are not equal to each other. This implies that the universal covers of G and H are not isomorphic, in particular at the nodes of the universal covers of G and H which, respectively, project to nodes v and w. Theorem 3.3 shows that biconnectivity of graphs can be detected from analyzing cyclic substructures of graphs, and Theorem 3.4 shows potential limitations in attempting to distinguish geometric properties of graphs without directly utilizing the cyclic substructures of graphs.

Reviewer comment (Evaluation):
It would be valuable if the authors could discuss the presence of cyclic structures in graph datasets and its impact on the proposed architecture's performance, whether it is missing or not.

Author response:
As the reviewer suggested, in Table 5 of the updated manuscript, we added a summary of cyclic substructures inherent in the graph datasets utilized in our experiments.

Reviewer comment (Experiments):
Ablation study: The authors have provided an ablation study for key components of the method. However, the topological positional embedding layer, the main contribution of this paper, appears to have a marginal effect.

Author response:
In Tables 1 through 4, we added new implementations of graph classifications and graph regressions by utilizing GATs and Cy2C-GNNs. The empirical results obtained in these tables demonstrate that TIGT outperforms both techniques in both classification and regression experiments. Unlike Cy2C-GNNs, TIGT effectively incorporates clique adjacency matrices, global attention layers, and graph information layers, the architectural design of which contributes to outperformance in graph classifications and regressions.

Reviewer comment (Evaluation):
Although the authors have provided an analysis of the computational complexity for the proposed method, they have not provided empirical results regarding running time compared to baselines. I would suggest the authors also measure the running time of the method and other baselines.

Author response:
In Table 7, we added specifications of hyperparameters and computation time taken in classifying graph datasets from utilizing TIGT and baseline techniques.

Reviewer comment (Experiments):
Baselines: Some baselines on graph transformer are either missing or not adequately discussed:  
1. Kong, Kezhi, et al. "GOAT: A Global Transformer on Large-scale Graphs." ICML 2023.  
2. Zhang, Zaixi, et al. "Hierarchical graph transformer with adaptive node sampling." NeurIPS 2022.

Author response:
We thank the reviewer for suggesting new references. We realized that the new references mostly focus on node classification of graph datasets, the results of which are difficult to make direct comparisons to TIGT which focuses primarily on graph classifications. Nevertheless, we added the new references to the introduction of the manuscript.

Reviewer comment (Novelty):
**Novelty.** This work is merely a simple adoption of Cy2C-GNNs with multi-head attention to package that as a graph transformer. The authors present the same layer, except for edge features E^{l-1}, as two different layers in Section 3.1 and Section 3.2. They are called Topological positional embedding layer and Dual-path MPNNs, respectively. After getting node features, then some multi-head attention-based graph transformer.

Author response:
One prominent novelty of this work comes from obtaining an optimal balance among encapsulating the merits of Cy2C-GNNs in incorporating topological invariants of graph datasets, utilizing the advantages of Graph Transformers in effectively analyzing node and edge features of graph datasets, and sustaining the expressive power of GNNs even under increases in number of hidden layers. Our newly added implementations demonstrate that TIGT successfully implements the optimal balance of the aforementioned three conditions, whereas other GNNS or transformer architectures may not effectively satisfy all three conditions simultaneously.

Reviewer comment (Theory):
**Scalability.** The authors discuss the scalability of the proposed method regarding the number of nodes, edges, and edges in cyclic subgraphs. The method has a quadratic complexity in terms of the number of nodes. This indicates that the method does not scale well, and it might be difficult to apply it to large-scale graphs.

Author response:
In Table 7, we added a comparison of the pre-processing computation time, computation time per epoch, and the number of hyperparameters utilized in classifying graph datasets by using Cy2C-GNNs, GPS+LapPE+RWSE, GRIT+RRWP, and TIGT. In Table 4, we also added the empiricial results obtained from Cy2C-GNNs with increased number of hyperparameters. While TIGT exhibits slightly increased computational costs and hyperparameters because it is comprised of various architectural components, we observe that TIGT exhibits meaningful performance boosts in graph classifications and regressions.

Reviewer comment (Reproducibility):
Please provide more implementation details of Cy2-GNN-1 and compare its computational cost and model parameters. Is it fair to compare between TIGT and Cy2-GNN-1.

Author response:
In Table 7, we added a comparison of the pre-processing computation time, computation time per epoch, and the number of hyperparameters utilized in classifying graph datasets by using Cy2C-GNNs, GPS+LapPE+RWSE, GRIT+RRWP, and TIGT. In Table 4, we also added the empiricial results obtained from Cy2C-GNNs with increased number of hyperparameters. While TIGT exhibits slightly increased computational costs and hyperparameters because it is comprised of various architectural components, we observe that TIGT exhibits meaningful performance boosts in graph classifications and regressions.