Paper: VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connections (ICLR-2024)

Reviewer comment (Novelty):
The novelty of this work is not high, as it mainly combines random walk with attention. Tokenize the graph to sequence use random walk is common used.

Author response:
- Second, we would like to mention that we (I) first analyzed the standard for graph tokenization with the theory derivation (i.e., Thm 3.1 and Table 1) and then (II) invented a virtual node based pagerank for graph tokenization with theoretical and empirical evaluation, in order to reduce the quadratic complexity of graph transformer for node classification in different settings like homophilous and heterophilous graphs.

Reviewer comment (Reproducibility):
The code is not provided, its reproducibility is unclear.

Author response:
* First, the code will be released upon the publication of the paper.

Reviewer comment (Novelty):
The novelty of this work is not high, as it mainly combines random walk with attention. Tokenize the graph to sequence use random walk is common used.

Author response:
- Second, we would like to mention that we (I) first analyzed the standard for graph tokenization with the theory derivation (i.e., Thm 3.1 and Table 1) and then (II) invented a virtual node based pagerank for graph tokenization with theoretical and empirical evaluation, in order to reduce the quadratic complexity of graph transformer for node classification in different settings like homophilous and heterophilous graphs.

Reviewer comment (Reproducibility):
The code is not provided, its reproducibility is unclear.

Author response:
* First, the code will be released upon the publication of the paper.

Reviewer comment (Novelty):
Several recent studies on designing graph Transformers with node sampling or node clustering are ignored.

Author response:
Thanks very much for your review. Your raised concerns are addressed as follows.

* First, all the suggested literature is added and discussed in the revised paper.

Reviewer comment (Evaluation):
Experimental results are inefficient in demonstrating the merits of the proposed method.

Author response:
* Third, in addition to the theoretical computational complexity analysis, the empirical efficiency comparison is conducted and updated. This running time question is also shared by Reviewer jsyZ. Therefore, on the largest dataset Amazon2M, we did the efficiency comparison. The eigendecompostion is based on DGL library. DGL itself is a Python library, but its underlying computations may involve languages like C and C++, and high-performance linear algebra libraries like SciPy and Fortran to ensure efficient numerical computations. Even without strong support, our PageRank sampling realized by Python (with parallel computing as theoretically designed in the paper) is slightly faster. Moreover, the reason why the content-based virtual node sampling is faster (i.e., ~620 v.s. ~409) is that the number of content-based virtual super nodes depends on the number of labels, which is usually less than the number of structure-based virtual super nodes. Hence, its PPR convergence is faster.

Table 3. Time Consumption on Amazon2M Dataset (added in Appendix 6.5)
| Method | Time (s) |
|--------|----------|
| **Eigendecomposition (by DGL library)** | $\sim$682 |
| **Virtual Structure Node based Parallel PageRank Sampling for Each Node (by Python)** | $\sim$620 |
| **Virtual Content Node based Parallel PageRank Sampling for Each Node (by Python)** | $\sim$409 |

Reviewer comment (Novelty):
The proposed method belongs to the line of designing scalable graph Transformers via node sampling. Several necessary studies [1,2,3] on this research topic should be cited and discussed in the paper. [1] and [3] leverage various node sampling strategies to obtain the token list for each node, where a super node-based strategy is also adopted in [3] to preserve the global information. [2] leverages the graph partition-based strategy to reduce the training cost of the Transformer model. These researches are highly related to the proposed method, especially [1] and [3].

Author response:
Thanks very much for your review. Your raised concerns are addressed as follows.

* First, all the suggested literature is added and discussed in the revised paper.

Reviewer comment (Experiments):
The results on heterophilous graph datasets are inefficient in demonstrating that the proposed method can handle graphs with heterophily. Important baselines [4,5,6] and datasets [7,8] are not considered in the experiment. In addition, [8] has revealed the drawbacks of the Squirrel dataset. Hence, experiments on heterophilous graph datasets need to be reorganized to support the claim of handling heterophily property.

Author response:
Thanks very much for your review. Your raised concerns are addressed as follows.

- Second, we add the benchmark large-scale heterophilous datasets [7] and include a comprehensive comparison with more baselines. The table is shown below. In short, our proposed method can still outperform the competitors. More details are included in the revised paper and colored blue in Appendix 6.4.

Table 1. Node Classification Accuracy on Large-Scale Heterophilous Graph Dataset (Without Minibatching)
|                               | arXiv-year                 |
|-------------------------------|---------------------------|
| **Edge Homophily**            | 0.22                      |
| **#Nodes**                    | 169,343                   |
| **#Edges**                    | 1,166,243                 |
| **#Features**                 | 128                       |
| **#Classes**                  | 5                         |
| **MLP**                       | 36.70 $\pm$ 0.21         |
| **L Prop 1-hop**              | 43.42 $\pm$ 0.17         |
| **L Prop 2-hop**              | 46.07 $\pm$ 0.15         |
| **LINK**                      | 53.97 $\pm$ 0.18         |
| **SGC 1-hop**                 | 32.83 $\pm$ 0.13         |
| **SGC 2-hop**                 | 32.27 $\pm$ 0.06         |
| **C\&S 1-hop**                | 44.51 $\pm$ 0.16         |
| **C\&S 2-hop**                | 49.78 $\pm$ 0.26         |
| **GCN**                       | 46.02 $\pm$ 0.26         |
| **GAT**                       | 46.05 $\pm$ 0.51         |
| **GCNJK**                     | 46.28 $\pm$ 0.29         |
| **GATJK**                     | 45.80 $\pm$ 0.72         |
| **APPNP**                     | 38.15 $\pm$ 0.26         |
| **H$_{2}$GCN**                | 49.09 $\pm$ 0.10         |
| **MixHop**                    | 51.81 $\pm$ 0.17         |
| **GPR-GNN**                   | 45.07 $\pm$ 0.21         |
| **GCNII**                     | 47.21 $\pm$ 0.28         |
| **WRGAT**                     | OOM                       |
| **GGCN**                      | OOM                       |
| **ACM-GCN**                   | 47.37 $\pm$ 0.59         |
| **LINKX (Minibatch)**         | 53.74 $\pm$ 0.27         |
| **GloGNN**                    | 54.68 $\pm$ 0.34         |
| **GloGNN++**                  | 54.79 $\pm$ 0.25         |
| **VCR-Graphormer**            | 54.15 $\pm$ 0.09         |

Table 2. Node Classification Accuracy on Large-Scale Heterophilous Graph Dataset (With Minibatching)
|                               | arXiv-year                |
|-------------------------------|---------------------------|
| **Edge Homophily**            | 0.22                      |
| **#Nodes**                    | 169,343                   |
| **#Edges**                    | 1,166,243                 |
| **#Features**                 | 128                       |
| **#Classes**                  | 5                         |
| **MLP Minibatch**             | 36.89 $\pm$ 0.11         |
| **LINK Minibatch**            | 53.76 $\pm$ 0.28         |
| **GCNJK-Cluster**             | 44.05 $\pm$ 0.11         |
| **GCNJK-SAINT-Node**          | 44.30 $\pm$ 0.22         |
| **GCNJK-SAINT-RW**            | 47.40 $\pm$ 0.17         |
| **MixHop-Cluster**            | 48.41 $\pm$ 0.31         |
| **MixHop-SAINT-Node**         | 44.84 $\pm$ 0.18         |
| **MixHop-SAINT-RW**           | 50.55 $\pm$ 0.20         |
| **LINKX Minibatch**           | 53.74 $\pm$ 0.27         |
| **VCR-Graphormer**            | 54.15 $\pm$ 0.09         |

Reviewer comment (Experiments):
The authors have highlighted that efficiency is one of the important metrics for graph tokenization methods, so a necessary experiment is required to support this claim.

Author response:
* Third, in addition to the theoretical computational complexity analysis, the empirical efficiency comparison is conducted and updated. This running time question is also shared by Reviewer jsyZ. Therefore, on the largest dataset Amazon2M, we did the efficiency comparison. The eigendecompostion is based on DGL library. DGL itself is a Python library, but its underlying computations may involve languages like C and C++, and high-performance linear algebra libraries like SciPy and Fortran to ensure efficient numerical computations. Even without strong support, our PageRank sampling realized by Python (with parallel computing as theoretically designed in the paper) is slightly faster. Moreover, the reason why the content-based virtual node sampling is faster (i.e., ~620 v.s. ~409) is that the number of content-based virtual super nodes depends on the number of labels, which is usually less than the number of structure-based virtual super nodes. Hence, its PPR convergence is faster.

Table 3. Time Consumption on Amazon2M Dataset (added in Appendix 6.5)
| Method | Time (s) |
|--------|----------|
| **Eigendecomposition (by DGL library)** | $\sim$682 |
| **Virtual Structure Node based Parallel PageRank Sampling for Each Node (by Python)** | $\sim$620 |
| **Virtual Content Node based Parallel PageRank Sampling for Each Node (by Python)** | $\sim$409 |

Reviewer comment (Reproducibility):
Clarification is needed on whether the sampling node set of the third component contains super nodes. If the answer is yes, how are the features of super nodes initialized?

Author response:
* Fourth, no matter structure-based virtual super nodes or content-based virtual super nodes, they do not participate in the sampling set. Therefore, their features are initialized as empty vectors and not updated. Their role is to decrease the pairwise distance to further extend the narrow receptive field.

Reviewer comment (Experiments):
The inclusion of additional graph learning tasks (edge/graph classification) would further establish the validity/generality of the token list preparation approach proposed.

Author response:
- Second, so far, we have the experiments for node classification but in different settings like homophilous and heterophilous datasets. Table 4 and Table 5 are the experiments on the homophilous datasets. Figure 2 and the newly added Table 6&7 are for heterophilous datasets. Extending to more general applications like link prediction and graph classification with graph transformers is a very interesting direction, and we will explore it in the future.

Reviewer comment (Writing):
Page 3: share the exactly same -> share exactly the same

Author response:
* Third, your pointed-out minor typos are corrected in the updated paper.

Reviewer comment (Writing):
Page 7: eigendecompostion -> eigendecomposition

Author response:
* Third, your pointed-out minor typos are corrected in the updated paper.

Reviewer comment (Writing):
Page 9: strcuture -> structure

Author response:
* Third, your pointed-out minor typos are corrected in the updated paper.

Reviewer comment (Writing):
Page 9: Targeting node-level tasks like graph classification -> ? (graph classification is not a node-level task)

Author response:
* Third, your pointed-out minor typos are corrected in the updated paper.

Reviewer comment (Writing):
The paper is hard to follow. Abuse of notations in Eq 3.1, 3.2, and 3.3 might be confusing for readers. Please formally define the operations or use well-known operations. Is {.} a set or a list? Please proper set-builder notations. Please put l (l-th step) to the name of the variable, r_u? Is the concatenation operator applied to both scalars and vectors? The cardinality of T_u in Eq 3.3 is 4 when you note it like this. Plus, it would be nice if the authors explained what insights we can see in a series of theorems 3.2 and 3.3.

Author response:
Thanks for your actionable suggestions. They are addressed as follows.

* First, the suggestions regarding some symbols and notation are addressed in the updated paper and marked blue. Moreover, the general insights from Thm 3.2 show the probability mass transfer from faded ordinary nodes to brightened ordinary nodes, which further illustrates that previous hardly-reachable nodes are obtaining more weight to extend the narrow receptive field. The general insights from Thm 3.3 state the computational complexity of the proposed graph transformer, which is less than the quadratic attention complexity of most graph transformers.

Reviewer comment (Evaluation):
The eigendecomposition is not a core component in NAGphormer. NAGphormer without structural encoding has shown no bad results. We can choose other cheap structural encodings rather than eigendecomposition, for example, PPR vectors as the authors say. Thus, the superiority of the proposed method against NAGphormer (e.g., cubic complexity) should be rewritten focusing on the core components (Hop2Token).

Author response:
Second, thanks for the reminder of the role of eigendecomposition in NAGphormer. We have updated Table 1 in the paper and made the necessary explanation in the context marked blue.

Reviewer comment (Experiments):
The performance increase in homophilous and large-scale datasets is marginal. VCR-Graformer's additional modules do not seem to be effective for these datasets. Instead, the proposed method is effective for heterophilous datasets. However, these datasets are small-scale thus, only evaluating small parts of this model (that targets large-scale graphs). It would be nice if the authors can use large-scale heterophilous datasets [Lim, Derek, et al.]. In addition, experiments to evaluate modeling long-range dependency (the third component) are not conducted.

Author response:
Third, according to the suggestion, we test the performance on large-scale datasets [Lim et al, 2021]. The experiments are shown below. In short, our proposed method can still outperform the competitors. More details are included in the paper and marked blue in Appendix 6.4.

Table 1. Node Classification Accuracy on Large-Scale Heterophilous Graph Dataset (Without Minibatching)
|                               | arXiv-year                 |
|-------------------------------|---------------------------|
| **Edge Homophily**            | 0.22                      |
| **#Nodes**                    | 169,343                   |
| **#Edges**                    | 1,166,243                 |
| **#Features**                 | 128                       |
| **#Classes**                  | 5                         |
| **MLP**                       | 36.70 $\pm$ 0.21         |
| **L Prop 1-hop**              | 43.42 $\pm$ 0.17         |
| **L Prop 2-hop**              | 46.07 $\pm$ 0.15         |
| **LINK**                      | 53.97 $\pm$ 0.18         |
| **SGC 1-hop**                 | 32.83 $\pm$ 0.13         |
| **SGC 2-hop**                 | 32.27 $\pm$ 0.06         |
| **C\&S 1-hop**                | 44.51 $\pm$ 0.16         |
| **C\&S 2-hop**                | 49.78 $\pm$ 0.26         |
| **GCN**                       | 46.02 $\pm$ 0.26         |
| **GAT**                       | 46.05 $\pm$ 0.51         |
| **GCNJK**                     | 46.28 $\pm$ 0.29         |
| **GATJK**                     | 45.80 $\pm$ 0.72         |
| **APPNP**                     | 38.15 $\pm$ 0.26         |
| **H$_{2}$GCN**                | 49.09 $\pm$ 0.10         |
| **MixHop**                    | 51.81 $\pm$ 0.17         |
| **GPR-GNN**                   | 45.07 $\pm$ 0.21         |
| **GCNII**                     | 47.21 $\pm$ 0.28         |
| **WRGAT**                     | OOM                       |
| **GGCN**                      | OOM                       |
| **ACM-GCN**                   | 47.37 $\pm$ 0.59         |
| **LINKX (Minibatch)**         | 53.74 $\pm$ 0.27         |
| **GloGNN**                    | 54.68 $\pm$ 0.34         |
| **GloGNN++**                  | 54.79 $\pm$ 0.25         |
| **VCR-Graphormer**            | 54.15 $\pm$ 0.09         |

Table 2. Node Classification Accuracy on Large-Scale Heterophilous Graph Dataset (With Minibatching)
|                               | arXiv-year                |
|-------------------------------|---------------------------|
| **Edge Homophily**            | 0.22                      |
| **#Nodes**                    | 169,343                   |
| **#Edges**                    | 1,166,243                 |
| **#Features**                 | 128                       |
| **#Classes**                  | 5                         |
| **MLP Minibatch**             | 36.89 $\pm$ 0.11         |
| **LINK Minibatch**            | 53.76 $\pm$ 0.28         |
| **GCNJK-Cluster**             | 44.05 $\pm$ 0.11         |
| **GCNJK-SAINT-Node**          | 44.30 $\pm$ 0.22         |
| **GCNJK-SAINT-RW**            | 47.40 $\pm$ 0.17         |
| **MixHop-Cluster**            | 48.41 $\pm$ 0.31         |
| **MixHop-SAINT-Node**         | 44.84 $\pm$ 0.18         |
| **MixHop-SAINT-RW**           | 50.55 $\pm$ 0.20         |
| **LINKX Minibatch**           | 53.74 $\pm$ 0.27         |
| **VCR-Graphormer**            | 54.15 $\pm$ 0.09         |

Reviewer comment (Experiments):
Runtime evaluation on efficiency is required. Although the time complexity is decreased in theory, the actual computations can be increased by METIS and PPR on three types of graphs.

Author response:
* Fourth, the efficiency comparison is conducted and updated. On the largest dataset Amazon2M, we did the efficiency comparison. The eigendecompostion is based on DGL library. DGL itself is a Python library, but its underlying computations may involve languages like C and C++, and high-performance linear algebra libraries like SciPy and Fortran to ensure efficient numerical computations. Even without strong support, our PageRank sampling realized by Python (with parallel computing as theoretically designed in the paper) is slightly faster. Moreover, the reason why the content-based virtual node sampling is faster (i.e., ~620 v.s. ~409) is that the number of content-based virtual super nodes depends on the number of labels, which is usually less than the number of structure-based virtual super nodes. Hence, its PPR convergence is faster. The below running time comparison contains the METIS partitioning time realized by DGL.

Table 3. Time Consumption on Amazon2M Dataset (added in Appendix 6.5)

| Method | Time (s) |
|--------|----------|
| **Eigendecomposition (by DGL library)** | $\sim$682 |
| **Virtual Structure Node based Parallel PageRank Sampling for Each Node (by Python)** | $\sim$620 |
| **Virtual Content Node based Parallel PageRank Sampling for Each Node (by Python)** | $\sim$409 |