Paper: A Fast and Effective Alternative to Graph Transformers (ICLR-2024)

Reviewer comment (Novelty):
The innovation is not strong enough, which diminishes the significance of the paper. Essentially, the authors replaced the multi-head self-attention module in the original Transformer with a global convolutional module, and then they claim their proposal is to improve efficiency. However, the necessity of this replacement needs to be considered, and it appears to be of limited significance. The primary issue lies in the absence of self-attention mechanism, resulting in a diminished capability to capture long-range dependencies. Experimental results on large datasets, such as PCQM and COCO, indicate that the model's performance is inferior to other Graph Transformer methods.

Author response:
>@W1 " However, the necessity of this replacement needs to be considered, and it appears to be of limited significance. The primary issue lies in the absence of self-attention mechanism, resulting in a diminished capability to capture long-range dependencies."
@W1 "Experimental results on large datasets, such as PCQM and COCO, indicate that the model's performance is inferior to other Graph Transformer methods."
@W2 "And the results provided to demonstrate the 'effectiveness' of the proposed method in capturing long-distance dependencies, as shown in Tables 1, 2, and 4, may not offer sufficiently strong evidence for its superior performance."
@W3 "Firstly, the motivation seems to enhance GT, but the authors care a lot about capturing long-range dependencies, which I have illustrated in last point, the results are not impressive enough."
@W3 "If the authors want to show the improvement of the enhanced GT in effectiveness, the results are not competitive."

We understand that the reviewer's main concern lies around understanding capability of GECO as a replacement for self-attention.

First and foremost, we emphasize that if the concern is to make a direct comparison between self-attention and GECO without using heuristics, one should compare the GECO and GraphGPS rows for a fair comparison in Tables 1, 2 and 3. Next, in the table below, we summarize (1) relative improvements w.r.t. GraphGPS's self attention and (2) relative improvements w.r.t. best GT baselines including orthogonal approaches (with available results).

We highlight empirical evidence showing that replacing self-attention with GECO does not diminish performance on most datasets; in fact, it often leads to improvements:

- **GECO has superior predictive quality on 8 out of 10 datasets w.r.t. GraphGPS with self-attention (Table 1, 2 and 3).**  Specifically, on the first five columns (long-range benchmark datasets), GECO achieves relative improvements of 12.53%, 6.31%, 1.44%, and 5.77%, with a -2.20% decrease in only on COCO-SP. For datasets in columns 6-10, GECO achieves significant relative improvements of by up to 10.36% on 4 datasets, with only a -0.41% decrease only on ogbg-ppa. 


- **GECO achieves superior quality to Dense GTs utilizing self attention (Graphormer/Graphormer-sample/GraphGPS) on 12 out of 14 datasets (Table 1, 2, 3 and 4).**  Please see table below and also refer to previous discussion and its table. Empirical evidence further supports that GECO does not have diminishing capability as a replacement GT's self-attention; in fact, it leads to improvements.

- **GECO has superior predictive quality on 9 out of 14 datasets w.r.t. across available GT Baselines including orthotogonal ones(Table 1, 2, 3 and 4).**  Please see below.


| Model           | PascalVOC-SP | COCO-SP | Peptides-func| Peptides-struct| PCQM-Contact  |ogbg-molhiv  | ogbg-molpcba  | ogbg-ppa| ogbg-code2 | PCQM4Mv2 | Flickr | Reddit | Yelp | Arxiv |
|-----------------|-------------------------|--------------------|--------------------|------------------------|---------------------|-----------------------|------------------------------|----------------------|------------------------|------------------------|--------------------|--------------------|------------------------|------------------------|
|| F1 score ↑ | F1 score ↑ | AP ↑ |  MAE ↓ | MRR ↑ |AUROC ↑ |  Avg. Precision ↑ |Accuracy ↑ | F1 score ↑ | MAE ↓  |Accuracy ↑ |Accuracy ↑|Accuracy ↑|Accuracy ↑|
| GraphGPS             | 0.3748 ± 0.0109         | 0.3412 ± 0.0044    | 0.6535 ± 0.0041    | 0.2500 ± 0.0005        | 0.3337 ± 0.0006     |0.7880 ± 0.0101       | 0.2907 ± 0.0028              | 0.8015 ± 0.0033      | 0.1894 ± 0.0024        | 0.0938     | OOM | OOM | OOM | OOM |
| Best GT  (Model Name)  | 0.3975 ± 0.0037 (Exphormer)         | 0.3455 ± 0.0009 (Exphormer)    | 0.6535 ± 0.0041 (GraphGPS)    | 0.2481 ± 0.0007 (Exphormer)       | 0.3637 ± 0.0020 (Exphormer)     |  0.7880 ± 0.0101 (GraphGPS)       |  0.2907 ± 0.0028 (GraphGPS)              | 0.8015 ± 0.0033 (GraphGPS)      | 0.1937 ± 0.0028 (K-Subtree SAT)      | 0.0858 (GraphGPS)   | 54.12 ± 0.51 (HSGT) | NA | 63.47 ± 0.45 (HSGT) | 72.58 ± 0.31 (HSGT) |
| GECO (Ours)    | 0.4210 ± 0.0080         | 0.3320 ± 0.0032    | 0.6975 ± 0.0025    | 0.2464 ± 0.0009        | 0.3526 ± 0.0016     | 0.7980 ± 0.0200       | 0.2961 ± 0.0008              | 0.7982 ± 0.0042      | 0.1915 ± 0.002         | 0.08413     | 55.55 ± 0.25 | 96.65 ± 0.05 | 63.18 ± 0.59 | 73.10 ± 0.24 |
| Relative Improvement w.r.t. GraphGPS with self-attention (%)|  12.53%    |  -2.20%  | 6.31% | 1.44%   | 5.78%  | 1.27% |  1.85%   | -0.41% |  1.11%  |  10.36%  | NA | NA | NA | NA |
| Relative Improvement w.r.t. Best GT Baseline (%)|  5.91%    |  -3.91%  | 6.31% | 0.69%   | -3.05%  | 1.27% |  1.85%   | -0.41% |  -1.36%  |  10.36%  | 2.64% | NA | -0.46 | %0.72 |

Please refer to the *Dataset Clarification* on the general discussion regards to definition of large graphs in our context.

Reviewer comment (Experiments):
Lacking experimental results to verify “fast” of the proposed method. Specifically, there is no emphasis in the experimental results, no parameter complexity analysis, no comparison of computing resource consumption or computing time. These are fundamental experiments in verifying “fast” of a certain method. And the results provided to demonstrate the 'effectiveness' of the proposed method in capturing long-distance dependencies, as shown in Tables 1, 2, and 4, may not offer sufficiently strong evidence for its superior performance. Overall, the title of this paper is ambitious and likely to capture attention with insufficient innovative approach, even though the authors claimed “they are the first to”.

Author response:
> @W2 "Lacking experimental results to verify “fast” of the proposed method. Specifically, there is no emphasis in the experimental results, no parameter complexity analysis, no comparison of computing resource consumption or computing time. These are fundamental experiments in verifying “fast” of a certain method." 
@W2 "Overall, the title of this paper is ambitious and likely to capture attention with insufficient innovative approach, even though the authors claimed “they are the first to”.
@Q3 "I am well aware that the comparison of computation complexity (theoretically) among several models including GECO. But what about the experimental verification?"

We have taken this concern to heart, recognizing its significance, particularly given the title of our paper, "fast". Please see the general discussion and revised manuscript.

Additionally, we imposed the same parameter budget constraints as previous research did for the datasets listed in Tables 1, 2, and 3. We included these details in the revised manuscript. Table 3 demonstrates that GECO outperforms the GraphGPS self-attention variant by a significant margin, using 8 times fewer parameters on PCQM4Mv2. When both models use the same number of parameters, GECO achieves a 12.53% relative improvement over GraphGPS. Furthermore, GECO's parameter usage is comparable to that of some traditional GNN architectures.

> @Q1 "What exact problem the authors want to solve? And how you directly verified that the problem is well solved, with what metric/way?"

Our primary motivation is to replace computationally intensive self-attention mechanism within graph transformers with a more efficient operator, while not sacrificing the prediction quality. To this end, we designed a compact layer combining local propagation and global convolutions. Our evaluation includes 3 steps to verify our approach:

1. **Verify competitive quality on commonly used Graph Transformer Datasets**: Please see discussions and tables above.

2. **Verify scalability and quality to larger node prediction datasets**: Please see discussions and tables above.

3. **Verify Runtime Efficiency**: In our revised version, we have included runtime efficiency benchmarks. We appreciate reviewers' feedbacks for encouraging us to include this study.


> @Q2 "How to balance the trade-off between fast and efficiency? Why is your method the best?"

There are multiple ways to enhance the scalability and efficiency of Graph Transformers, such as coarsening (HSGT [5]), expander graphs (Exphormer [6]) or sampling strategies [5]. Our work, GECO, takes another direction by directly replacing the self-attention kernel. Unlike prior attempts that compromised predictive quality with self-attention approximations [4], GECO maintains robust performance while improving scalability. Moreover, its design is orthogonal with existing scalability methods like HSGT, offering a promising future research direction. Please also see the discussion above.

> @W1 "The innovation is not strong enough, which diminishes the significance of the paper."

Please refer to general discussion.

[1] A Generalization of Transformer Networks to Graphs, AAAI Workshop'21

[2] Representing Long-Range Context for Graph Neural Networks with Global Attention, NeurIPS'21

[3] Do transformers really perform badly for graph representation?, NeurIPS'21

[4] GraphGPS: General Powerful Scalable Graph Transformers, NeurIPS'22

[5] Hierarchical Transformer for Scalable Graph Learning, IJCAI'23

[6] EXPHORMER: Sparse Transformers for Graphs, ICML'23

[7] Inductive Representation Learning on Large Graphs, NeurIPS'17

[8] Fast Graph Representation Learning with PyTorch Geometric, ICLR Workshop'19, [Reddit Dataset Link](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Reddit.html#torch_geometric.datasets.Reddit)

Reviewer comment (Experiments):
The design, organization, and writing of this paper are not very clear. Firstly, the motivation seems to enhance GT, but the authors care a lot about capturing long-range dependencies, which, as illustrated in the last point, are not demonstrated with impressive enough results. If the authors want to show the outperformance of trade-off between capturing long-range dependencies and fast calculation/computation, there is a lack of comparison of baselines including those methods not using Transformers. If the authors want to show the improvement of the enhanced GT in effectiveness, the results are not competitive. The authors also need to refer to some recent studies such as “Hierarchical Transformer for Scalable Graph Learning”. Furthermore, the authors aim to illustrate that their proposed method is fast and has distinct difference from GraphGPS, but why GraphGPS is chosen as the main comparison is unclear and confusing.

Author response:
> @W3 "Next, the authors aim to illustrate that their proposed method is fast and has distinct difference from GraphGPS. But why GraphGPS? It confuses me."

We understand your confusion. Let us clarify it by reviewing the literature chronologically.

1. Graph Transformer has been proposed, suggesting that attention should only be applied to connected nodes, consequently utilizing the adjacency matrix as an attention mask [1].
2. GraphTrans [2] and Graphormer [3] have been concurrently proposed, utilizing a self-attention mechanism, demonstrating that attention can be applied to the entire set of nodes and significantly improve predictive performance. Specifically, Graphormer achieved impressive results by incorporating several effective structural encodings.
3. Many GTs have been proposed, many of which are approaches proposing different structural/positional encodings, extracting additional substructure tokens, or applying attention to specific substructures centered around a node, which we detail in our manuscript. Many of these approaches are orthogonal approaches and they utilize self-multi-head attention as their kernel.
4. GraphGPS [4] has been proposed, combining GNN and attention modules side by side at each layer, offering recipes for off-the-shelf GNN, attention and positional/structural encodings combinations, and achieving state-of-the-art results across many graph datasets.
5. **We proposed GECO as a compact layer consisting of local propagation and global convolution model blocks.** As both GECO and GraphGPS utilizes local and global blocks, we clarify their distinction to address potential reader questions. Unlike previous methods, in GECO, we designed a compact layer comprising local and global modules without intermediate parameters and non-linearities, applying skip connections to the layer as a whole. Please refer to our response to R-XpCe @W1 for more details. In addition, our work can complement other GT approaches focusing on various positional/structural encodings, or clustering/coarsening/substructure-based strategies.

> @W3 "If the authors want to show the outperformance of trade-off between capturing long-range dependencies and fast calculation/computation, there is a lack of comparison of baselines including those methods not using Transformers"
@W3 "And I think the authors also need to refer to some recent studies such as “Hierarchical Transformer for Scalable Graph Learning”

We have added the Hierarchical Transformer for Scalable Graph Learning (HSGT) [5] along with additional baselines from [5] to our revised manuscript. Below, we summarize the comparison between the two works on common datasets:


| Model         | Flickr  | Yelp   | Arxiv   |
|---------------|:------------:|:--------:|:----------------:|
| Graphormer [5] | OOM | OOM | OOM | 
| Graphormer-SAMPLE [5] | 51.93 ± 0.21 | 60.01 ± 0.45  | 70.43 ± 0.20  | 
| SAN [5] |  OOM | OOM | OOM | 
| SAT [5] |  OOM | OOM | OOM | 
| GraphGPS (Transformer) |  OOM | OOM | OOM | 
| SAT-SAMPLE [5] | 50.48 ± 0.34 | 60.32 ± 0.65 | 68.20 ± 0.46  |
| Exphormer [6]| NA | NA | 72.44 ± 0.28 |
| ANS-GT [5] | NA | NA | 68.20 ± 0.46|
| HSGT [5] | 54.12 ± 0.51        | **63.47 ± 0.45**          | 72.58 ± 0.31             |
| GECO   | **55.55 ± 0.25**         | 63.18 ± 0.59        | **73.10 ± 0.24**           |

- **HSGT and GECO are orthogonal approaches.** HSGT's attention components could be swapped with GECO's for a hierarchical graph learning model using global convolutions, offering a potential future direction. GECO provides kernel-wise scalability in place of self-multi-head attention, while HSGT achieves structural scalability through graph partitioning and coarsening.
- **The comparison shows competitive performance:** GECO outperforms HSGT on Flickr and Arxiv, while HSGT surpasses GECO on Yelp.
- **GECO vs Dense GTs (Graphormer/Graphormer-Sample/GraphGPS)**: While Graphormer and GraphGPS, which rely on self-attention, encounters out-of-memory issues in every scenario, GECO effectively scales to these datasets. Furthermore, Graphormer with sampling falls short in achieving competitive quality when compared to GECO across all datasets.

Additionally, Reddit dataset comes in multiple variants. In our study, we used the variant originally proposed in [7] available in PyG [8]. However, we noticed that HSGT uses a different variant. Below, we summarize the difference.


| Dataset       |      N   (Ours)    |    M  (Ours)   |    N  (HSGT [5])    |    M (HSGT [5])  | 
|---------------|:------------:|:--------:|:----------------:|:----------------:|
| Reddit        |  233K     |114.6 M|  233K |  11.6M |

Reviewer comment (Evaluation):
What exact problem do the authors want to solve? And how do they directly verify that the problem is well solved, with what metric or method?

Author response:
> @Q1 "What exact problem the authors want to solve? And how you directly verified that the problem is well solved, with what metric/way?"

Our primary motivation is to replace computationally intensive self-attention mechanism within graph transformers with a more efficient operator, while not sacrificing the prediction quality. To this end, we designed a compact layer combining local propagation and global convolutions. Our evaluation includes 3 steps to verify our approach:

1. **Verify competitive quality on commonly used Graph Transformer Datasets**: Please see discussions and tables above.

2. **Verify scalability and quality to larger node prediction datasets**: Please see discussions and tables above.

3. **Verify Runtime Efficiency**: In our revised version, we have included runtime efficiency benchmarks. We appreciate reviewers' feedbacks for encouraging us to include this study.

Reviewer comment (Evaluation):
How to balance the trade-off between fast and efficiency? Why is the proposed method the best?

Author response:
> @Q2 "How to balance the trade-off between fast and efficiency? Why is your method the best?"

There are multiple ways to enhance the scalability and efficiency of Graph Transformers, such as coarsening (HSGT [5]), expander graphs (Exphormer [6]) or sampling strategies [5]. Our work, GECO, takes another direction by directly replacing the self-attention kernel. Unlike prior attempts that compromised predictive quality with self-attention approximations [4], GECO maintains robust performance while improving scalability. Moreover, its design is orthogonal with existing scalability methods like HSGT, offering a promising future research direction. Please also see the discussion above.

Reviewer comment (Experiments):
The reviewer is aware that the comparison of computation complexity (theoretically) among several models including GECO is presented, but experimental verification of this computational complexity comparison is missing.

Author response:
@Q3 "I am well aware that the comparison of computation complexity (theoretically) among several models including GECO. But what about the experimental verification?"

We have taken this concern to heart, recognizing its significance, particularly given the title of our paper, "fast". Please see the general discussion and revised manuscript.

Additionally, we imposed the same parameter budget constraints as previous research did for the datasets listed in Tables 1, 2, and 3. We included these details in the revised manuscript. Table 3 demonstrates that GECO outperforms the GraphGPS self-attention variant by a significant margin, using 8 times fewer parameters on PCQM4Mv2. When both models use the same number of parameters, GECO achieves a 12.53% relative improvement over GraphGPS. Furthermore, GECO's parameter usage is comparable to that of some traditional GNN architectures.

Reviewer comment (Theory):
The LCB module conducts neighborhood information propagation for each node. It directly utilizes the connectivity information via adjacency matrix. In the meantime, GECO is implicitly learning this ‘connectivity’ via the convolutional filters. Is there any information overlap here?

Author response:
Yes, there is a slight information overlap. LCB block tells us where to pay attention using adjacency matrix, however if adjacency lacks information due to missing/spurious edges, GCB compliments it by enabling information flow from the rest of the vertices.

The ablation study at Table 5 can be seen as an empirical support for our motivation to use LCB and GCB together. For clarity, we will use the relative rows of Table 5 below:

| Model         | PascalVOC-SP   | Peptides-func   | Peptides-struct   |
| ------------- | -------------- | --------------- | ----------------- |
|               | F1 score ↑     | AP ↑            | MAE ↓             |
| GECO (Conv-1)| 0.2752         | 0.6589          | 0.2587            |
| GECO (LCB)   | **0.3220**         | **0.6876**          | **0.2454**            |

Above, Conv-1 refers to an alternative approach in GECO where LCB is substituted with a 1D-Convolution having a filter size of 1, essentially replacing LCB with an identity function.  This scenario represents the case where we do not utilize the adjacency matrix and explicitly learn connectivity information using only GCB. Notably, when LCB is employed, we observe significant improvements in predictive performance across all datasets.

Reviewer comment (Novelty):
The technical contribution is limited. According to this survey [2], the proposed LCB module can be treated as the GNN-as-Auxiliary-Modules in the Alternatively form (Figure 1 in [2]). Additionally, the writing of this paper seems rushed. Many details are missing and hard to understand. For example, in Algorithm 1, line 3, what is $V_t \leftarrow (P)_t FFTConv(F_i, V)_t$. Actually, I found more details of this algorithm in Algorithm 3, page 8, [link](https://arxiv.org/pdf/2302.10866.pdf)[1]. The forward pass of GCB Operator is nearly identical to Hyena, which is not new.

Author response:
> @W1 "The technical contribution is limited. According to this survey [2], the proposed LCB module can be treated as the GNN-as-Auxiliary-Modules in the Alternatively form (Figure 1 in [2])."

Thanks for pointing out the related work which we included in the revised manuscripts.

Approaches combining of-the-shelf GNNs and attention mechanisms were already mentioned in our submission. In GraphTrans [3] (Before Trans type in [2]), GNN and Transformer blocks are applied sequentially, while GraphGPS [4] (Parallel type in [2]) employs them in parallel at each layer. It is worth noting that these models straightforwardly combine pre-existing GNN and Transformer models, resulting in separate parameters and intermediate non-linearities for each module, with skip connections applied independently.

**GECO does not precisely align with the taxonomy defined in [2].** In GECO, we did not just use LCB as an auxiliary module to Transformer. Instead, we designed a new compact layer comprising local and global blocks. We eliminated the intermediate non-linearities and parameters to reduce the overall number of parameters, simplifying the model. We applied skip connections to the entire GECO layer as a whole, rather than separately. These deliberate design choices distinguish GECO from the use of off-the-shelf methods.

Mesh Graphormer [5], the only related work categorized under 'Alternatively form' in [2], similarly diverges from GECO regarding intermediate non-linearities, parameters, and skip connections, as discussed above. It also differs in the order of local and global modules, with Mesh Graphormer placing attention before the GNN block, whereas in GECO, local follows global.


> @W2 "Additionally, the writing of this paper seems rushed. Many details are missing and hard to understand. For example, in Algorithm 1, line 3, what is V_t \leftarrow (P)_t FFTConv(F_i, V)_t."

In Section 3.3's last paragraph, we detail Algorithm 1's logic and aim to enhance clarity with pseudocode comments. The GCB operator conducts global convolutions sequentially for projections and filters, gating output with subsequent results. P_i is for projections, F_i for filters, and V for values in pseudocode comments. FFTConv represents FFT Convolutions. We added more clarifications to the pseudocode in the revised manuscript.

We have put genuine effort into creating a comprehensive manuscript that includes details on related work, background, pseudocode, implementation, and dataset information, with appropriate citations for further details. If there are particular details you would like us to provide, please let us know.

Reviewer comment (Theory):
The paper claims that the proposed model is "fast", and provides detailed time complexity analysis. Unfortunately, from the theoretical perspective, GECO has the same level complexity as Message-passing GNN $O(NlogN+M)$ and it can only surpass vanilla transformer when $M<<N^2$. Additionally, no experiments regarding the running time efficiency are presented.

Author response:
> @W4 "The paper claims that the proposed model is "fast", and provides detailed time complexity analysis.  Unfortunately, from the theoretical perspective, GECO has the same level complexity as Message-passing GNN  and it can only surpass vallia transformer when M << N^2"

**M << N^2 always holds for real-world datasets.** This property is the main reason why we need sparse computations to train GNNs. To put in more concrete form, we provide a table where we explicitly calculate M/N^2 for each dataset we use. Specifically, large node prediction, the graphs are even sparser. We also added these columns to the revised version of the manuscript.

| Dataset       |      N       |    M     |    M / N^2       |
|---------------|:------------:|:--------:|:----------------:|
| PCQM-Contact  |     30.1     |   61.0   | 6.79 x 10^-2      |
| Peptides-func |    150.9     |  307.3   | 1.36 x 10^-2      |
| Peptides-struct|   150.9     |  307.3   | 1.36 x 10^-2      |
| COCO-SP       |    476.9     | 2,693.7  | 1.20 x 10^-2      |
| PascalVOC-SP  |    479.4     | 2,710.5  | 1.20 x 10^-2      |
| PCQM4Mv2      |     14.1     |   14.6   | 7.25 x 10^-2      |
| Molhiv        |     25.5     |   27.5   | 4.29 x 10^-2      |
| Molpcba       |     26.0     |   28.1   | 4.13 x 10^-2      |
| Code2         |    125.2     |  124.2   | 1.59 x 10^-2      |
| PPA           |    243.4     | 2,266.1  | 3.25 x 10^-2      |
| Flickr        |   89,250     | 899,756  | 1.12 x 10^-4      |
| ogbn-arxiv    |  169,343     | 1,166,243| 3.97 x 10^-5      |
| Reddit        |  232,965     |114,615,892| 1.95 x 10^-6     |
| Yelp          |  716,847     |13,954,819| 2.26 x 10^-6      |

Additionally, while our approach entails higher complexity than MPNN, it exhibits near-linear complexity which is a significant advancement compared to graph transformers with dense attention, all the while having a robust predictive quality, as demonstrated through Table 1-4.

> @W4 "Additionally, no experiments regarding the running time efficiency are presented."

Thank you for your suggestion. Please see the general rebuttal above.