Paper: Graph Convolutions Enrich the Self-Attention in Transformers! (ICLR-2024)

Reviewer comment (Theory):
The time complexity of GFSA exceeds that of both single-head and multi-head self-attention. Specifically, in scenarios with a large number of tokens, the computation of $\mathbf{A}^{2}$ will demand a substantial amount of computational resources.

Author response:
**Q3. The time complexity of GFSA exceeds that of both single-head and multi-head self-attention. Specifically, in scenarios with a large number of tokens, the computation of  will demand a substantial amount of computational resources.**

The approximation for $\bar{A}^K$ with $A$ and $\bar{A}^2$ provides a simpler computation that can significantly reduce the required computational resources and time.  Nevertheless, the computational resources may still exist due to matrix multiplication that calculates $\bar{A}^2$. However, in Tables 18 to 23 in Appendix L, the increases of empirical training time are trivial in most cases. 

Also, to handle a large number of tokens, many studies have proposed Sparse Transformers [1,2,3,4] to reduce computational and memory complexities while maintaining performance by treating attention as a sparse graph. For future work, we hope to develop technical approaches to combine our model with sparse attention.


> [1] Manzil Zaheer et al. "Big bird: Transformers for longer sequences." NeurIPS, 2020
> 
> [2] Iz Beltagy, Matthew E. Peters, and Arman Cohan. "Longformer: The long-document transformer." arXiv preprint arXiv:2004.05150, 2020
> 
> [3]  Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. "Reformer: The Efficient Transformer." In International Conference on Learning Representations, 2020.
> 
> [4] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. "Fast transformers with clustered attention." NeurIPS, 2020.

Reviewer comment (Evaluation):
The bottleneck of standard Transformer in practice is its high time/space complexity O(N^2), the proposed method introduce high-order neighbor information aggregation introduces more computation/memory overheads, even with approximate computation.

Author response:
**Q1. The proposed method introduce high-order neighbor information aggregation introduces more computation/memory overheads, even with approximate computation.**

In Tables 18 to 23 in Appendix L, we report the training time of various methods with GFSA. In general, the training time of methods with GFSA is slightly longer than that of existing methods. For example, the Transformer for the automatic speech recognition task increases from 190.5 seconds to 191.6 seconds on Librispeech 100h dataset, as increases of only 1 second. Instead of computing higher-order polynomial terms, our GFSA approximates them, with only a small increase in runtime, which is not very significant.

Reviewer comment (Evaluation):
In section 4.6, the paper talked about the overhead of training time. It is better to add the comparison of inference time.

Author response:
**Q3. It is better to add the comparison of inference time?**

We appreciate the suggestion. In the tables below, we report a comparison of inference time between Transformers with and without GFSA to provide a more comprehensive analysis of the computational implications of GFSA. We included those results in our revised version.

Our inference time remains the same or has a minor increase as seen in text classification. For example, in the CoLA dataset, BERT$_{\text{BASE}}$ +GFSA only increases by 1 second, or in the MNLI dataset, it increases by 1.3 seconds. However, if the number of tokens exceeds 1000, such as GPT2, the time increases by about 5 seconds, as in WikiText-2. However, considering the performance increase, we believe this increase in inference time is trivial.



* Inference time on the text classification task


| Method | CoLA | SST-2 | MRPC | QQP | STS-B | MNLI-m/mm | QNLI | RTE |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| BERT$_{\text{BASE}}$ | 1.0s | 1.4s | 1.2s | 48.7s | 1.9s | 15.5s | 10.1s | 1.2s | 10.0s |
| BERT$_{\text{BASE}}$ + GFSA | 1.1s | 1.4s | 1.2s | 52.3s | 2.0s | 16.8s | 11.0s | 1.3s | 11.0s |
| ALBERT$_{\text{BASE}}$  | 1.1s | 1.6s | 1.4s | 58.4s | 2.2s | 18.4s | 12.1s | 1.3s | 12.0s |
| ALBERT$_{\text{BASE}}$  + GFSA | 1.2s | 1.7s | 1.4s | 62.1s | 2.3s | 19.7s | 13.1s | 1.4s | 13.0s |
| RoBERTa$_{\text{BASE}}$  | 1.0s | 1.4s | 1.1s | 47.0s | 1.9s | 15.0s | 9.9s | 1.2s | 10.0s |
| RoBERTa$_{\text{BASE}}$  + GFSA | 1.1s | 1.4s | 1.2s | 50.4s | 2.0s | 16.3s | 10.8s | 1.2s | 11.0s |

* Inference time on the language modeling task

| Method | PTB | WikiText-2  | WikiText-103 |
| -------- | -------- | -------- | -------- |
| GPT2        | 3.2s     | 7.4s     | 7.4s     |
| GPT2 + GFSA | 5.5s     | 12.9s     | 12.9s     |


* Inference time on the image classification task


| Method | #Layer | Inference Time|
| -------- | -------- | -------- |
| DeiT-S        | 12 | 52s |
| DeiT-S + GFSA | 12 | 53s |
| DeiT-S        | 24 | 68s |
| DeiT-S + GFSA | 24 | 69s |
| CaiT-S        | 24 | 105s|
| CaiT-S + GFSA | 24 | 107s|
| Swin-S        | 24 | 17s |
| Swin-S + GFSA | 24 | 17s |

* Inference time on the graph classification task

| Method | ZINC | PCQM4M  | PCQM4Mv2 |
| -------- | -------- | -------- | -------- |
| Graphormer        | 8s     | 99s     | 31s     |
| Graphormer + GFSA | 8s     | 117s     | 39s     |

* Inference time on the speech recognition task

| Method | LibriSpeech 100h | LibriSpeech 960h|
| -------- | -------- | -------- |
| Transformer         | 328.1s | 323.7s |
| Transformer + GFSA  | 329.5s | 343.3s |
| Branchformer        | 299.4s | 328.7s |
| Branchformer + GFSA | 305.5s | 354.1s |

* Inference time on the code defect prediction task


| Method   | Inference Time |
| -------- | -------- |
|RoBERTa             | 22.4s |
|RoBERTa + GFSA      | 23.9s |
|CodeBERT            | 23.8s |
|CodeBERT + GFSA     | 24.1s |
|PLBART      | 37.7s |
|PLBART + GFSA       | 39.3s |
|CodeT5-small        | 78.2s |
|CodeT5-small + GFSA | 82.5s |
|CodeT5-base         | 83.2s |
|CodeT5-base + GFSA  | 88.5s |

Reviewer comment (Reproducibility):
In Eq. (6), do we have any constraints over the coefficients w0, w1, wk? For example w0, w1, wk > 0? In standard Transformer, there is a residual connection in self-attention layer. What is the difference or connection between w0 and this residual connection? Are these learnable coefficients added to all layers? If yes, are they shared across different layers? Similar questions for multi-head attention, are they shared across different heads?

Author response:
**Q4. Are there any constraints on the coefficients $w_0$, $w_1$, $w_K$? For example $w_0, w_1, w_K > 0$? In a standard Transformer, there is residual connectivity in the self-attention layer. What is the difference or connection between $w_0$ and this residual connection?**

The coefficients $w_0$, $w_1$, and $w_K$ in Eq.(5) are all real numbers without constraints on their values.
Regarding the connection with the residual connection in the standard Transformer, the role of w0 in our proposed method is similar to the residual connection. The original residual connection adds the input vector $\mathbf{X}$ to the result of multi-head attention. However, $w_0$ in GFSA adjusts the weight of the value vector $\mathbf{X}\mathbf{W}_{val}$ in each head. Therefore, our GFSA has the advantage of being applicable without changing the internal structure of any Transformers except for the self-attention matrix.

---

**Q5. Are these learnable coefficients added to all layers? If yes, are they shared across different layers? Similar questions for multi-head attention, are they shared across different heads?**

Thank you for raising these questions. The learnable coefficients are applied to all layers in our proposed method. However, they are not shared across different layers. Each layer has its own set of coefficients. Similarly, for multi-head attention, the learnable coefficients are not shared across different heads. Each head has its own set of coefficients.

Reviewer comment (Novelty):
Missed references: 1) Graph Attention Networks, ICLR, 2017  2) Diffusion Improves Graph Learning, NIPS, 2019  3) Multi-hop Attention Graph Neural Networks, IJCAI 2021

Author response:
**Q6. Missed references**

Thanks for your suggestion. We have included the missed references you mentioned in our revised paper.

Reviewer comment (Theory):
Theorem 3.1 establishes an error bound for the Taylor approximation in Eqn 5. However, the error bound presented in Equation 9 relies on ||A-bar^2 - A-bar||, making this error bound meaningless since it lacks any bound for ||A-bar^2 - A-bar||.

Author response:
**Q3. the error bound presented in Eq. (7) relies on $||\bar{A}^2 - \bar{A}||$, making this error bound meaningless since it lacks any bound for $||\bar{A}^2 - \bar{A}||$.**

Thank you for pointing it out, we further develop the error bound in Eq. (8) as follows:

$E_K \leq 2K,$

where $K$ is the order of polynomial.

Since $||\bar{A}||\leq 1$ and $||\bar{A}^2||\leq ||\bar{A}||^2 \leq 1^2=1$, the upper bound of $E_k$ is becomes as follows:


$E_k \leq 2+(K-1)||\bar{A}^2-\bar{A}||\leq 2+(K-1)(||\bar{A}^2||+||\bar{A}||) \leq 2+(K-1)(1+1) = 2K.$

We have included this response in our revised version.

Reviewer comment (Theory):
When explaining the over-smoothing problem in transformers, the authors claim: “This problem is obvious to understand since Transformers’ aggregation methods for value vectors are simply weighted averages.” This argument is weak and vague.

Author response:
**Q5. When explaining over-smoothing problem in Transformers, the authors claim: “This problem is obvious to understand since Transformers’ aggregation methods for value vectors are simply weighted averages”. This argument is weak and vague.**

We apologize for the vagueness of the argument. We corrected the ambiguity in the sentence raised by the reviewer in the revised paper.

Reviewer comment (Evaluation):
The experimental results display inconsistencies. It is challenging to find the 3.3% improvement mentioned in the introduction for the image classification task, as it is not evident in Figure 1 and Table 3. Likewise, the claim of a 1.63% improvement for DeiT-S + GFSA over DeiT-S does not correspond with the results presented in Table 3. Furthermore, the alleged 6.23% improvement in the natural language understanding task cannot be observed when comparing Figure 1 and Table 5.

Author response:
**Q6. The experimental results display inconsistencies**

Thanks for pointing it out. We apologize for one error in the performance improvement percentages we wrote in the Introduction section. We also found that the notation of performance improvements in Table 3 could be confusing, so we modified the name of the rightmost column. We have uploaded the corrected version.

Reviewer comment (Novelty):
The paper's novelty and contribution are lacking. The concept of self-attention as a weighted graph has already been discussed in prior works [1, 2, 3, 4]. Although the authors introduce the idea of viewing self-attention as a graph filter, they fail to provide a solid explanation for how this perspective addresses the issue of over-smoothing in Transformers.

Author response:
**Q7. the concept of self-attention as a weighted graph has already been discussed in prior works [1, 2, 3, 4]**

We appreciate your evaluation and the concerns raised. In addition to the papers mentioned by the reviewer, research using GAT or self-attention (scaled-dot product attention) as an adjacency matrix for graphs is popular and active.

However, our study is different from the research line that experiments on graph tasks by learning the edge weights of the graph from attention. While the concept of self-attention as a weighted graph has been discussed in prior works, our paper extends this idea by introducing the perspective of viewing self-attention as a graph filter. From this perspective, we describe the similarities and differences in the papers mentioned by the reviewer:

- The paper [1], which uses a hierarchical fusion strategy, was inspired by JKNet [5] and used it as is for self-attention of Transformers. On the other hand, we design self-attention from the perspective of a graph filter in graph signal processing.
- GTN [2] focuses on creating a meta-path adjacency matrix to utilize the graph type of heterophilic nodes. The attention score used by GTN is the attention score for the candidate adjacency matrix when generating the meta-path adjacency matrix, so it is different from the Transformer's self-attention covered in our paper.
- Hypercube Transformer [3] belongs to the line of research on sparse attention. This model focuses on how dense self-attention should be to reduce complexity and at the same time maintain performance, and does not interpret self-attention from the perspective of a graph filter.
- GAT [4], the most popular graph neural network, also uses attention scores to consider it as a weighted graph. However, GAT's attention is Bahdanau et al. attention [6] and Vaswani et al. self-attention [7] is different. While GAT is a model for solving the problems of existing spectral and spatial GCNs, we solve the problem of Transformer's self-attention from the perspective of graph signal processing.

> [1] Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen Lee, and James T Kwok. Revisiting over-smoothing in BERT from the perspective of graph. ICLR, 2022. 
> 
> [2] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J. Kim. "Graph transformer networks." NeurIPS, 2019. 
> 
> [3] Yuxin Wang, Chu-Tak Lee, Qipeng Guo, Zhangyue Yin, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. "What dense graph do you need for self-attention?." ICML, 2022. 
> 
> [4] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. "Graph attention networks.". ICLR, 2018.
> 
> [5] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. "Representation learning on graphs with jumping knowledge networks." ICML, 2018.
> 
> [6] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." ICLR, 2015.
> 
> [7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." NeurIPS, 2017.

Reviewer comment (Evaluation):
Table 5 is the only table to provide uncertainty on the estimates. If this is for computational reasons, it would be good to comment on, either in the main text or in the appendix. Generally, it's good to comment on how secure your estimates are. (I see in the appendix that you specify the fixed seed you run on for the other experiments -- great, but I still maintain my point.)

Author response:
We apologize for confusion. We conducted experiments following the experimental environments of Graphormer using 4 different seeds and reported mean values of results. However, as you pointed out, there is an inconsistency in that the uncertainty on the estimations is reported only in Table 5. To address this, we have removed the uncertainty from Table 5. Due to space constraints, we include the results with uncertainty in Tables 14 and 15 in Appendix of the revised version.

Reviewer comment (Writing):
What do you mean here? "$w_K$ is learned to negative value" (page 5, comparison to GCNs)

Author response:
In GFSA, the coefficient $w_K$ can be learned to have both positive and negative values. The negative value of $w_K$ allows GFSA to extract high-frequency information from the value vectors, which is different from self-attention which can extract only low-frequency information.

Reviewer comment (Writing):
It seems like you are citing Schwartz et al. here, but to me it seems a little strong to claim that "The computations required for machine learning research are doubling every few months". If this were true we would very quickly be in a very dangerous situation (if it's true doubling). If not, is it possible to tone down the statement to 'rapidly growing' or similar -- or adding 'currently doubling'.

Author response:
Thank you for the suggestion. We agree that the statement can be toned down to avoid potential misinterpretation. We will revise the expression to convey that the computations required for machine learning research are rapidly growing.

Reviewer comment (Writing):
Page 3: missing capitalization of Equation in multiple places

Author response:
All minor comments you raised have been reflected in the revised paper.

Reviewer comment (Writing):
"latent representations tend to become similar to each other, leading to a loss of distinctiveness" (I propose to rephrase this sentence, perhaps deleting the second clause as it does not add any new information -- or add 'loss of distinctiveness in the representations'.)

Author response:
All minor comments you raised have been reflected in the revised paper.

Reviewer comment (Writing):
"sometimes uniformity among patches or tokens" >> would be nice to be a bit more precise here. English-wise, maybe you could say 'sporadic' instead of sometimes if you want an adjective, or delete 'sometimes' entirely.

Author response:
All minor comments you raised have been reflected in the revised paper.

Reviewer comment (Writing):
Section 4.4, would it look nicer to write "graph transformers" in plural instead of 'graph transformer'? Both in section title and in text

Author response:
All minor comments you raised have been reflected in the revised paper.

Reviewer comment (Presentation):
I would increase the column width a little bit in Table 3, and put #Layers in plural to be consistent across layers and params.

Author response:
All minor comments you raised have been reflected in the revised paper.