Paper: Distinguished In Uniform: Self-Attention Vs. Virtual Nodes (ICLR-2024)

Reviewer comment (Novelty):
It seems that the presented theoretical results in the non-uniform are relatively trivial, as they are straightforward results of the combination of MLP’s universality and PE’s discrimination capacity.

Author response:
It is true that the proofs in the non-uniform setting are rather straightforward as they effectively restate a known proof, showing that it can be applied to a variety of settings (and not just graph transformers). It is therefore not the main technical contribution of the paper. We included this section for two reasons: 
1. They complete the formal picture and provide a contrast to the uniform setting results.
2. They clarify some prior results from the graph transformer literature. 
Some works, such as GPS, highlight the fact that their approach can universally approximate functions in a non-uniform setting as an important property of their method without clarifying that this is the case for virtually any architecture when using the same positional encoding.
We believe it is important to emphasize formally that this somewhat trivial result is not a unique property of graph transformers and therefore certainly not an advantage.

Reviewer comment (Experiments):
The difference and respective advantages deserve to be connected to practical tasks on molecular graphs, as there have been many public tasks, some of which graph transformers outperform traditional message-passing gnns, yet some are not. Such connections must be helpful for the community and make the theoretical results practical.

Author response:
We do not have a clear recommendation to use one or the other model for this or that specific task. Our results convey a general message: Neither GPS nor MPGNN+VN performs strictly better than the other across all learning tasks. We believe that this message, although general, is relevant to practice: It can increase awareness among practitioners that they should consider both models when tackling a learning task instead of believing one model to be superior and dismissing the other without additional thought.\
Moreover, our results do offer a general connection: If the target function is estimated to include the global sum of some data of the nodes then MPGNN+VN may be the better choice to start with, and if the target function is estimated to include a sophisticated global weighted average where the weight depends on both sides then GPS may be the better choice.

Reviewer comment (Experiments):
In the synthetic experiments, the authors said they were interested in the generalization behavior of the trained models. However, the setting is not as usual. It is not an i.i.d. generalization but o.o.d. extrapolation (graph size <= 50 during training and > 50 in test). Generally and intuitively, a model that is more sophisticated with o.o.d. extrapolation is often due to its more reasonable inductive bias or limited hypothesis space, such that it captures the underlying actual mapping rather than fitting the training data by other consistent yet different mappings. Thus, I need clarification about the rationale behind experimental design. Could you explain this to me?

Author response:
By "generalization" we mean correctly handling graphs of sizes larger than seen during training, which is indeed generalization in the o.o.d. sense. We hypothesize that the latter corresponds with uniform expressivity i.e.:
1) That uniform inexpressivity implies an extremely low probability of learning a GNN that generalizes (to o.o.d. graph sizes).
2) That there is a good probability of learning uniformly expressive networks (which by definition generalize) when they exist.

If our hypothesis is true, then differences (between architectures) in uniform expressivity are meaningful in practice, making theoretical results regarding uniform expressivity practically relevant. Our synthetic-data experiments are meant to examine the above:
1) To test the magnitude of the generalization error of a learned GNN when the architecture is uniformly inexpressive of the target function.
2) To test how well we can learn from data (using SGD) a well-generalizing GNN, when a uniformly expressive GNN is known to exist.

We will try to add clarification regarding the above, in the updated manuscript.

Reviewer comment (Theory):
The theoretical analysis focuses on comparing one variant of GTs (GPS) and MPGNN+VNs. Results could vary for different architectures within these families. How accurate would this generalization be?

Author response:
The first question regards the families of MPGNN+VN and GT architectures to which our theory applies. 
Formally reasoning about these architectures naturally requires us to fix some specific design choices, but we would argue that both GPS and MPGNN+VN represent very large and practically relevant classes of models.
In particular, our theory is completely agnostic to the choice of the message passing module and holds true for any popular MPGNN layer such as GCN, GIN, GAT, or GatedGCN.
This is clear when looking at the graphs we use to prove the differences because they both contain no edges so no messages are sent during the message passing step.\
The intuition underlying Theorem 4.3 and Corollary 4.4 is that weighted averages like softmax self-attention struggle to uniformly approximate functions that depend on the absolute number of vertices.
We expect that this generalize to most graph transformers with global information exchange built on such weighted averages.
Of course, the formal proof details may need to be adapted for GTs outside of the GPS framework.
We will update the camera-ready version to make this part clearer.

Reviewer comment (Experiments):
On some realworld datasets, MPGNN+VNs do not fully close the performance gap compared to GTs. It is unclear if this limitation is fundamental or if deeper MPGNN+VNs could match GTs.

Author response:
This is a broad, but highly important question, we are also very interested in. At least in our experiments, going deeper (while keeping the parameter limits of LRGB) did not lead to better results. Most notably, there is a gap on the vision dataset COCO where pure vision transformers achieve even better results. Nevertheless, it is an intriguing question what aspects are key to the vastly different performance of the various graph learning models on that dataset.

Reviewer comment (Writing):
The most important weakness of this paper is writing. The introduction is not easy to understand.

Author response:
**Q**: Weaknesses (1): "The most important weakness of this paper is writing. ..."\
**A**: We agree that the writing can be improved throughout the manuscript and will address this in the revised version. We will also update the paper with a clearer introduction.

Reviewer comment (Theory):
**Lack of Justification**: The paper focuses on scenarios where positional encoding is injective. However, there is a noticeable lack of justification for why this particular scenario is important or realistic. To ensure that the results derived hold value in practical applications, it is essential to provide a clear context and relevance for the chosen scenario.

Author response:
**Q**: Weaknesses (2): "the paper focuses on scenarios where positional encoding is injective..."\
**A**: We consider injective positional encoding only for section 3, where we extend known results in the non-uniform expressivity setting. The main focus of the paper is the uniform expressivity setting, discussed mainly in section 4. Our results there are *not limited to* having (injective) positional encodings. Rather, the inexpressivity results for GPS hold *even when* (pg 7, lines 1;4;8) the initial graph is featured with positional encodings.
We will review the presentation to see how these points can be made clearer in the camera-ready version.

Reviewer comment (Theory):
The process of proof is not so clear. For example, in Section 4.2, the author attempts to prove GPS do not subsume MPGNN+VNs. However, I can not understand the main path of proof of Theorem 4.3 and Corollary 4.4.

Author response:
**Q**: Weaknesses (3): "the process of the proof is not so clear..."\
**A**: We assume that you refer to the proof overview (in the main part), we agree that it should be made clearer, and we intend to do so in the camera-ready version.

Reviewer comment (Theory):
The strong assumption of uniform setting made in the paper may not align with the practical case.

Author response:
**Q**: Weaknesses (4): "The strong assumption of uniform setting..."\
**A**: We argue that the uniform setting is highly relevant to practice for the following reasons:
As demonstrated in our experiments, the uniform setting is a good representation of scenarios where the graph sizes at training time are smaller than the graph sizes during inference. In practice, it is desirable to learn models that robustly infer missing labels on data outside of the training distribution.
Furthermore, due to compute-resource limitations and relevancy requirements, it may be desired not to re-train whenever new graphs are larger than seen before e.g. when the system modeled by the graphs is growing over time.
We agree that a justification like the above should be mentioned explicitly in the paper and we intend to add it in the camera-ready version.

Reviewer comment (Novelty):
**Redundancy in Experimental Results**: Similar experimental outcomes have been presented in multiple prior works, notably:
- Tönshoff, Jan, et al. "Where did the gap go? Reassessing the long-range graph benchmark." arXiv preprint arXiv:2309.00367 (2023).
- Cai, Chen, et al. "On the connection between MPNN and graph transformer." arXiv preprint arXiv:2301.11956 (2023).  
While building upon prior work is a hallmark of research progression, it's crucial to ensure that the presented findings either provide a novel perspective or build significantly upon the existing literature.

Author response:
**Q**: Weaknesses (5): "Redundancy in Experimental Results..."\
**A**: First, we would like to clarify that the experimental part of this work is meant to complement the theoretical part, it is *a* contribution but not the main contribution.
The core contributions of our experiments are two-fold:
1) Our study on synthetic data is new. It provides the first direct empirical comparison of graph transformers and virtual nodes with regard to uniform function approximation.
2) Our experiments on real-world data provide a thorough evaluation of MPGNNs with virtual nodes on LRGB and other benchmark datasets.
There is some overlap with the experimental setup of Cai et al., but we should point out that their models do not adhere to the official 500k parameter budget and are comparatively under-tuned, as our configurations yield better results with smaller models. 
Tönshoff et al. identify some weaknesses in the LRGB results reported for MPGNNs and provide improved values, but only for models without VN. 
Our experiments extend their methodology to MPGNNs *with* VN to obtain novel and improved results for this class of models.

We will improve the presentation of the experiments in the updated manuscript to highlight this more clearly.

Reviewer comment (Writing):
The paper utilizes many acronyms, which may necessitate repeated explanations. For instance, in Figure 1, it would be helpful if the author could reiterate the meanings of "MP," "VN," “SA,” “PH,” and "FF" in the figure caption. Additionally, in Figure 2(b), clarifying the definitions of "l" and "r" would enhance writing clarity. It is also recommended that the authors create a table in the appendix with all the acronyms.

Author response:
**Q**: Weaknesses (1): "Regarding the writing..."\
**A**: We agree with the suggestions and will implement them in the manuscript.

Reviewer comment (Theory):
The assumptions and limitations of the proposed theory are somewhat unclear. For instance, it is not clearly defined whether the theory is applicable to various graph tasks or solely focused on graph classification tasks.

Author response:
**Q**: Weaknesses (2a): "The assumptions and limitations of the proposed theory are somewhat unclear..."\
**A**: Our theoretical results concern graph-level function-approximation tasks. Some of the results can be extended to node-level tasks without much effort, while for other results more work is required to make conclusions concerning node-level tasks.
For example, it is not difficult to create a node-level function that GTs cannot learn while MPGNN+VNs can (generalizing Corollary 4.4).
We will add a clarification to the manuscript.

Reviewer comment (Evaluation):
The message conveyed in Section 5.2 is vague and disconnected from previous sections. Given that Table 1 indicates that the best performance appears somewhat random, it is not convincing how the theoretical insights can guide practice. It may be valuable to conduct a more thorough investigation into the utilization of theoretical insights for model selection based on the characteristics of the dataset.

Author response:
**Q**: Weaknesses (2b): "The message conveyed in Section 5.2 is also vague..."\
**A**: Unlike the synthetic data experiments, our experiments on real-world problems are not meant to demonstrate our theoretical results directly. Rather, they offer another view leading to the general message of the paper: Neither GPS nor MPGNN+VN performs strictly better than the other across all learning tasks. Hence, the first conclusion from both our theoretical and experimental results is that both models should be considered in the general case. However, the theoretical insights suggest that if the target function is estimated to include the sum of some data of the nodes then MPGNN+VN may be the better choice to start with, and if the target function is estimated to include sophisticated weighted averages where the weight depends on both sides then GPS may be the better choice.
We will try to add a clarification in the manuscript.

Reviewer comment (Writing):
Please add a list of math symbols and abbreviations in the appendix and clarify the above writing questions.

Author response:
**Q**: Weaknesses (1): "Regarding the writing..."\
**A**: We agree with the suggestions and will implement them in the manuscript.

Reviewer comment (Theory):
How can we utilize theory to understand the results of Section 5.2?

Author response:
**Q**: Weaknesses (2b): "The message conveyed in Section 5.2 is also vague..."\
**A**: Unlike the synthetic data experiments, our experiments on real-world problems are not meant to demonstrate our theoretical results directly. Rather, they offer another view leading to the general message of the paper: Neither GPS nor MPGNN+VN performs strictly better than the other across all learning tasks. Hence, the first conclusion from both our theoretical and experimental results is that both models should be considered in the general case. However, the theoretical insights suggest that if the target function is estimated to include the sum of some data of the nodes then MPGNN+VN may be the better choice to start with, and if the target function is estimated to include sophisticated weighted averages where the weight depends on both sides then GPS may be the better choice.
We will try to add a clarification in the manuscript.