Paper: Graph Positional and Structural Encoder (ICLR-2024)

Reviewer comment (Theory):
**Clarity on computational complexity:** A major limitation of most hand-crafted PSEs is their high complexity on large graphs. The paper could benefit from a deeper dive into GPSE's computational complexity, particularly when compared against the often complex hand-crafted PSEs and other encoding strategies.

Author response:
We appreciate the reviewer’s constructive comment regarding the computational complexity of GPSE. As the reviewer has suggested, we added a detailed analysis with an empirical study demonstrating the computational benefit of using GPSE instead of directly computing hand-crafted PSEs. As a concrete example, computing all PSEs on MolPCBA took about four hours, while GPSE only took about four minutes. We kindly refer the reviewer to the global response, “Computational complexity of GPSE,” for more details, as well as Appendix [H] for results and additional visualizations.

Reviewer comment (Experiments):
Q1: For downstream tasks, is the pretrained GPSE kept frozen or is it finetuned? Would fine-tuning the GPSE improve its performance? It is noteworthy that, in other modalities like NLP and computer vision, fine-tuning pretrained models often outperforms freezing pretrained models.

Author response:
We thank the reviewer for the insightful comments. We note that all GPSE results presented in the main paper are obtained with a GPSE model with frozen weights (no fine-tuning). On the flip side, we experimented with fine-tuning but no concrete conclusion about whether it is helpful was drawn. Specifically, we have tried fine-tuning GPSE on the two primary molecular property benchmarking tasks: ZINC and PCQM4Mv2. As we have presented in Appendix Table F.5, fine-tuning could help in certain cases, but no consistent pattern shows the superiority of fine-tuning.

Fine-tuning pre-trained models is a vast topic, and how to effectively do so for better downstream performance in our specific setting is an under-explored area. Thus, an exciting future avenue is to study the optimal way to fine-tune GPSE to further adapt to specific downstream datasets.

Reviewer comment (Novelty):
The research primarily excels in engineering advancements for graph neural networks, focusing more on practical architectural solutions than on extending theoretical foundations. Although these innovations address key issues in GNNs, they potentially add complexity and computational overhead that aren't thoroughly examined.

Author response:
We thank the reviewer for recognizing the engineering advancements of GPSE and its practical contributions to the community. We understand the importance of providing a more thorough analysis of GPSE’s computational complexity to improve its practical impact further. We kindly refer the reviewer to our global response, “Computational complexity of GPSE,” for more detailed information regarding the newly added analysis. In short, we have demonstrated and explained the computational benefit of using GPSE instead of computing all exact PSEs.

Reviewer comment (Evaluation):
The model's performance, when combined with GPSE, is less impressive, considering its complexity, which includes 20 MPNN layers, MLP heads for each PSE, a gating mechanism, and a virtual node.

Author response:
To avoid any potential confusion, we would like to underline that these listed components are *of the GPSE model* that are used to generate latent PSEs *in inference mode*, and do not pertain to the downstream models (GPS/GIN/GCN etc.) used. All GPS experiments have been run with 3 to 10 layers, while GCN and GraphSAGE for the node-level tasks use only 3 message-passing layers, for example. The MoleculeNet experiments (Table 4) are similarly conducted with a 5-layer GINE, identical to Sun et al. (2022). In short, the models we use to obtain the results are appropriate for their respective benchmarking settings.

There is of course a limitation of how much PSEs by themselves can help improve performance, which is bound by how much positional information is useful for the given dataset and task at hand. This also depends on the PSEs that are learned; with more powerful PSEs, we can expect the benefits of GPSE also compound. For the scope of our paper, we would like to emphasize that **we consistently match or beat individual PSEs as well as their combinations (please note the added AllPSE experiments)**, an important result by itself.

Finally, despite the suggested complexities, **GPSE is nevertheless much more efficient than the alternative of explicitly computing PSEs**. We have conducted extensive studies on computational runtimes on both (a) increasing number of graphs, and (b) increasing graph sizes while keeping the number of graphs constant, and demonstrated that GPSE has both better wall-clock times and scaling properties on average than individual PSEs, and their combinations. We kindly refer the reviewer to our global response “Computational complexity of GPSE” as well as Appendix [H] for results and additional visualizations.

Reviewer comment (Experiments):
For fair comparison in Table 2 and Table 3, GPSE should be compared not only with single combinations like GPS+LapPE or GPS+RWSE, but also with a more comprehensive set-up, including GPS+LapPE+ElstaticPE+RWSE+EigValSE+HKdiagSE+CycleSE, with simple projection layers for each PSE type.

Author response:
We appreciate the reviewer for bringing up this valid concern regarding a more proper baseline of concatenating all PSEs. We kindly refer the reviewer to our global response, “Comprehensive comparison against AllPSE baselines”, for more detailed information about the newly added baseline as suggested. In brief, we observed that GPSE outperforms or matches the performance of AllPSE in nearly all of our primary benchmarks.

Reviewer comment (Experiments):
In Table 4, GPSE appears to be less advantageous compared to the SSL `Self-supervised pre-trained (Hu et al., 2020b)` model. Hu et al. outperform GPSE on 3 out of 5 datasets by some margins.

Author response:
We first highlight that the Hu et al. results in fact report the *best of 5 distinct self-supervised pre-trained methods*; we opted to show only the best for each dataset for compactness. Our original impression was that the table became overcrowded and detracted from the main points of our results regarding information transfer to downstream tasks. We thank the reviewer for pointing this out, as the current setup may be misleading in that it puts the SSL methods in Hu et al., 2020b in a better light than they actually are. We have therefore added the expanded version of Table 4 to Appendix [I]. 

Additionally, we have updated our GPSE results in Table 4 after conducting more hyperparameter search on the *mixing ratio between GPSE and the original graph features* - a standard parameter to tune in GPS. Previously, we did not optimize the GPSE performance on the MoleculeNet benchmarks as we did not intend to imply a direct comparison between SSL and GPSE: **The two approaches are complementary** to each other and the key conclusion we wanted to draw was that **GPSE also shows strong ability to transfer knowledge** from large pre-train datasets to the downstream task. While we still think that was the primary intention, we also understand the reviewer’s concern about GPSE appearing to be less advantageous to SSL on paper. With these latest results, *GPSE matches or outperforms the combined SSL on 7/8 MoleculeNet datasets*, and overall are consistently more powerful than any single SSL method.

Finally, we want to reiterate the difference in GPSE’s and SSL’s abilities to transfer knowledge. SSL methods are constrained to transfer knowledge between graphs that have the exact format: node features and edge features. This is due to the nature of SSL expecting the same input graph format between pre-train and downstream datasets. On the flip side, GPSE by design, is more flexible: it can transfer knowledge to *any* graph structured data, as it only uses graph connectivity for the computation, not node/edge features. This flexibility makes the obtained competitive results against SSL methods even more impressive.

Reviewer comment (Writing):
The GPSE architecture is unclear in section 2.2, even after re-reading the appendix and the section multiple times. It would have been easier to follow if the approach was explained in a step-by-step manner and aided with equations.

Author response:
We thank the reviewer for suggesting improving the clarity of the GPSE’s computation. We have added a more detailed formulation of the GPSE model with equations to Appendix B.1. In brief, GPSE takes input node features as random noise generated from the standard Gaussian distribution, which is then linearly projected to the hidden dimension. It then passes the processed features through a 20-layer GatedGCN with virtual nodes. Finally, the processed hidden representations are projected to predict one of the PSEs via a two-layer MLP.

Reviewer comment (Novelty):
The contributions in the paper seem to be limited as many of the methods have been adopted similarly but it is unclear how they are different from the existing approach. (please refer to questions)

Author response:
We thank the reviewer for raising this question. While we disagree with the statement that “similar methods have already been adopted and that this limits the contributions of the paper”, we understand that this likely stems from the confusion in the scope of GPSE and LPSE papers, which are completely distinct as we have addressed in our global response, “Relation between GPSE and LPSE”. We would nevertheless like to further elucidate our contributions and overall novelty of our method here.

Our novelty lies in the idea that we can **pre-train a positional encoder from random features using a standard MPNN** – a task that is *novel in the literature with no prior attempt* to the best of our knowledge. We not only show that our learned PSEs consistently and significantly outperform conventional PSEs across a wide range of datasets and tasks, but also are much **faster to compute** and have better scaling properties (please see Appendix H for the new computational cost and scaling experiments). Additionally, we show that GPSE is extremely generalizable, a property that is inherently absent in conventional PSEs. As mentioned in the paper, a scalable and generalizable PSE has huge implications for building efficient foundational models for graph data.

Finally, we would like to demonstrate that designing an MPNN that is suitable for this task is also **technically challenging**, even if many submodules we employed are readily available from the literature: As indicated by our ablation studies, typical GNNs (such as GCN or GIN) or network depths (<5 layers) are completely unsuitable for learning PSEs. Instead, we found that gated message passing (GatedGCN) and unusually deep networks are essential for this task. Furthermore, we have (1) discussed these design decisions in Section 2.2, with *theoretical justifications* in Appendix C, (2) showed *evidence supporting the effectiveness* of our choices in Table 1, Figure 2 and Tables F.1 and F.3, and (3) demonstrated the effectiveness of the GPSE model on specific *1-WL toy examples* in appendix E. These extensive discussions we provided and the ablation studies we conducted are not just supplemental information. They stand testament to the effectiveness of GPSE and its ability to capture rich positional and structural representations of graphs in a unified framework for augmenting downstream prediction models.

Reviewer comment (Experiments):
Why are GNN models used as a baseline for testing the extreme OOD node-classification benchmarks (table 6) instead it will be interesting to see how transformer+GPSE will perform on the dataset as they don't have the inductive bias of GNNs.

Author response:
Classical GNNs like GCN and GraphSAGE were used as baselines for the OOD node-classification benchmarks because they are known to outperform graph Transformers and related models (e.g. GPS) on large, homophilic graphs typically used for node classification, for reasons we elaborate on below. Nevertheless, we have run experiments with both graph Transformer and GPS models, we report the results below, the full table is also added to Appendix F, table F.7. In both cases, we see that GPSE leads to clear improvements, even though the overall performance falls short of GCN/SAGE/GATEv2, still leading architectures for large graphs.

Graph Transformers typically have quadratic complexity in the number of nodes to employ global attention between node pairs. These computational requirements become prohibitively large for graphs with nodes in the order of thousands. Additionally, graph Transformers are known to run into over-smoothing issues much more quickly than MPNNs, which leverage the sparsity of these graphs. Especially for homophilic large graphs, the inductive biases of MPNNs are therefore typically preferable to graph Transformers for both theoretical and empirical reasons.

Building scalable graph Transformers is expectedly a major area of interest in graph learning. One simple and effective approach to improve the scalability of both MPNNs and graph Transformers is neighbor sampling, where a subset of neighbors (or nodes in the case of a Transformer) are employed for message-passing. However, we may expect neighbor sampling to simulate the information flow of a graph Transformer only approximately, leading to some loss of information in the process; this is also implied by the comparatively weak performance of the Transformer-based models.

| ****                   | ****      | **arXiv**             | **Proteins**          |
|------------------------|-----------|--------------------------------|--------------------------------|
|                        | **+GPSE?** | **ACC %** | **AUROC \%** |
| GCN                    | NO    | 71.74 ± 0.29                   | 79.91 ± 0.24                   |
|                        | YES    | 71.67 ± 0.12                   | 79.62 ± 0.12                   |
| SAGE                   | NO    | 71.74 ± 0.29                   | 80.35 ± 0.07                   |
|                        | YES    | 72.19 ± 0.32                   | 80.14 ± 0.22                   |
| GAT(E)v2               | NO    | 71.69 ± 0.21                   | 83.47 ± 0.13                   |
|                        | YES    | 72.17 ± 0.42                   | 83.51 ± 0.11                   |
| Transformer            | NO    | 57.00 ± 0.79                   | 73.93 ± 1.44                   |
|                        | YES    | 59.17 ± 0.21                   | 74.67 ± 0.74                   |
| GPS                    | NO    | 70.60 ± 0.28                   | 69.55 ± 5.67                   |
|                        | YES    | 70.89 ± 0.36                   | 72.05 ± 3.75                   |

Reviewer comment (Novelty):
The paper [1] has a similar research question and method to GPSE, can the authors help me understand the difference?

Author response:
We kindly refer the reviewer to our global response, “Relation between GPSE and LPSE,” for a more in-depth explanation of the difference between the two.