Paper: Graph Transformers for Large Graphs (ICLR-2024)

Reviewer comment (Experiments):
Baselines: LargeGT is compared to "constrained versions" of various baselines, notably all models are constrained to 2 hops only, while LargeGT has access to 4-hops worth of neighbors (in the local module). Including the non-constrained versions of these same baselines is critical for evaluation, even if they are more computationally demanding. Currently it is unclear whether adopting LargeGT leads to lower performance compared to state-of-the-art methods, at the expense of computational efficiency.

Author response:
**Authors:** We appreciate your feedback on the selection of baselines for comparison. The decision to compare LargeGT with 2-hop constrained versions of various models was initially to maintain a consistent scope of expensive computation incurred across all models. For instance, the cost of fetching l-hop neighbors for a node in a very large graph would be the same and agnostic of any selected model. Nevertheless, below is a comparison of the baselines along with LargeGT scores on ogbn-products without restriction to 2-hop. For instance, GOAT models are run with original hyperparameters which samples 20, 10, 5 neighbors at the 3 hops.


| Model | 		Test Acc (original) | 	Epoch Time (original) |Test Acc (2-hop ) | Epoch Time (2 hop)|
| - | - | - | - | - | 
|GraphSAGE | 		78.17 |		22s|		76.62|			7s|
|GAT | 			79.21 |		86s|		77.38|			31s|
|GT-sparse | 		67.87 |		40s|		60.76|			12s|
|NAGphormer |	77.71 |		22s|		75.28|			8s|
|GOAT-local | 		81.29 |		490s|		81.17	|		120s|
|GOAT-global | 	70.28 |		3.5s|		70.28	|		3.5s|
|GOAT-full | 		81.21 |		500s|		79.88	|		205s|
|LargeGT-local |	78.95 |		45s|		78.95	|		45s|
|LargeGT-full | 		79.81 |		68s|		79.81	|		68s|

We can observe that the best performing model remains the same while LargeGT (in both cases uses up to 2 hop operations) remains better and competitive compared to the best performing model. In the revised manuscript, we have included a section Appendix A.4 which details these comparisons to provide a clearer perspective on LargeGT's performance relative to state-of-the-art methods.

Reviewer comment (Evaluation):
Additionally, no auxiliary label propagation or augmentations are used for the baseline methods, when they are used in methods reported in the OGB leaderboard. These enhancements are not altering the receptive field of the baselines, and thus shouldn't impact computational performance, but might improve classification performance. This should be taken into account when comparing with approaches that might still outperform the proposed method, even under constrained training (2-hop).

Author response:
**Authors:** Thank you for pointing out the absence of auxiliary label propagation or augmentations for baseline methods. We acknowledge that including these enhancements could have a chance to provide improved performance for specific models. However, it has been observed in the leaderboards that the composition of several enhancement techniques blurs the dissection on which model components are critical and contributes highly to the performance of a model. For example, a label propagation technique known as Correct & Smooth (C&S) [1] is employed in various models on the OGB leaderboard for ogbn-products [2]. It demonstrates that MLP-based models with C&S perform both better and worse than TransformerConv [3] in different settings. This provides a somewhat blurred perspective on whether MLP or Transformer models are superior for ogbn-products from a neural network perspective. We  mention our reasoning in Section 4.1 while GOAT, which we closely follow, also follow a similar experimental setting on not using enhancement tricks.

[1] Huang, Q., He, H., Singh, A., Lim, S.N. and Benson, A.R., 2020. Combining label propagation and simple models out-performs graph neural networks.     
[2] https://ogb.stanford.edu/docs/leader_nodeprop/#ogbn-products.    
[3] Shi, Y., Huang, Z., Feng, S., Zhong, H., Wang, W. and Sun, Y., 2020. Masked label prediction: Unified message passing model for semi-supervised classification.

Reviewer comment (Evaluation):
The main innovation can seemingly be credited to the use of the global codebook, so it is hard to define the main contribution of this work. If the focus of this work is combining all these different building blocks into a compute efficient framework, I would expect to see a more expansive breakdown of the computational costs of different components, memory usage and requirements. Notably, how is "Epoch time" defined in Figure 2? All models might be processing different amounts of data and thus might have different definitions of an "epoch" due to differences in sampling strategies. How many nodes does each model process in an epoch? Different models might require different numbers of epochs to converge, shouldn't total training time be more important?

Author response:
**Authors:** We would like to clarify here that we do not mention the global codebook as the main contribution of our work. In fact, we refer to Section 3.2 in page 5 where we state that we adopt GOAT’s global module for our global module as well. One of our key contributions is addressing the bottleneck brought by GOAT’s local module, to which we propose our own local module that is described in detail in Section 3.2. To highlight in a sentence here, our local module uses tokens that are sampled/computed offline prior to training, prepares mini batches to allow 4-hop receptive field, and performs a Transformer encoder layer over the input tokens. 

For the latter part of the question on epoch time, we use wall clock time for an epoch to report the epoch time. Thank you for your question on convergence. We have included the learning curves in Appendix A.1 showing the performance of different baselines and LargeGT within a fixed training budget. Upon comparing the learning curves of LargeGT with four other baselines in Appendix A.1, we observe faster convergence and relatively earlier learning stability for LargeGT. We hope that section will answer the question on epoch time vs. total training time.

Reviewer comment (Presentation):
[Minor] A lot of the content in the first 4 pages is repetitive.

Author response:
**Authors:** We thank you for your feedback on writing. We tried trimming some repeated references in Section 3. However, we will do our best to reduce the redundancies for our final version.

Reviewer comment (Experiments):
What are the memory constraints of using LargeGT compared to other baselines? How is the choice of batch size impacted by the choice of hyperparameter K?

Author response:
**Authors:** The memory requirement of LargeGT is determined similar to how Transformers operate on tokens of data, as done in text or images. LargeGT was designed with an emphasis on scalability and efficiency for large-scale graphs. From our computational complexity analyzed in Section 3.2, the memory requirements depend on the hyperparameter K and the codebook size B (in addition to model parameters) which determines what will be the size of one sample in the mini batch for which local and global modules are operated on respectively. While we have provided the value of K in the main paper, we have included the batch size details and hardware information in the appendix. The batch size and K were selected in two ways (i) first by adopting the sizes used in original baseline’ papers, and (ii) second by making sure it fits the hardware we have used for our experiments.

Reviewer comment (Experiments):
How important is the choice of a 4-hop neighborhood for the local module. Can the model still perform competitively given that it still has access to global information through the global module?

Author response:
**Authors:** The choice of a 4-hop neighborhood for the local module is a critical aspect of LargeGT's design, aimed at balancing the depth of the receptive field with computational efficiency. It can also be seen as a feasible local receptive field, as we achieve it using only 2-hop local operations. Additionally, several recent works, such as GOAT, or MPNNs with Neighbor sampling, were seen to use a 3-hop neighborhood sampling since beyond 3-hop becomes infeasible for graphs of large sizes. For the latter part of the question, we refer to the performance of GOAT-global-δ which is a baseline equivalent to LargeGT-global, i.e. LargeGT without the local module. We can observe that the performance of GOAT-global-δ is significantly worse compared to LargeGT.

Reviewer comment (Theory):
In Algorithm 1, some nodes are sampled with replacement, while some are sampled without replacement. It is uncertain whether this will introduce bias in the sampling.

Author response:
**Authors:** Thank you for highlighting this aspect of our sampling method in Algorithm 1. We take this opportunity to clarify that the “sampling with replacement” only applies to those nodes whose 1 and 2 hop neighbors are less than K-1. This becomes a corner case and applies to only a small fraction of the nodes in general. As such it does not bias how we perform our overall sampling, except that we apply special handling of the nodes with lesser neighbors than K-1 . It is true that such corner cases can be handled differently as well, such as using padding nodes, instead of sampling with replacements.

Reviewer comment (Experiments):
It lacks some key baselines such as SGC[1], SIGN[2].

Author response:
**Authors:** We thank you for pointing out the absence of comparisons with baselines like SGC and SIGN. The decision to exclude these baselines initially was based on their different focus compared to LargeGT. However, aspects of these works are closely related such as “offloading heavy computations away from the training stage” which we also follow in Algorithm 2 in the form of “Hop context features” . While we had already included SIGN in Section 2 of our paper, we have now also included SGC in the revised manuscript. This addition falls under the collection of related works that 'perform information propagation prior to the training stage', aiming to reduce the training cost of GCNs."

Reviewer comment (Evaluation):
The mechanism of why LargeGT runs faster than baselines like GOAT is unclear. Since the proposed neighbor sampling has a bigger input matrix than a simple 2-hop neighbor sampling method, does it run longer than the traditional method?

Author response:
**Authors:** There are two reasons for this. (i) Conceptually,  LargeGT runs faster than baselines like GOAT due to the introduction of our local sampling technique that is performed offline and is restricted to 2-hop neighbors, as opposed to GOAT which performs sampling during mini-batching and frequently uses 3+ hop in practice. (ii) Experimentally, in our paper, we keep 2-hops for LargeGT and GOAT-δ. In such a case, the efficiency in LargeGT comes from the fact that these 2-hop neighbors are sampled offline and not during mini-batching. Although we have discussed the characteristics of LargeGT in terms of “Scalability” in Section 3 under the subheading “Characteristics of the framework”, we will elaborate further on these points in the revised manuscript for clarity. Additionally, the computational complexity of global module in both LargeGT and GOAT are the same. However, GOAT’s local module is dependent on the number of nodes in a graph and the sampling hop length, unlike LargeGT’s local module, which enables LargeGT to scale better. We refer to GOAT’s paper where they highlight their local module as a major limitation which is addressed in our work as part of our contribution.

Reviewer comment (Experiments):
The runtime highly depends on the hyperparameter $K$, which is the number of nodes for sampling. Authors need to provide a fair and solid comparison with the traditional 2-hop neighbor sampling method.

Author response:
**Authors:** Since the computational complexity of LargeGT’s local module is dependent on K, the runtime depends on K. We would like to refer to Figure 1 and Eqns. 1-6 in our paper (further in Appendix A.2) which informs that the hyperparameter K determines the size of node sampled “prior to training” as well as the number of tokens for LargeGT’s local module, which is 3K (see “Complexity” in Section 3). As such, the hyperparameter K will influence the size of the data (or input tokens) the local module is operating on, thus affecting the runtime. Our sampling method prior to training is random sampling as outlined in Algorithm 1 – similar to how traditional sampling method is used. The only difference, say with GOAT’s sampling method or other similar sampling methods, is that Algorithm 1 is executed offline, as illustrated in Section 3 (“Offline Step Prior to Training”); while GOAT’s sampling is done during mini batching. In addition, unlike the sampling performed during mini-batching, Algorithm 1 paves the way for parallelizing the sampling of each graph on distributed computing resources.

Reviewer comment (Evaluation):
In Table 2, why does GOAT-local-δ have better accuracy in ogbn-products?

Author response:
**Authors:** Thank you for pointing out these key observations from our experiments. On ogbn-products, GOAT-local-δ shows better performance due to the dataset's homophilic nature, where local-only information aggregation is often sufficient. Conversely, for snap-patents, a non-homophilic dataset, LargeGT excels by incorporating both local and global neighbor information. This adaptability of LargeGT to different dataset characteristics explains the observed performances. We have included our observation to this regard in Section 4.2 under the heading “On Performance”, which we will revise for more clarity in our next revision.

Reviewer comment (Evaluation):
For snap-patents in Table 3, why does the model accuracy drop when $K>50$?

Author response:
**Authors:** Continuing on the line of reasoning of the previous question, the observed drop in model accuracy for snap-patents when K>50 is due to the over-sampling of neighbors from 1-hop and 2-hop, which might introduce noise or irrelevant information beyond a certain point. This phenomenon indicates that there is an optimal range for K, assisting the model to classify the labels correctly on snap-patents’ non-homophilic task.