Paper: SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning (ICLR-2024)

Reviewer comment (Experiments):
Lack of comparison with existing works: GraphFormers [1], Patton [2]. There is another line of work [1,2] that tries to use only a language model to capture both semantic information and structure information in a textual graph. It would be more comprehensive to see how the performance comparison is between SimTeG and those methods.

Author response:
Thanks for the question. GraphFormer and Patton are valuable and related works. We have also included them in our related works. We report the results of GraphFormer on our three datasets as follows. For Patton, their implementation only supports their customized datasets and is not trivial to be adopted in our benchmarks. However, since patton and GraphFormer follow similar GNN-nested transformer architecture, it would be valuable to compare the performance of GraphFormer first.

As shown in the following table, SimTeG performs consistently better than GraphFormer. It is possibly because 1) the GNN-nested architecture of GraphFormer solely allows 1-hop message passing, which limits the express ability of GNN models; 2) GraphFormer’s implementation modifies the architecture code of Bert and cannot be easily extended to other SOTA embedding models nowadays.

We have added the GraphFormer results to Table A8 in the paper.

| method | ogbn-arxiv | ogbn-products | ogbl-citation (MRR) |
|---|---|---|---|
|GraphFormer | 72.81 ± 0.20 | 74.72 ± 0.16 | 82.78 ± 0.24 |
| SimTeG + GraphSAGE | 76.84 ± 0.34 | 84.59 ± 0.44 | 85.13 ± 0.73 |
| SimTeG + SOTA GNN | 77.04 ± 0.13 | 85.40 ± 0.28 | 86.66 ± 1.21 |

Reviewer comment (Novelty):
Excitement of the findings and studies. I appreciate the authors’ detailed study of the two-stage pipeline. However, the finding is quite straightforward and not exciting enough to me. It is intuitive that the initial node feature vectors are very important and if a language model is trained on the downstream task first to generate the node feature vectors for the GNN methods, it will contribute to a very good performance.

Author response:
We provide a general response section above to clarify our motivation and contributions. We sincerely appreciate the reviewer checking it. Besides, we also moderately adjust our introduction section to make our motivation and argument clearer. 

We partially agree that “if a language model is trained on the downstream task first to generate the node features, it will contribute to a very good performance.” However several crucial questions remain open: (1) what is the extent of “very good”, especially with a simple GNN architecture? (2) Are advanced text embeddings good alternatives to complicated GNN architecture?

We kindly disagree that “the findings are quite straightforward and not exciting enough”. We have summarized our contributions in the general response and hope it would address the concerns of the reviewer. Basically, the central argument is that advanced text features are good alternatives for complex GNNs. Validating this argument is beneficial for the application of GNNs in industry because industry applications of GNN architectures are limited by computational efficiency. To date, a notable exception is Pinsage [1], a GraphSAGE-based recommender system for Pinterest. If incorporating advanced text features could bypass the necessity of using complex GNN models, it would significantly boost the application of GNNs in industry.

Reviewer comment (Novelty):
Technical novelty. Correct me if I’m wrong, but this method can be seen as a single step for GLEM. The original GLEM involves iterative training of LM and GNN, while SimTeg contains only one round (LM training then GNN training). The performance comparison with GLEM is also very marginal regarding SOTA GNN.

Author response:
As the statement of the general response and our abstract, we acknowledge that the proposed method is not technically novel, and this two-stage training has been explicitly or implicitly applied in various applications, e.g., GLEM [2]. Besides, related works [3][4] also utilize a similar mechanism. However, we state two key differences that distinguish our work.

First, the motivation of our work is not to propose a novel framework that could generally outperform the other methods. Instead, we would like to utilize this simple two-stage training to demonstrate our key argument: *by incorporating advanced text features, one can bypass the necessity of using complex features.* This is mainly demonstrated by the performance improvement of a simple two-layer graphsage.

Besides the key findings and motivation, other *unique empirical findings* in our work (e.g. The impact of PEFT in Section 5.3 and The impact of LM variants in Section 5.4) could also be generalized to any similar frameworks.

[1] Ying, Rex, et al. "Graph convolutional neural networks for web-scale recommender systems." Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018.
[2] Zhao, Jianan, et al. "Learning on large-scale text-attributed graphs via variational inference." arXiv preprint arXiv:2210.14709 (2022).
[3] He, Xiaoxin, et al. "Explanations as Features: LLM-Based Features for Text-Attributed Graphs." arXiv preprint arXiv:2305.19523 (2023).
[4] Chen, Zhikai, et al. "Exploring the potential of large language models (llms) in learning on graphs." arXiv preprint arXiv:2307.03393 (2023).

Reviewer comment (Evaluation):
What is the performance comparison between SimTeG, GraphFormers [1], and Patton [2]?

Author response:
Thanks for the question. GraphFormer and Patton are valuable and related works. We have also included them in our related works. We report the results of GraphFormer on our three datasets as follows. For Patton, their implementation only supports their customized datasets and is not trivial to be adopted in our benchmarks. However, since patton and GraphFormer follow similar GNN-nested transformer architecture, it would be valuable to compare the performance of GraphFormer first.

As shown in the following table, SimTeG performs consistently better than GraphFormer. It is possibly because 1) the GNN-nested architecture of GraphFormer solely allows 1-hop message passing, which limits the express ability of GNN models; 2) GraphFormer’s implementation modifies the architecture code of Bert and cannot be easily extended to other SOTA embedding models nowadays.

We have added the GraphFormer results to Table A8 in the paper.

| method | ogbn-arxiv | ogbn-products | ogbl-citation (MRR) |
|---|---|---|---|
|GraphFormer | 72.81 ± 0.20 | 74.72 ± 0.16 | 82.78 ± 0.24 |
| SimTeG + GraphSAGE | 76.84 ± 0.34 | 84.59 ± 0.44 | 85.13 ± 0.73 |
| SimTeG + SOTA GNN | 77.04 ± 0.13 | 85.40 ± 0.28 | 86.66 ± 1.21 |

Reviewer comment (Novelty):
The technical contribution of the paper appears to be limited, especially when considering the work of [1]. The core idea closely mirrors that of [1], which also leverages embeddings learned from a language model to enhance the learning of textual graph data via a variational expectation-maximization joint-training framework. The distinguishing factor in the proposed method is its two-step approach. However, I struggle to identify substantial contributions that differentiate it from [1].

Author response:
Thanks for the question. As mentioned in the general response and our abstract, the primary objective of this paper is not proposing a brand new and novel technique to achieve SOTA results on the leaderboard. Instead, we would like to use the proposed simple framework to validate our key argument: incorporating advanced text features could bypass the necessity of using SOTA GNNs. We refer the reviewer to look through our general responses above for a detailed introduction of our motivation and contributions.

Reviewer comment (Novelty):
The authors argue that prior methods have crafted intricate tasks and structures to bolster the performance of textual graph learning with LMs. However, existing methods like [1,2,3,4,5] are conceptually simple and their frameworks are straightforward. Moreover, their training processes do not necessitate significant modifications to the prevalent model architectures.

Author response:
Thanks for the comments. In comparison with SimTeG, prior methods [1,2] and concurrent works [3,4,5] are relatively intricated in terms of their designed tasks or frameworks. For examples, GIANT [1] designs a graph-structure-related self-supervised task to generate text embeddings; GLEM [2] utilizes a joint training framework for GNNs and LMs to iteratively generate representative text embeddings; while the concurrent works [3,4,5] try to utilize LLMs to help generate more powerful text embeddings. Though showing promising results, the introduced tasks or techniques prevent us from directly validating our hypothesis from these works – whether incorporating advanced text features could bypass the necessity of using SOTA GNNs. As a result, we propose a simple framework to assess this argument.

Reviewer comment (Experiments):
While the authors have undertaken link prediction experiments, there is a noticeable absence of comparisons with some of the state-of-the-art (SOTA) methods that incorporate LMs. It would be beneficial for them to showcase, for example, the performance of GLEM or other notable methods on the link prediction task. Such a comparison could further attest to the efficacy of their proposed model.

Author response:
Thanks for the comments. We did not include the link prediction results of some SOTA methods as few previous works test their methods on link prediction tasks, and their provided code only implemented node classification and is non-trivial to be extended to link prediction tasks. To mitigate the reviewer’s concerns, we tried several related works and included the results of graphformer on our link prediction datasets. We provide it as follows and have added this baseline to Table A8 of the paper.

| method | ogbn-arxiv | ogbn-products | ogbl-citation (MRR) |
|---|---|---|---|
|GraphFormer | 72.81 ± 0.20 | 74.72 ± 0.16 | 82.78 ± 0.24 |
| SimTeG + GraphSAGE | 76.84 ± 0.34 | 84.59 ± 0.44 | 85.13 ± 0.73 |
| SimTeG + SOTA GNN | 77.04 ± 0.13 | 85.40 ± 0.28 | 86.66 ± 1.21 |

Following the general response, we note that competing for  SOTA results is not the focus of our paper. Instead, we aim to compare the performance of SOTA GNN and GraphSAGE with and without SimTeG, which could validate the central argument of our paper – incorporating advanced text features could bypass the necessity of using complex GNN architectures.

Reviewer comment (Experiments):
The review suggests compare the proposed method with GNNs which utilize bag-of-words feature not just the word-embedding feature in Ogbn-Arxiv.

Author response:
Thanks for the suggestions. As for our used datasets, the OGB does not provide the bag-of-words features, and we did not find GNN models and papers that utilize the bag-of-words feature on OGB datasets. Could you kindly provide more information or reference on this?

Intuitively, we generally believe that the dense word embedding methods should work better than bag-of-words features because the former is recognized to be more representative on summarizing sentences and paragraphs.

Reviewer comment (Evaluation):
On Ogbn-arxiv, SimTeG outperforms GLEM. From my understanding, GLEM can adaptively optimize the input embedding for GNNs, which will show better performance compared with SimTeG. Can the authors provide more discussions about this?

Author response:
Thanks for the question. SimTeG is similar to the first stage of GLEM, except for two key differences.

SimTeG utilizes LoRA during the LM finetuning stage to alleviate the overfitting problem to generate more representative features.
Second, SimTeG does not utilize pseudo labels during the GNN training stage. The pseudo labeling is crucial for GLEM and helps it iteratively optimize the embeddings. However, pseudo labeling does not help much on OGBN-Arxiv. As shown in the Figure 2 of GLEM paper, GNN already achieves the best performance on the first stage on OGBN-Arxiv dataset because of the ‘good prediction’ of LM. Instead, GLEM turns out to be more effective on more noisy datasets (e.g., OGBN-Products), where SimTeG fails to achieve on-par performance. We provided a related discussion in Observation 4 of Section 5.1 in the paper.

As stated in the previous response, we believe that our empirical findings like utilizing LoRA in Section 5.3, and the study about impact of LM variants in Section 5.4 would also be beneficial to improve the performance of methods like GLEM.

Reviewer comment (Experiments):
Could the authors provide further insights into the specific LM variants that can significantly enhance GNN models? For instance, it would be valuable to understand whether larger LM parameters or other factors play a substantial role in this improvement.

Author response:
Thanks for the question. We have discussed the influence of LM variants in section 5.4 and Table A7 of the original submission. We further elaborate on it and add new empirical results to make the discussion more complete. Basically, we summarize our findings below: 
the number of LM parameters is not substantial for the improvement. Including the results shown in our paper, we further test 2 more LMs that have different numbers of parameters, instructor-xl (1.5B) and deberta-v2-xxlarge (2.8B). The results on OGBN-Arxiv and statistics are shown in the following table and included in Figure 4 (right subfigure) of our updated manuscript. When we try to increase the number of parameters, the performance does not improve further. This is possibly because even though models with larger parameters could have better fine tuning node classification performance, the resulted text embedding is not correspondingly improved. Thus, we conjecture that the key factor for improving the performance of GNN is the quality of text embeddings, which, as shown as MTEB (https://huggingface.co/spaces/mteb/leaderboard), is not positively related to the number of parameters.

| Metric | Model | all-MiniLM-L6-v2 | all-roberta-large-v1 | e5-large | deberta-v2-xxlarge | instructor-xl | 
|--------|----------|----------|-----------|--------|-------|---------|
| #Params. |    | 22M       | 355M     | 335M      | 1.5B | 2.8B | 
 | Acc.   | MLP | 70.56 ± 0.09 | 74.32 ± 0.12 | 74.06 ± 0.13 |  72.87 ± 0.09   | 74.82 ± 0.08  |
|          | GraphSAGE | 75.14 ± 0.30 | 76.18 ± 0.37 | 76.84 ± 0.34 |  75.93 ± 0.29  | 76.59 ± 0.36 | 

We performed an ablation study regarding the quality of text embeddings on Table A7 in our original paper, where we systematically compared two models with the exactly the same architecture, roberta-large and all-roberta-large-v1. The second one is the finetuned version of the first one using contrastive learning. The results show that for both LM finetuning stage and GNN training stage, all-roberta-large-v1 consistently outperforms roberta-large, indicating that language models specifically finetuned for text embedding tasks are more suitable for our cases.

Reviewer comment (Evaluation):
Statistical significance tests are missing. It is unclear whether the gaps between SimTeG and the baselines are statistically significant or not. In fact, some gaps in Tables 1-3 are subtle and unlikely significant given the reported standard deviation.

Author response:
Thanks for the suggestions. We compute the p values for two comparisons, SimTeG v.s. baseline (GIANT/OGB) and GraphSAGE v.s. SOTA GNN on three datasets. As shown in the following second table where we compare between GraphSAGE and SOTA GNN, the p values of SimTeG are significantly smaller than the baseline embeddings. Specifically, the p values of SimTeG on OGBN-Arxiv and OGBL-Citation2 are close or larger than 0.05. This further supports our key findings: in cooperation with advanced text embeddings, one can bypass the necessity of using complex GNN models. We have also added the p-value results to the paper (Section A1.2).

| Dataset       | Backbone              | SimTEG | Baseline (GIANT/OGB) | P-Value         | P < 0.05 |
|---------------|---------------------------------|------------------------|----------------------|-----------------|----------|
| OGBN-Arxiv    | SOTA GNN   | 77.04               | 75.93             | 7.77e-14        | True     |
| OGBN-Arxiv    | GraphSAGE  | 76.84               | 73.70             | 4.11e-17        | True     |
| OGBN-Products | SOTA GNN   | 85.40               | 86.12             | 4.15e-06        | True     |
| OGBN-Products | GraphSAGE  | 84.59               | 82.84             | 9.01e-10        | True     |
| OGBL-Citation | SOTA GNN   | 91.42               | 90.92             | 0.0023          | True     |
| OGBL-Citation | GraphSAGE  | 91.62               | 85.13             | 5.01e-12        | True     |

| Dataset  | Method   | GraphSAGE | SOTA GNN | P-Value  | P < 0.05 |
|---------------|--------|--------|------|----------|----------|
| OGBN-Arxiv    | SimTEG   | 76.84 | 77.04  | 0.0427   | True     |
| OGBN-Arxiv    | Baseline  | 73.70 | 75.93   | 7.79e-22 | True     |
| OGBN-Products | SimTEG    | 84.59    | 85.40  | 8.74e-06 | True     |
| OGBN-Products | Baseline  | 82.84   | 85.40   | 7.87e-17 | True     |
| OGBL-Citation | SimTEG    | 91.62  | 91.42   | 0.1380   | False    |
| OGBL-Citation | Baseline  | 85.13    | 90.92    | 2.98e-15 | True     |

It is worth noting that as the results of GLEM are reported by the original paper and we do not have the results for each individual experiment, we are not able to compute the corresponding p values. We do acknowledge that there is a subtle difference between SimTeG and GLEM and GLEM outperforms SimTeG on OGBN-Products. This phenomenon is discussed in our observation 4 in Section 5.1 of the paper

Reviewer comment (Experiments):
An important baseline, GraphFormers [1], is not compared.

Author response:
Thanks for the suggestions. GraphFormer is closely related to our research line and we report the performance of GraphFormers as follows. The results of GraphFormer on OGBN-Arxiv and OGBN-Products are directly borrowed from the GLEM paper [1] since the datasets and split are exactly the same. We run GraphFormer on ogbl-citation2 for 10 times and report the mean with std. For the hyperparameter setting, we use the default parameters, and the batch size is set to 100 to make it consistent with the reported results in GLEM. As shown in the table, SimTeG performs consistently better than GraphFormer. It is possibly because 1) the GNN-nested architecture of GraphFormer solely allows 1-hop message passing, which limits the express ability of GNN models; 2) GraphFormer’s implementation modifies the architecture code of Bert and cannot be easily extended to other SOTA embedding models nowadays.

We have added the GraphFormer results to Table A8 in the paper.

| method | ogbn-arxiv | ogbn-products | ogbl-citation (MRR) |
|---|---|---|---|
|GraphFormer | 72.81 ± 0.20 | 74.72 ± 0.16 | 82.78 ± 0.24 |
| SimTeG + GraphSAGE | 76.84 ± 0.34 | 84.59 ± 0.44 | 85.13 ± 0.73 |
| SimTeG + SOTA GNN | 77.04 ± 0.13 | 85.40 ± 0.28 | 86.66 ± 1.21 |

Reviewer comment (Experiments):
Only LoRA is examined in the proposed method as the PEFT strategy. It is unclear whether other strategies, such as Prefix-Tuning and Adapter, can also help tackle the overfitting problem. If so, the observed necessity of PEFT would be strengthened.

Author response:
Many thanks for the suggestion. It would be beneficial to understand other PEFT strategies to check the effects. Due to the time limitation, we conducted the experiments for PEFT with IA3 [2] on OGBN-Arxiv. The results are as follows. As the case study showed, different PEFT may result in similar effects since all of them restrict the learnable parameters of LMs.

|   |  LoRA | IA3 |
|---|---|---|
| LM stage | 74.06 ± 0.13 | 74.23 ± 0.18  |
| GNN stage | 76.84 ± 0.34 | 76.54 ± 0.24  |

[1] Zhao, Jianan, et al. "Learning on large-scale text-attributed graphs via variational inference." arXiv preprint arXiv:2210.14709 (2022).
[2] Liu, Haokun, et al. "Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning." Advances in Neural Information Processing Systems 35 (2022): 1950-1965.