Paper: Efficient Large Language Models Fine-Tuning on Graphs (ICLR-2024)

Reviewer comment (Novelty):
The proposed method can only work for text-attributed graphs and node classification tasks. However, some recent works have validated that LLMs can process nearly all kinds of graphs and node-level/edge-level/graph-level tasks, which makes the contribution of this work being less significant.

Author response:
**Q1.** The proposed method can only work for text-attributed graphs and node classification tasks. However, some recent works have validated that LLMs can process nearly all kinds of graphs and node-level/edge-level/graph-level tasks, which makes the contribution of this work being less significant.

**A1.** We respectively disagree with this comment. We would like to point out that the proposed training GNN-LLM algorithm is general and can be used for any feature extraction backbones and downstream prediction tasks, enabling data and tasks beyond text-attributed graphs and node classification tasks.

First of all, our proposed method is a general end-to-end GNN training algorithm, which supports any feature extraction backbones that process various kinds of node attributes including but not limited to language, images, video and audio. It works by simply changing the LLM backbone to any feature extraction backbone. However, in this work, we focus on text-attributed graphs
due to the property of chosen benchmark graph datasets and to ensure a fair comparison with the baselines.

Second, our algorithm framework can support any downstream graph learning tasks. However, exploring LLM for text-attributed graphs is still a pretty new area, and most datasets for graph classification or link prediction do not provide the original raw text attributes. Therefore, we choose node classification tasks following existing state-of-the-art literature such as GIANT [1], GLEM
[2], and [3] to ensure a fair comparison.

To provide an example of link prediction, we run the proposed algorithm on Cora dataset. We manually partition the links into distinct sets for training, validation, and testing. This division is performed with two distinct split ratios: (1) a low ratio of 10/30/60 for train/valid/test, and (2) a high ratio of 85/5/10 for the same sets. We use GCN as the downstream GNN. To prioritize faster execution and simplicity, we choose DistilBert as the language model. As indicated in the following table, LEADING exhibits performance advantages over the baselines, especially in scenarios with limited training data. This aligns with the findings presented in our paper concerning node classification.

# Link prediction performance AUC (%) on Cora
| Methods     | Low     | High     |
|---------|---------|---------|
| Shallow Embedding  | 74.7  | 94.9  |
| Pre-trained DistilBERT  | 64.7 | 68.7  |
| Supervised-FT DistilBERT  | 66.3  | 89.4  |
| LEADING(DistilBERT)  | 81.8  | 95.2  |

Third, we provide a comprehensive survey of related works in our paper and no existing work can achieve end-to-end fine-tuning of LLMs on graphs. Our proposed algorithm is the first to achieve this goal and shows tremendously improved scalability: the training cost of LEADING is close to the training of LLMs without graphs (refer to the scalability analysis in Table 2 in our submission), which is not thought to be possible in the literature. Our approach significantly minimizes computation redundancy as shown in Fig. 3 and Fig. 4 in our paper.

To summarize, our algorithm is general and novel. It makes significant contributions to the end-to-end training of GNN-LLMs. Please kindly let us know the specific references that use raw text language attributes to complete ”nearly all kinds of graphs and node-level/edge-level/graph-level tasks”, and we will be happy to provide a discussion and comparison.

[1] Chien, Eli, et al. ”Node feature extraction by self-supervised multi-scale neighborhood prediction.” arXiv preprint arXiv:2111.00064 (2021).

[2] Zhao, Jianan, et al. ”Learning on large-scale text-attributed graphs via variational inference.” arXiv preprint arXiv:2210.14709 (2022).

[3] Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs, arXiv:2023.07

Reviewer comment (Evaluation):
The performance of the proposed method is not superior. For example, on Cora dataset, the proposed method equipped with LLMs is inferior to the Shallow Embedding.

Author response:
**Q2.** The performance of the proposed method is not superior. For example, on Cora dataset, the proposed method equipped with LLMs are inferior to the Shallow Embedding.

**A2.** We would like to point out that the proposed LEADING method outperforms all LLM baselines on all datasets. Moreover, LEADING outperforms ”Shallow Embedding” by 4% and around 2% on ogbn-arxiv and PubMed, respectively. We also provide a further experiment based on ogbn-products. For simplicity, we utilize GraphSAGE as the downstream GNN backbone in this context. 

In the presented table, LEADING exhibits superior performance compared to all baselines, particularly surpassing shadow embedding by 4.4%. It's noteworthy that GIANT encounters memory issues in our experiments due to high training costs. As a result, we report the accuracy of the well-trained model available from their official repository. These improvements already verify the superiority of our LEADING method.

# Performance and computation cost comparisons on ogbn-products

| Methods     | Accuracy(%)     | Running Time(s)    | Peak Memory Usage(GB)    |
|---------|---------|---------|---------|
| Shallow Embedding| 79.7  | ---  | ---  |
| Pre-trained DeBERTa  | 62.0  | ---  | ---  |
| Supervised-FT DeBERTa  | 82.2  | 67200 | 25.1  |
| GIANT | 83.2  | N/A | OOM |
| GLEM (DeBERTa)  | 83.2  | 356648 | 25.2 |
| LEADING (DeBERTa)  | 84.1  | 124164 | 25.6 |

For Cora, we believe adding more training data or using a better LLM model can outperform Shallow Embedding. To validate this, we conducted the same experiments using SentenceBERT on Cora and Pubmed with the same low labeling rate. The following table shows it is easy to beat Shallow Embedding, which also verifies the superiority of the proposed LEADING method.

# Performance(%) comparison using SentenceBERT

| Methods     | GCN(Cora)     | GAT(Cora)    | GCN(Pubmed)  | GAT(Pubmed)    |
|---------|---------|---------|---------|---------|
| Shallow Embedding  | $82.0 \pm 0.7$ | $82.3 \pm 0.7$  |$78.9 \pm 2.0$  |$77.7 \pm 0.9$  |
| LEADING (SBERT)   | $\mathbf{83.3 \pm 0.3}$  | $\mathbf{83.8 \pm 0.5}$  |$\mathbf{80.5 \pm 0.4}$  |$\mathbf{80.2 \pm 0.8}$  |

For high labeling rate case, we conduct statistical significance tests on ogbn-arxiv.  We perform 5 times experiments on both our proposed algorithm and the best baselines, GLEM, which has the closest performance as ours on ogbn-arxiv, here we show the results of those performances. The p-value is 9.85e-06 for Rev-GAT and 1.64e-05 for GCN, much less than the commonly used desired significance level 0.01 and highly significance level 0.001. It clearly show that we can reject the null hypothesis. The difference is highly significant.

# Performance(%) on ogbn-arxiv

| Runs | GLEM (GCN)   | LEADING(GCN)     | GLEM (Rev-GAT)      | LEADING(Rev-GAT)   |
|---------|---------|---------|---------|---------|
| 1 | 76.90  | 77.42  | 75.90  |76.18  |
| 2 | 76.80  | 77.30  | 75.95  |76.10 |
| 3 | 77.00  | 77.26  | 75.93  |76.08  |
| 4 | 76.85 | 77.34  | 75.86  |76.13 |
| 5 | 76.78  | 77.34  | 75.87  |76.12 |

Reviewer comment (Experiments):
The experiments are somewhat limited. Specifically, the authors only evaluate the proposed method in homophilous graphs. I notice that some concurrent works [1,2] in LLMs for graphs report the results on heterophilous graphs. So, how about the performance of the LEADING algorithm perform on heterophilous graphs?

Author response:
**Q3.** The experiments are somewhat limited. Specifically, the authors only evaluate the proposed method in homophilous graphs. I notice that some concurrent works [1,2] in LLMs for graphs report the results on heterophilous graphs. So, how about the performance of the LEADING algorithm perform on heterophilous graphs?/How about the performance of the LEADING algorithm
perform on heterophilous graphs?


**A3.** Thanks for your valuable suggestion. Exploring LLM for text-attributed graphs is still a pretty new area, after an extensive search, all benchmark dataset for heterophilic graphs do not provide the original raw text attributes. Therefore, we choose homophilous graphs following existing state-of-the-art literature to ensure a fair comparison and demonstrate our improvements in efficiency and accuracy. For those two cited papers in your comment, GraphText[1] only employs homophilous graphs Cora and Citeseer as TAGs (please refer to the appendix A.2) because it doesn’t require the graph has raw text attributes; and [2] restricts its analysis to some specific nodes which have heterophilous property in homophilous graph Pubmed to draw conclusions. We would be
pleased to incorporate additional experiments once such graph datasets with raw text attributes become available.

[1] GRAPHTEXT: GRAPH REASONING IN TEXT SPACE (arXiv:2310.01089v1)

[2] Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs（arXiv:2307.03393v3）

Reviewer comment (Experiments):
The authors only use some small and out-of-date language models (BERT and DeBERTa). Why not try more powerful LLMs such as GPT-3.5 and LLaMA [3]?

Author response:
**Q4.** The authors only use some small and out-of-date language models (BERT and DeBERTa). Why not try more powerful LLMs such as GPT-3.5 and LLaMA?

**A4.** Thanks for this nice suggestion. We would like to point out that the proposed LEADING algorithm is versatile and applicable to any LLM architecture. We choose BERT and DeBERTa as the backbone language models in our initial submission to ensure a fair comparison with state-of-the- art baselines since these are the models they used for evaluation. For instance, we employed BERT
following GIANT and DeBERTa following GLEM. 

Following your suggestion, we are happy to provide further experiments on LEADING using larger models. Based on our current computation resources and the code availability of those larger language models, we choose to fine-tune GPT-2 on Cora and ogbn-arxiv datasets. The results shown in the following table indicate that the proposed LEADING algorithm effectively fine-tunes GPT-
2 to achieve better performance, which is consistent with our experiments on other language models. Regarding the computation cost, LEADING is capable of maintaining computational costs nearly identical to supervised fine-tuning of GPT without graphs. The additional running time arises due to the sequential execution of two pipelines in LEADING, yet this can be effectively mitigated through parallel computing. It incurs significantly less computation overhead or memory cost compared to baselines such as GLEM. GIANT runs out of memory in our experiments. It's crucial to emphasize that enhancing model size may not be essential for achieving superior performance, as evident in the comparison between the results of GPT and the DeBERTa presented in our submission. The effectiveness of fine-tuning is influenced by a range of factors beyond mere model size. Note that we found that in the low labeling case because of the poor quality of generated pseudo labels, GLEM actually achieves its best accuracy in a very special case when the ratio of pseudo labels is set to be 0, this will reduce GLEM to supervised FT LLM, hence we report the same performance as supervised FT case.

Besides, Meta requires a request form to access their Llama model parameters. We submitted the request a week ago but have not received a response yet. We will add new results on Llama once we get access to it. We will provide a complete evaluation of these models in our revision.

# LEADING performance(%) with GPT-2
| Method | GCN(Cora) | GAT(Cora)|GCN(Arxiv)| Rev-GAT(Arxiv)|
|----------|----------|----------|----------|----------|
| Pre-trained GPT-2   | 51.9   | 54.7   | 64.8  | 66.9   |
| Supervised-FT GPT-2   | 70.8   | 71.7   | 73.2   | 73.8   |
| GLEM(GPT-2)   | 70.8   | 71.7   | 74.0   | 75.1   |
| LEADING(GPT-2)   | 80.5   | 81.5   | 74.1   | 75.2   |

# Scalability comparison with GPT-2 on ogbn-arxiv
| Method | Memory(GB) | Running Time(s) | 
|----------|----------|----------|
| Supervised-FT GPT-2    | 26.8   | 15555   | 
| GIANT   | OOM  | N/A   | 
| GLEM(GPT-2)   | 26.8   | 82930   | 
| LEADING(GPT-2)   | 27.1   | 27920  |

Reviewer comment (Reproducibility):
The codes for reproducing the results are not provided.

Author response:
**Q5.** The codes for reproducing the results are not provided.

**A5.** We would like to kindly point out that the code is not mandatory at the submission stage. However, we will be happy to make our code available upon the acceptance of our work. Thanks for your understanding.

To conclude, we believe we have fully addressed all of your concerns. Please kindly let us know if you have any further concerns.

Reviewer comment (Experiments):
How about the performance of the LEADING algorithm perform on heterophilous graphs?

Author response:
**Q3.** The experiments are somewhat limited. Specifically, the authors only evaluate the proposed method in homophilous graphs. I notice that some concurrent works [1,2] in LLMs for graphs report the results on heterophilous graphs. So, how about the performance of the LEADING algorithm perform on heterophilous graphs?/How about the performance of the LEADING algorithm
perform on heterophilous graphs?


**A3.** Thanks for your valuable suggestion. Exploring LLM for text-attributed graphs is still a pretty new area, after an extensive search, all benchmark dataset for heterophilic graphs do not provide the original raw text attributes. Therefore, we choose homophilous graphs following existing state-of-the-art literature to ensure a fair comparison and demonstrate our improvements in efficiency and accuracy. For those two cited papers in your comment, GraphText[1] only employs homophilous graphs Cora and Citeseer as TAGs (please refer to the appendix A.2) because it doesn’t require the graph has raw text attributes; and [2] restricts its analysis to some specific nodes which have heterophilous property in homophilous graph Pubmed to draw conclusions. We would be
pleased to incorporate additional experiments once such graph datasets with raw text attributes become available.

[1] GRAPHTEXT: GRAPH REASONING IN TEXT SPACE (arXiv:2310.01089v1)

[2] Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs（arXiv:2307.03393v3）

Reviewer comment (Experiments):
Why not try more powerful LLMs such as GPT-3.5 and LLaMA?

Author response:
**Q4.** The authors only use some small and out-of-date language models (BERT and DeBERTa). Why not try more powerful LLMs such as GPT-3.5 and LLaMA?

**A4.** Thanks for this nice suggestion. We would like to point out that the proposed LEADING algorithm is versatile and applicable to any LLM architecture. We choose BERT and DeBERTa as the backbone language models in our initial submission to ensure a fair comparison with state-of-the- art baselines since these are the models they used for evaluation. For instance, we employed BERT
following GIANT and DeBERTa following GLEM. 

Following your suggestion, we are happy to provide further experiments on LEADING using larger models. Based on our current computation resources and the code availability of those larger language models, we choose to fine-tune GPT-2 on Cora and ogbn-arxiv datasets. The results shown in the following table indicate that the proposed LEADING algorithm effectively fine-tunes GPT-
2 to achieve better performance, which is consistent with our experiments on other language models. Regarding the computation cost, LEADING is capable of maintaining computational costs nearly identical to supervised fine-tuning of GPT without graphs. The additional running time arises due to the sequential execution of two pipelines in LEADING, yet this can be effectively mitigated through parallel computing. It incurs significantly less computation overhead or memory cost compared to baselines such as GLEM. GIANT runs out of memory in our experiments. It's crucial to emphasize that enhancing model size may not be essential for achieving superior performance, as evident in the comparison between the results of GPT and the DeBERTa presented in our submission. The effectiveness of fine-tuning is influenced by a range of factors beyond mere model size. Note that we found that in the low labeling case because of the poor quality of generated pseudo labels, GLEM actually achieves its best accuracy in a very special case when the ratio of pseudo labels is set to be 0, this will reduce GLEM to supervised FT LLM, hence we report the same performance as supervised FT case.

Besides, Meta requires a request form to access their Llama model parameters. We submitted the request a week ago but have not received a response yet. We will add new results on Llama once we get access to it. We will provide a complete evaluation of these models in our revision.

# LEADING performance(%) with GPT-2
| Method | GCN(Cora) | GAT(Cora)|GCN(Arxiv)| Rev-GAT(Arxiv)|
|----------|----------|----------|----------|----------|
| Pre-trained GPT-2   | 51.9   | 54.7   | 64.8  | 66.9   |
| Supervised-FT GPT-2   | 70.8   | 71.7   | 73.2   | 73.8   |
| GLEM(GPT-2)   | 70.8   | 71.7   | 74.0   | 75.1   |
| LEADING(GPT-2)   | 80.5   | 81.5   | 74.1   | 75.2   |

# Scalability comparison with GPT-2 on ogbn-arxiv
| Method | Memory(GB) | Running Time(s) | 
|----------|----------|----------|
| Supervised-FT GPT-2    | 26.8   | 15555   | 
| GIANT   | OOM  | N/A   | 
| GLEM(GPT-2)   | 26.8   | 82930   | 
| LEADING(GPT-2)   | 27.1   | 27920  |

Reviewer comment (Experiments):
This work mainly studies encoder-only PLMs such as BERT and DeBERTa. I do not see how the entire study can be easily generalized to encoder-decoder and decoder-only PLMs from the perspective of either methodologies or evaluation protocols. Therefore, I do not quite agree with the term "Large Language Models" in the title because encoder-only PLMs are usually much smaller than encoder-decoder and decoder-only ones. The study is still a complete one even if only encoder-only PLMs are studied, but the used term somehow overclaims what this study has done.

Author response:
**A1.** Thanks for this nice suggestion. We would like to point out that the proposed LEADING algorithm is versatile and applicable to any LLM architecture. We choose BERT and DeBERTa as the backbone language models in our initial submission to ensure a fair comparison with state-of-the- art baselines since these are the models they used for evaluation. For instance, we employed BERT following GIANT and DeBERTa following GLEM. 

Following your suggestion, we are happy to provide further experiments on LEADING using other transformers. We select one of the decoder only language models, GPT-2, conducted similar experiments on the cora and ogbn-arxiv dataset. Due to the utilization of causal (auto-regressive) attention in the decoder-only model, we opt to use the last token for classification instead of the [CLS] token in encoder-only model. One simple example can be found here [1]. The results shown in the following table indicate that the proposed LEADING algorithm effectively fine-tunes GPT-2 to achieve better performance, which is consistent with our experiments on other language models. Regarding the computation cost, LEADING is capable of maintaining computational costs nearly identical to supervised fine-tuning of GPT without graphs. The additional running time arises due to the sequential execution of two pipelines in LEADING, yet this can be effectively mitigated through parallel computing. It incurs significantly less computation overhead or memory cost compared to baselines such as GLEM. GIANT runs out of memory in our experiments. Note that we found that in the low labeling case because of the poor quality of generated pseudo labels, GLEM actually achieves its best accuracy in a very special case when the ratio of pseudo labels is set to be 0, this will reduce GLEM to supervised FT LLM, hence we report the same performance as supervised FT case.

Besides, Meta requires a request form to access their Llama model parameters. We submitted the request a week ago but have not received a response yet. We will add new results on Llama once we get access to it. We will provide a complete evaluation of these models in our revision.

# LEADING performance(%) with GPT-2
| Method | GCN(Cora) | GAT(Cora)|GCN(Arxiv)| Rev-GAT(Arxiv)|
|----------|----------|----------|----------|----------|
| Pre-trained GPT-2   | 51.9   | 54.7   | 64.8  | 66.9   |
| Supervised-FT GPT-2   | 70.8   | 71.7   | 73.2   | 73.8   |
| GLEM(GPT-2)   | 70.8   | 71.7   | 74.0   | 75.1   |
| LEADING(GPT-2)   | 80.5   | 81.5   | 74.1   | 75.2   |

# Scalability comparison with GPT-2 on ogbn-arxiv
| Method | Memory(GB) | Running Time(s) | 
|----------|----------|----------|
| Supervised-FT GPT-2    | 26.8   | 15555   | 
| GIANT   | OOM  | N/A   | 
| GLEM(GPT-2)   | 26.8   | 82930   | 
| LEADING(GPT-2)   | 27.1   | 27920  |

[1] https://huggingface.co/docs/transformers/model_doc/gpt2

Reviewer comment (Evaluation):
Statistical significance tests are missing. It is unclear whether the gaps between LEADING and the baselines are statistically significant or not. In fact, the gaps on arXiv are quite subtle in Table 1, therefore p-values should be reported.

Author response:
**A2.** Thanks for your suggestion. We perform 5 times experiments on both our proposed algorithm and the best baselines, GLEM, which has the closest performance as ours on ogbn-arxiv, here we show the results of those performances. The p-value is 9.85e-06 for Rev-GAT and 1.64e-05 for GCN, much less than the commonly used desired significance level 0.01 and highly significance level 0.001. It clearly show that we can reject the null hypothesis. The difference is highly significant.

# Performance(%) on ogbn-arxiv

| Runs | GLEM (GCN)   | LEADING(GCN)     | GLEM (Rev-GAT)      | LEADING(Rev-GAT)   |
|---------|---------|---------|---------|---------|
| 1 | 76.90  | 77.42  | 75.90  |76.18  |
| 2 | 76.80  | 77.30  | 75.95  |76.10 |
| 3 | 77.00  | 77.26  | 75.93  |76.08  |
| 4 | 76.85 | 77.34  | 75.86  |76.13 |
| 5 | 76.78  | 77.34  | 75.87  |76.12 |

Reviewer comment (Experiments):
An important baseline, GraphFormers [1], is cited but not compared.

Author response:
**A3.** Thanks for your comments. We do not report the performance of GraphFormer due to the following reasons:

(1) GraphFormer faces severe scalability issues since it requires the encoding of all neighboring nodes. This results in significant computation overhead, and we can not run it on large-scale datasets.

(2) The focus of our paper is to provide a flexible LLM fine-tuning framework for arbitrary LLM backbones, but
Graphformer is a specialized architecture. Therefore, it is hard to make fair and meaningful comparisons. 

Due to these reasons, the papers of baselines we compare with do not report the performance of GraphFormer, and we majorly follow their evaluation settings to ensure fair comparisons.

Reviewer comment (Evaluation):
To better understand the proposed techniques and their impact on data efficiency, it would be beneficial to delve into the underlying mechanisms through analysis or intuition.

Author response:
**A1.** Thanks for your suggestion. Please allow us to clarify that our algorithm improves knowledge transfer from LLMs to downstream tasks, particularly in scenarios with limited data, resulting in significant data efficiency. This is achieved by supporting end-to-end fine-tuning directly on the downstream tasks, a feature not supported by existing algorithms.

At the beginning of Section 3, we point out that existing algorithms, such as GIANT and GLEM, can not be trained or fine-tuned in an end-to-end manner for the downstream tasks. For instance, GIANT pre-trains the LLMs using self-supervised learning, but their LLMs can not be trained on the downstream tasks due to scalability issues of existing algorithms. Therefore, we believe that due to the mismatch between training and testing, they can not effectively transfer the knowledge of LLMs to downstream tasks, especially when the training data is limited, as shown in our experiments.

Nevertheless, there is no existing work in the literature can support the end-to-end training of LLM-GNN. Therefore, the major motivation of our work is to design an efficient and scalable algorithm to support the end-to-end training of LLM-GNN, which enables the LLM models to be optimized directly for the downstream task, leading to improved knowledge transfer and performance.

Specifically, we first analyze the scalability challenges in end-to-end training of LLM-GNN and identify two primary bottlenecks in Section 3.1: encoding redundancy and propagation redundancy. The proposed techniques including neighbor decoupling and implicit modeling are to solve these two challenges in Section 3.2 and Section 3.3 respectively. The proposed LEADING algorithm is the first to achieve end-to-end training of LLM-GNN, and it shows tremendously improved scalability: the training cost of LEADING is close to the training of LLMs without graphs (refer to the scalability analysis in Table 2 in our submission).

Reviewer comment (Experiments):
In order to fully grasp the importance of the techniques, it is advisable to conduct ablation studies that elucidate their individual contributions.

Author response:
**A2.**  Thanks again for your suggestion. As discussed in previous answer, the improvements in prediction accuracy come from the improved scalability to support end-to-end training. Therefore, we would like to provide an ablation study to elucidate the individual contributions of the techniques in terms of algorithm efficiency (running time and memory cost).

We employ the APPNP as GNN backbones and DeBERTa as the language model for our experiments on ogbn-arxiv. We conduct tests in four settings:
1. DeBERTa + APPNP (batch size 32)
2. DeBERTa + APPNP + Neighbor Decoupling (batch size 32)
3. DeBERTa + Implicit Modeling + Neighbor Decoupling (batch size 32)
4. DeBERTa + Implicit Modeling + Neighbor Decoupling (batch size 128)

For consistency, we use  two-hop neighbors(5 neighbors for each hop) for all cases. The obtained results reveal that both neighbor
decoupling and implicit modeling play crucial roles in reducing memory costs. The additional running time is attributed to the sequential execution of two pipelines in our algorithm. This issue can be easily mitigated through parallel training. Moreover, our LEADING algorithm can use a much larger batch size (128) to reduce the running time due to the significantly reduced memory cost.
This ablation study and analysis elucidates the contributions of our techniques and we will incorporate them into our revision.

# Computation cost comparison on ogbn-arxiv

| Methods    | Running time per epoch(s)     | Peak memory usage(MB)    |
|---------|---------|---------|
| DeBERTa+APPNP(BS:32)  | 5770  | 47243  |
| DeBERTa+APPNP + Neighbor Decoupling(BS:32)  | 12117  | 8522  |
| DeBERTa+ Neighbor Decoupling + Implicit Modeling(BS:32)  | 12117  | 6259  |
| DeBERTa+ Neighbor Decoupling + Implicit Modeling(BS:128)  | 2996  | 25103  |

Reviewer comment (Experiments):
To demonstrate the scalability of the algorithm, a comprehensive evaluation on a range of graph structures and larger scale datasets would be advantageous.

Author response:
**A3.** Thanks for your suggestion. Here, we present the performance results for ogbn-products, which has over 2.4 M nodes, much larger than ogbn-arxiv. We also provide the associated computational costs. For simplicity, we utilize GraphSAGE as the downstream GNN backbone in this context.

From the table, in terms of performance, LEADING outperforms all baselines, including cascaded training (Supervised-FT), self-supervised training (GIANT) and iterative training (GLEM). Regarding computational costs, our findings align with those in our papers: LEADING requires nearly identical memory compared with supervised FT without a graph, and it significantly outperforms
GIANT and GLEM. It is worth noting that GIANT runs out of memory in our experiment due to its high training costs, so we report the accuracy of the well-trained model provided by their official repository.

# Performance and computation cost comparisons on ogbn-products

| Methods     | Accuracy(%)     | Running Time(s)    | Peak Memory Usage(GB)    |
|---------|---------|---------|---------|
| Shallow Embedding| 79.7  | ---  | ---  |
| Pre-trained DeBERTa  | 62.0  | ---  | ---  |
| Supervised-FT DeBERTa  | 82.2  | 67200 | 25.1  |
| GIANT | 83.2  | N/A | OOM |
| GLEM (DeBERTa)  | 83.2  | 356648 | 25.2 |
| LEADING (DeBERTa)  | 84.1  | 124164 | 25.6 |

Reviewer comment (Novelty):
In the work section, it would be beneficial to provide a comprehensive context by discussing existing approaches for efficient training of Graph Neural Networks (GNNs) and implicit models.

Author response:
**A4.** Thanks for your suggestion. Here we provide a brief introduction about the related work on those efficient training algorithm, and we will add a more comprehensive study to our revision later:

Currently, efficient training strategies for GNNs can be broadly classified into three categories. First, sampling methods adopt mini-batch training strategies to mitigate computation and memory requirements by sampling nodes and edges, mitigating the neighbor explosion issues through techniques like neighbor sampling or feature memory updates. Second, pre-computing or post-computing
methods separate feature aggregation and prediction models into distinct stages, involving actions such as pre-computing feature aggregation before training or post-processing with label propagation after training. Finally, distributed methods distributes large graphs across multiple servers, parallelizing GNNs training to enhance scalability and efficiency. However, these techniques are not sufficient to support end-to-end GNN-LLM training due to the high computation and memory costs of both. Therefore, existing works choose self-supervised training (GIANT) or iterative training (GLEM), which avoids end-to-end training of LLM and GNNs.

Regarding implicit models, implicit models such as DEQs, Neural ODES and IGNN draw significant research attention due to their memory efficiency. DEQs introduce the concept of deep equilibrium, where the model seeks a fixed-point solution iteratively. It models the forward propagation as solving a fixed point equation. Neural ODEs treat neural networks as continuous dynamical systems, utilizing ordinary differential equations to model the evolution of hidden states. This continuous-depth representation enables seamless integration of depth, offering a flexible and computationally efficient alternative to traditional discrete layer-wise architectures. Both DEQ and Neural ODE can achieve constant memory costs because they do not need to maintain intermediate hidden layers. By using implicit modeling, LEADING also achieves small memory costs, and it can also reduce computation cost by reducing computation redundancy through historical memory.

Reviewer comment (Writing):
Certain parts of the explanation could benefit from in-depth details or intuitive explanations, especially when describing how the techniques enhance data efficiency.

Author response:
**A5.** Similar as answer 1, the end-to-end GNN-LMM training enhances the knowledge transfer from LLM to the downstream GNN tasks, which contributes to the improvement of data efficiency. However, no existing works can address the significant computation overhead brought by the end to end training LLM and GNN. We explore the primary challenges arising from encoding redundancy
and propagation redundancy. Therefore, we propose two techniques, neighbor decoupling and implicit modeling to solve these issues. The primary objective of neighbor decoupling is to address encoding redundancy efficiently. In a normal coupling case, each node serves as a target node once and neighbor nodes many times per epoch (please refer to Fig.1 in our submission), However, our decoupling strategy introduces two distinct pipelines: each node is only encoded as a target node once in pipeline1 and as neighbor nodes once in pipeline2 (please refer to Fig.2), this can significantly reduce redundant encoding times.

Implicit modeling aims to alleviate propagation redundancy. It considers forward propagation as the solution to a fixed-point equation, therefore the feature aggregation can be utilized in just one layer, and the gradient computation involving straightforward first-order derivatives of this equation, there is no need to store intermediate values, in contrast to conventional neural networks. Consequently, the memory complexity remains constant and independent of the number of layers employed. This characteristic represents a notable advantage in reducing memory and computation overhead associated with deep architectures.

Reviewer comment (Experiments):
By expanding the scope of the ablation study and conducting experiments, valuable insights into these proposed methods can be gained.

Author response:
**A6.** Thank you for your valuable suggestion. Regarding the ablation study of the two proposed techniques, we have addressed this in detail in response 2. In the provided table, we illustrate the individual contributions of each technique to our efficiency improvements. The table outlines the impact of each method on the overall enhancement in computation efficiency.

In relation to large datasets, we have conducted additional experiments on ogbn-products, as outlined in response 3. In these experiments, our proposed algorithm demonstrates superior performance compared to all state-of-the-art baselines while maintaining nearly identical computational costs to supervised fine-tuning without graphs. We will incorporate all the results of these experiments
into our revisions.

Reviewer comment (Theory):
The analysis findings indicate that there is redundancy in encoding and propagation. It is not clear how neighbor decoupling and implicit modeling specifically contribute to improved knowledge transfer from LLMs. Could you provide some insight or analysis to explain the underlying mechanisms?

Author response:
**A1.** Thanks for your suggestion. Please allow us to clarify that our algorithm improves knowledge transfer from LLMs to downstream tasks, particularly in scenarios with limited data, resulting in significant data efficiency. This is achieved by supporting end-to-end fine-tuning directly on the downstream tasks, a feature not supported by existing algorithms.

At the beginning of Section 3, we point out that existing algorithms, such as GIANT and GLEM, can not be trained or fine-tuned in an end-to-end manner for the downstream tasks. For instance, GIANT pre-trains the LLMs using self-supervised learning, but their LLMs can not be trained on the downstream tasks due to scalability issues of existing algorithms. Therefore, we believe that due to the mismatch between training and testing, they can not effectively transfer the knowledge of LLMs to downstream tasks, especially when the training data is limited, as shown in our experiments.

Nevertheless, there is no existing work in the literature can support the end-to-end training of LLM-GNN. Therefore, the major motivation of our work is to design an efficient and scalable algorithm to support the end-to-end training of LLM-GNN, which enables the LLM models to be optimized directly for the downstream task, leading to improved knowledge transfer and performance.

Specifically, we first analyze the scalability challenges in end-to-end training of LLM-GNN and identify two primary bottlenecks in Section 3.1: encoding redundancy and propagation redundancy. The proposed techniques including neighbor decoupling and implicit modeling are to solve these two challenges in Section 3.2 and Section 3.3 respectively. The proposed LEADING algorithm is the first to achieve end-to-end training of LLM-GNN, and it shows tremendously improved scalability: the training cost of LEADING is close to the training of LLMs without graphs (refer to the scalability analysis in Table 2 in our submission).

Reviewer comment (Experiments):
The ablation study seems to have limitations. It would be beneficial to conduct ablation experiments to assess the individual contributions of neighbor decoupling and implicit modeling towards the observed improvements.

Author response:
**A2.**  Thanks again for your suggestion. As discussed in previous answer, the improvements in prediction accuracy come from the improved scalability to support end-to-end training. Therefore, we would like to provide an ablation study to elucidate the individual contributions of the techniques in terms of algorithm efficiency (running time and memory cost).

We employ the APPNP as GNN backbones and DeBERTa as the language model for our experiments on ogbn-arxiv. We conduct tests in four settings:
1. DeBERTa + APPNP (batch size 32)
2. DeBERTa + APPNP + Neighbor Decoupling (batch size 32)
3. DeBERTa + Implicit Modeling + Neighbor Decoupling (batch size 32)
4. DeBERTa + Implicit Modeling + Neighbor Decoupling (batch size 128)

For consistency, we use  two-hop neighbors(5 neighbors for each hop) for all cases. The obtained results reveal that both neighbor
decoupling and implicit modeling play crucial roles in reducing memory costs. The additional running time is attributed to the sequential execution of two pipelines in our algorithm. This issue can be easily mitigated through parallel training. Moreover, our LEADING algorithm can use a much larger batch size (128) to reduce the running time due to the significantly reduced memory cost.
This ablation study and analysis elucidates the contributions of our techniques and we will incorporate them into our revision.

# Computation cost comparison on ogbn-arxiv

| Methods    | Running time per epoch(s)     | Peak memory usage(MB)    |
|---------|---------|---------|
| DeBERTa+APPNP(BS:32)  | 5770  | 47243  |
| DeBERTa+APPNP + Neighbor Decoupling(BS:32)  | 12117  | 8522  |
| DeBERTa+ Neighbor Decoupling + Implicit Modeling(BS:32)  | 12117  | 6259  |
| DeBERTa+ Neighbor Decoupling + Implicit Modeling(BS:128)  | 2996  | 25103  |

Reviewer comment (Experiments):
Have you explored graphs or additional datasets beyond those that were tested? Conducting experiments on a larger scale could highlight the scalability benefits more effectively.

Author response:
**A3.** Thanks for your suggestion. Here, we present the performance results for ogbn-products, which has over 2.4 M nodes, much larger than ogbn-arxiv. We also provide the associated computational costs. For simplicity, we utilize GraphSAGE as the downstream GNN backbone in this context.

From the table, in terms of performance, LEADING outperforms all baselines, including cascaded training (Supervised-FT), self-supervised training (GIANT) and iterative training (GLEM). Regarding computational costs, our findings align with those in our papers: LEADING requires nearly identical memory compared with supervised FT without a graph, and it significantly outperforms
GIANT and GLEM. It is worth noting that GIANT runs out of memory in our experiment due to its high training costs, so we report the accuracy of the well-trained model provided by their official repository.

# Performance and computation cost comparisons on ogbn-products

| Methods     | Accuracy(%)     | Running Time(s)    | Peak Memory Usage(GB)    |
|---------|---------|---------|---------|
| Shallow Embedding| 79.7  | ---  | ---  |
| Pre-trained DeBERTa  | 62.0  | ---  | ---  |
| Supervised-FT DeBERTa  | 82.2  | 67200 | 25.1  |
| GIANT | 83.2  | N/A | OOM |
| GLEM (DeBERTa)  | 83.2  | 356648 | 25.2 |
| LEADING (DeBERTa)  | 84.1  | 124164 | 25.6 |

Reviewer comment (Novelty):
It would be advantageous to expand upon the work section by discussing approaches such as sampling methods for GNN training and explicit models like Neural ODE/DEQ, providing more context and motivation for the techniques employed in your research.

Author response:
**A4.** Thanks for your suggestion. Here we provide a brief introduction about the related work on those efficient training algorithm, and we will add a more comprehensive study to our revision later:

Currently, efficient training strategies for GNNs can be broadly classified into three categories. First, sampling methods adopt mini-batch training strategies to mitigate computation and memory requirements by sampling nodes and edges, mitigating the neighbor explosion issues through techniques like neighbor sampling or feature memory updates. Second, pre-computing or post-computing
methods separate feature aggregation and prediction models into distinct stages, involving actions such as pre-computing feature aggregation before training or post-processing with label propagation after training. Finally, distributed methods distributes large graphs across multiple servers, parallelizing GNNs training to enhance scalability and efficiency. However, these techniques are not sufficient to support end-to-end GNN-LLM training due to the high computation and memory costs of both. Therefore, existing works choose self-supervised training (GIANT) or iterative training (GLEM), which avoids end-to-end training of LLM and GNNs.

Regarding implicit models, implicit models such as DEQs, Neural ODES and IGNN draw significant research attention due to their memory efficiency. DEQs introduce the concept of deep equilibrium, where the model seeks a fixed-point solution iteratively. It models the forward propagation as solving a fixed point equation. Neural ODEs treat neural networks as continuous dynamical systems, utilizing ordinary differential equations to model the evolution of hidden states. This continuous-depth representation enables seamless integration of depth, offering a flexible and computationally efficient alternative to traditional discrete layer-wise architectures. Both DEQ and Neural ODE can achieve constant memory costs because they do not need to maintain intermediate hidden layers. By using implicit modeling, LEADING also achieves small memory costs, and it can also reduce computation cost by reducing computation redundancy through historical memory.

Reviewer comment (Writing):
Some sections of the paper lack sufficient details. For instance, the explanation of how your techniques reduce redundancy can be vague at times. Providing details or intuitive explanations would enhance comprehension.

Author response:
**A5.** Similar as answer 1, the end-to-end GNN-LMM training enhances the knowledge transfer from LLM to the downstream GNN tasks, which contributes to the improvement of data efficiency. However, no existing works can address the significant computation overhead brought by the end to end training LLM and GNN. We explore the primary challenges arising from encoding redundancy
and propagation redundancy. Therefore, we propose two techniques, neighbor decoupling and implicit modeling to solve these issues. The primary objective of neighbor decoupling is to address encoding redundancy efficiently. In a normal coupling case, each node serves as a target node once and neighbor nodes many times per epoch (please refer to Fig.1 in our submission), However, our decoupling strategy introduces two distinct pipelines: each node is only encoded as a target node once in pipeline1 and as neighbor nodes once in pipeline2 (please refer to Fig.2), this can significantly reduce redundant encoding times.

Implicit modeling aims to alleviate propagation redundancy. It considers forward propagation as the solution to a fixed-point equation, therefore the feature aggregation can be utilized in just one layer, and the gradient computation involving straightforward first-order derivatives of this equation, there is no need to store intermediate values, in contrast to conventional neural networks. Consequently, the memory complexity remains constant and independent of the number of layers employed. This characteristic represents a notable advantage in reducing memory and computation overhead associated with deep architectures.

Reviewer comment (Experiments):
To further strengthen the paper, consider expanding the ablation study, incorporating references to work conducting large scale experiments, and providing additional intuition and details where necessary.

Author response:
**A6.** Thank you for your valuable suggestion. Regarding the ablation study of the two proposed techniques, we have addressed this in detail in response 2. In the provided table, we illustrate the individual contributions of each technique to our efficiency improvements. The table outlines the impact of each method on the overall enhancement in computation efficiency.

In relation to large datasets, we have conducted additional experiments on ogbn-products, as outlined in response 3. In these experiments, our proposed algorithm demonstrates superior performance compared to all state-of-the-art baselines while maintaining nearly identical computational costs to supervised fine-tuning without graphs. We will incorporate all the results of these experiments
into our revisions.

Reviewer comment (Experiments):
The main claim of this paper is on large language models whereas the experiments are conducted on small language models like "BERT" and "DeBERTa". Given the parameter-efficient tuning, it's not that hard to perform experiments on large models such as GPT-2 and Llama-2.

Author response:
**Q1.** The main claim of this paper is on large language models whereas the experiments are conducted on small language models like ”BERT” and ”DeBERTa”. Given the parameter-efficient tuning, it’s not that hard to perform experiments on large models such as GPT-2 and Llama-2.

**A1.** Thanks for this nice suggestion. We would like to point out that the proposed LEADING algorithm is versatile and applicable to any LLM architecture. We choose BERT and DeBERTa as the backbone language models in our initial submission to ensure a fair comparison with state-of-the-art baselines since these are the models they used for evaluation. For instance, we employed BERT following GIANT and DeBERTa following GLEM.

Following your suggestion, we are happy to provide further experiments on LEADING using larger models. We fine-tune GPT-2 on Cora and ogbn-arxiv datasets. The results shown in the following table indicate that the proposed LEADING algorithm effectively fine-tunes GPT-2 to achieve better performance, which is consistent with our experiments on other language models. Regarding the computation cost, LEADING is capable of maintaining computational costs nearly identical to supervised fine-tuning of GPT without graphs. The additional running time arises due to the sequential execution of two pipelines, yet this can be effectively mitigated through parallel computing. It incurs significantly less computation overhead or memory cost compared to baselines such as GLEM. Note that GIANT runs out of memory in our experiments. It's crucial to emphasize that enhancing model size may not be essential for achieving superior performance, as evident in the comparison between the results of GPT and the DeBERTa presented in our submission. The effectiveness of fine-tuning is influenced by a range of factors beyond mere model size.

Besides, Meta requires a request form to access their Llama model parameters. We submitted the request a week ago but have not received a response yet. We will add new results on Llama once we get access to it. We will provide a complete evaluation of these models in our revision.

# LEADING performance(%) with GPT-2
| Method | GCN(Cora) | GAT(Cora)|GCN(Arxiv)| Rev-GAT(Arxiv)|
|----------|----------|----------|----------|----------|
| Pre-trained GPT-2   | 51.9   | 54.7   | 64.8  | 66.9   |
| Supervised-FT GPT-2   | 70.8   | 71.7   | 73.2   | 73.8   |
| GLEM(GPT-2)   | 70.8  | 71.7   | 74.0   | 75.1   |
| LEADING(GPT-2)   | 80.5   | 81.5   | 74.1   | 75.2   |

# Scalability comparison with GPT-2 on ogbn-arxiv
| Method | Memory(GB) | Running Time(s) | 
|----------|----------|----------|
| Supervised-FT GPT-2    | 26.8   | 15555   | 
| GIANT   | OOM  | N/A   | 
| GLEM(GPT-2)   | 26.8   | 82930   | 
| LEADING(GPT-2)   | 27.1   | 27920  |

Reviewer comment (Novelty):
The technical novelty is limited. Caching neighborhoods has been one of the common techniques to speed up GNN-LMs.

Author response:
**Q2.** The technical novelty is limited. Caching neighborhoods has been one of the common techniques to speed up GNN-LMs.

**A2.** We respectively disagree with this comment. In our paper, we provide a comprehensive survey of related works, and no existing work can achieve end-to-end fine-tuning of LLMs on graphs. Our proposed algorithm is the first to achieve this goal and shows tremendously improved scalability: the training cost of LEADING is close to the training of LLMs without graphs (refer to the scalability analysis in Table 2 in our submission).

Caching neighborhoods is indeed a common technique in accelerating GNNs, and we have cited many related works in our paper but they all require the encoding of both target and neighbor nodes, which are distinctive from our algorithm. Even state-of-the-art scalable GNN training algorithms such as GNNAutoScale (GAS) [1] and GraphFM [2] still encode target nodes and their first-hop neighbors together, which encounter encoding redundancy issues as pointed out in our paper.

In contrast, our proposed algorithm completely decouples target nodes from their neighbors, which is not thought to be possible in the literature. Our approach ensures that in each epoch, each node is only encoded by LLM twice, which minimizes computation redundancy as shown in Fig. 3 and Fig. 4 in our paper. It brings significant cost reduction compared to other memory/cache-
based GNN training strategies.

Here we also provide a memory cost comparison between LEADING and GAS on ogbn-arxiv dataset, employing the same small batch size of 16. Utilizing DeBERTa as the language model, our results demonstrate distinct advantages. Note that even though GAS only considers 1-hop neighbors, it takes significantly more memory usage.

Our algorithm is novel and makes significant contributions to the end-to-end training of GNN-LMs. Please kindly let us know the specific references that speed up the training or fine-tuning of GNN-LMs, and we will be happy to provide a discussion and comparison.

# Memory cost comparison on ogbn-arxiv
| Methods    | GAS     | LEADING   |
|---------|---------|---------|
| Memory(MB)  | 47780  | 3892  |


[1] Fey, Matthias, et al. ”Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings.” International conference on machine learning. PMLR, 2021.

[2] Yu, Haiyang, et al. ”GraphFM: Improving large-scale GNN training via feature momentum.” International Conference on Machine Learning. PMLR, 2022.

Reviewer comment (Evaluation):
Evaluations are limited on node classifications only. Given the limited scope and "old" benchmark on node classification, I don't think the contribution of the proposed idea is significant enough.

Author response:
**Q3.** Evaluations are limited on node classifications only. Given the limited scope and "old" benchmark on node classification, I don't think the contribution of the proposed idea is significant enough. / Can this approach used in other applications such as link prediction and graph classification?

**A3.** We would like to point out that the proposed training LLM-GNN algorithm is general and can be used for any downstream prediction tasks. However, exploring LLM for text-attributed graphs is still a pretty new area, and most datasets for graph classification or link prediction do not provide the original raw text attributes. Therefore, we choose node classification tasks following existing state-of-the-art literature such as GIANT [3], GLEM [4], and [5] to ensure a fair comparison and demonstrate our improvements in efficiency
and accuracy.

To provide an example of link prediction, we run the proposed algorithm on Cora dataset. We manually partition the links into distinct sets for training, validation, and testing. This division is performed with two distinct split ratios: (1) a low ratio of 10/30/60 for train/valid/test, and (2) a high ratio of 85/5/10 for the same sets. We use GCN as the downstream GNN. To prioritize faster execution and simplicity, we choose DistilBert as the language model. As indicated in the following table, LEADING exhibits performance advantages over the baselines, especially in scenarios with limited training data. This aligns with the findings presented in our paper concerning node classification.

# Link prediction performance AUC (%) on Cora
| Methods     | Low     | High     |
|---------|---------|---------|
| Shallow Embedding  | 74.7  | 94.9  |
| Pre-trained DistilBERT  | 64.7 | 68.7  |
| Supervised-FT DistilBERT  | 66.3  | 89.4  |
| LEADING(DistilBERT)  | 81.8  | 95.2  |


[3] Chien, Eli, et al. ”Node feature extraction by self-supervised multi-scale neighborhood prediction.” arXiv preprint arXiv:2111.00064 (2021).

[4] Zhao, Jianan, et al. ”Learning on large-scale text-attributed graphs via variational inference.” arXiv preprint arXiv:2210.14709 (2022).

[5] Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs, arXiv:2023.07

Reviewer comment (Experiments):
What's the performance on larger benchmarks like ogbn-mag/obgn-product, especially the computational cost?

Author response:
**Q4.** What’s the performance on larger benchmarks like ogbn-mag/obgn-product, especially the computational cost?

**A4.** To address your concern, we present the performance results for ogbn-products, where the raw text attributes are available. We also provide the associated computational costs. For simplicity, we utilize GraphSAGE as the downstream GNN backbone in this context.

From the following Table, in terms of performance, LEADING outperforms all baselines, including cascaded training (Supervised-FT), self-supervised training (GIANT), and iterative training (GLEM). Regarding computational costs, our findings align with those in our papers: LEADING requires nearly identical memory compared to supervised FT without a graph, and it significantly outperforms
GIANT and GLEM. It is worth noting that GIANT runs out of memory in our experiment due to the high training costs associated, so we report the accuracy of the well-trained model provided by their official repository.

# Performance and computation cost comparisons on ogbn-products

| Methods     | Accuracy(%)     | Running Time(s)    | Peak Memory Usage(GB)    |
|---------|---------|---------|---------|
| Shallow Embedding| 79.7  | ---  | ---  |
| Pre-trained DeBERTa  | 62.0  | ---  | ---  |
| Supervised-FT DeBERTa  | 82.2  | 67200 | 25.1  |
| GIANT | 83.2  | N/A | OOM |
| GLEM (DeBERTa)  | 83.2  | 356648 | 25.2 |
| LEADING (DeBERTa)  | 84.1  | 124164 | 25.6 |

To conclude, we believe we have fully addressed all of your concerns. Please kindly let us know if you have any further concerns.