Paper: Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning (ICLR-2024)

Reviewer comment (Reproducibility):
The time analysis and money estimation are lacking. The paper utilizes the chatgpt API, nonetheless, there is a large restraint on the word limitation per day, and the cost of one dataset should also be taken into consideration.

Author response:
**Authors:** We acknowledge the importance of a comprehensive analysis of time and cost, given the constraints of the ChatGPT API.  We incorporated it in appendix D.10 (in blue) in our revised manuscript.
Time and Cost Implications: Our primary dataset, ogbn-arxiv, with 169,343 nodes and 1,166,243 edges, serves as a representative case for our approach. On average, our input sequences consist of approximately 285 tokens, while the output sequences comprise around 164 tokens. For the ChatGPT-3.5 Turbo API, priced at $\textdollar$0.0015 per 1,000 input tokens and $\textdollar$0.002 per 1,000 output tokens, with a token per minute rate limit of 90,000, the monetary estimation for ogbn-arxiv is as follows:

Cost = ((285 * 0.0015) / 1000 + (164 * 0.002) / 1000) * 169,343 ≈ 128 USD

Considering the token rate limit, we estimate the deployment time as follows:

Time = 169,343/ (90,000/285) ≈ 536 min ≈ 9h

**Cost-Effective Alternatives**: Additionally, we have explored cost-effective alternatives, such as leveraging open-source LLMs like llama2. The use of llama2 is entirely free, and the querying process to llama2-13b-chat takes approximately 16 hours when utilizing 4 A5000 GPUs (24GB). 

**Efficiency through Single Query and Reuse**: Our method requires only one query to the LLM, with predictions and explanations stored for subsequent use. This not only enhances efficiency but also minimizes the number of API calls, contributing to cost-effectiveness. 

**Dataset Sharing**: We also make the GPT responses available for public use.

Reviewer comment (Experiments):
The robustness of the prompt is lacking. The paper proposed a single prompt for the node classification. Nonetheless, is another similar prompt can achieve a similar performance.

Author response:
**Authors:** We have extensively explored the influence of various prompts, as outlined in Table 8 and Table 9 (in blue) in the appendix. Our findings reveal a positive correlation between the zero-shot accuracy of the LLM and the overall accuracy of our method, i.e. the higher the zero-shot prediction score, the better the accuracy of TAPE. Besides, despite employing different prompt designs, our approach consistently maintains similar accuracy, ranging from 0.7660 to 0.7750, demonstrating the robustness of the prompt.

Reviewer comment (Experiments):
There may lack of ablation study on the LM. Current LM focuses on BERT, while other LM are ignored. Recent paper demonstrates that they can achieve satisfying performance with only SentenceBert Embedding.

Author response:
**Authors:** We thank the referee for pointing out the necessity of a more comprehensive ablation study regarding the LM. In response, we ran additional experiments involving different LMs to assess their impact on TAPE.

Before revealing our findings, we would like to address a recent paper, SimTeG [1], which uses SentenceBert and might have been referenced by the reviewer. SimTeG indeed demonstrates satisfactory results with SentenceBert Embedding. However, it is crucial to acknowledge that, similar to our model, their approach entails fine-tuning the LM-SentenceBert, with raw text and node labels. Additionally, SimTeG highlights the seamless integration of our method with theirs. A claim substantiated through experimental testing, showing that SimTeG+TAPE with Ensembling establishes a new SOTA performance on the ogbn-arxiv dataset.

Moving forward, we extended our investigation beyond the deberta-base model by studying the effect of different LMs on ogbn-arxiv. In alignment with the approach taken in SimTeG [1], we incorporated two additional widely-used LMs from the MTEB leaderboard: all-roberta-large-v1 and e5-large. The detailed outcomes of our study are presented in Table 13 (in blue). Notably, our model demonstrates insensitivity to the choice of a specific LM, emphasizing its robustness to variations in LM selection.

Reviewer comment (Experiments):
There lack some experimental results in table 1, especially Giant.

Author response:
**Authors:** We acknowledge the reviewer's observation. Our initial comparison with GIANT was conducted solely on the ogbn-arxiv dataset, as the original GIANT paper did not include experiments on Cora/PubMed, etc. In response to this feedback, we extended our comparison by applying the GIANT model to additional datasets using their official code. The comprehensive results are now presented in Table 1 (in blue). We appreciate the suggestion for a more thorough comparison across datasets. Notably, our results demonstrate consistent outperformance against GIANT on all these datasets, providing substantial support for the effectiveness of our method.

Reviewer comment (Experiments):
Lack of baselines. Many important baselines that conduct learning with language models on TAGs including GraphFormers [1] and Patton [2] are missing in the experimental section.

Author response:
**Authors:** We appreciate the reviewer’s valuable feedback and acknowledge the concern regarding the absence of certain baselines in our experimental section, particularly GraphFormers [1] and Patton [2]. We would like to clarify the reason for this absence.

Our primary focus in this work is the node-prediction task in TAG. For this specific task, we concentrated on the most widely used datasets, i.e. ogbn-arxiv and ogbn-products. Additionally, recognizing the high popularity of Cora and Pubmed as node-classification datasets, we augmented these datasets by collecting text information, title and abstract, to create two additional TAGs (also made publicly available). We further introduced a new dataset, arxiv-2023, ensuring it is not part of the training set of GPT-3.5.

In the context of ogbn-arxiv and ogbn-products, we chose to compare our method against GLEM [3] and GIANT [5], recognized as the strongest baselines for these datasets. While we understand the concern about the absence of GraphFormers and Patton, we believe our study offers a robust comparison by evaluating against the strongest baselines within the constraints of the available datasets.

Specifically addressing GraphFormers, it was primarily designed for link prediction tasks, making a direct comparison less suitable due to differences in task objectives. Similarly, Patton involves pre-training on Microsoft Academic Graph (MAG) or Amazon e-commerce networks, followed by fine-tuning in a few-shot manner, making it also challenging for a direct comparison.

Moreover, it is important to note that the datasets used in the GraphFormers and Patton papers do not overlap with the datasets in our study. Consequently, we are unable to report from their papers the performance on our datasets.

Nevertheless, to address the referee’s concern, we conducted additional experiments within the rebuttal period. We extended our experiment to include MAG-Economics, a 40-way node classification task, to facilitate a comparison with Patton. The results, reported in Table 15, demonstrate that our technique TAPE outperforms GraphFormers and Patton by a significant margin of at least +13% accuracy on the MAG-Economics dataset.

In the revised version, we will commit to extending the comparative analysis to other datasets such as MAG-Mathematics, MAG-Geology, Amazon-Clothes, and Amazon-Sports. We believe this comprehensive evaluation across multiple datasets will address the gap mentioned by the reviewer and provide a more holistic comparison with existing baselines.

We thank the reviewer for the insightful comment, which contributes to the enhancement of the experimental section of our work.

Reviewer comment (Theory):
The theorem is not specific to this problem. The theorem in 4.4 is not particular for LLM and LM for TAGs, and needs strong assumptions.

Author response:
**Authors:** While it is true that Theorem 4.4 could be applied in other contexts, we found it valuable to provide a rigorous foundation for the intuitions guiding the proposed technique.

Our intention was to reinforce the understanding of why the technique is effective, beyond solely relying on experimental results. Specifically, the theorem underscores the importance of fidelity in explanations with LLM's reasoning, and emphasizes the non-redundancy between LLM and LM. These aspects constitute the key assumptions under which the technique was designed to operate successfully.

Reviewer comment (Experiments):
Lack of ablation studies. 1) If we need to finetune the LM in step 2 or not?; 2) Why do we need to have different LM for original text encoding and explanation encoding?

Author response:
**Authors:** Thank you for highlighting the need for a more comprehensive ablation study. In response to the query, we conducted additional experiments specifically on the ogbn-arxiv dataset to investigate the impact of LM fine-tuning.
Addressing the reviewer’s specific points:
1) The significance of fine-tuning the LM in step 2: As illustrated in Table 12, the row labeled "without fine-tuning" starkly exhibits inferior performance when using a pre-trained LM without fine-tuning, compared to the last two rows showcasing the positive impact of LM fine-tuning. 
2) The necessity of having different LMs for the original text and explanations: Table 12, with rows "fine-tuning (same LM)" versus "fine-tuning (different LMs)," demonstrates that training the same or distinct LMs for encoding the original text and the explanations yields similar results. There is a slight advantage for using two LMs, but the marginal difference suggests that our approach can be simplified and expedited by employing one LM.
We appreciate the reviewer's insight, which helps simplify the presented technique.

Reviewer comment (Novelty):
Limit novelty. While I appreciate the introduction of the new arxiv-2023 dataset, the technique contribution of this work is very limited, which basically introduces LLM to conduct data augmentation for node classification on TAGs.

Author response:
**Authors:** We reproduce below our general response regarding the importance and novelty of our work:

With the emerging dominance of LLMs, it has become critical to integrate LLMs with GNNs for text- and graph-based tasks such as TAG, knowledge graphs, and text reasoning. Our work stands out as a pioneering effort in this direction.

The task of representation learning on TAGs involves integrating the raw text with graphs in a robust and scalable way. Existing literature can be broadly categorized into two paradigms: 1) End-to-end or joint training of LM and GNN, exemplified by approaches like GLEM [3] and DRAGON [4]; and 2) Two-stage training, where LM is initially trained followed by GNN training. Our approach belongs to the second paradigm, inheriting its efficiency and ease of plugging into existing GNN pipelines, but incorporates an LLM-to-LM interpreter to extract and interpret the LLM's useful prior knowledge.

The first approach is generally considered more attractive because the LM is trained to directly leverage graph structure. However, it often involves complex language and graph architectures, encounters training instabilities, and demands significant GPU resources, particularly challenging with LLMs. In this context, an important contribution of our work is the finding that with the help of our LLM-to-LM interpreter, the second paradigm of two-stage training can achieve equal or better performance than end-to-end or joint training, with much greater efficiency. In particular, our LLM-to-LM interpreter allows the valuable prior knowledge from LLMs to be embedded into GNN features, and our work shows that this does not sacrifice performance but greatly improves efficiency, compared to joint training.

Specifically, we decompose the text+graph combination into two distinct parts: a LLM+LM interpreter harnessing LLM capabilities but challenging to fine-tune, and a GNN using text features fine-tuned for the graph task. This decomposition significantly stabilizes the training process, as each part is independently executed — prompting LLM, fine-tuning LM with node labels, and supervised training of the GNN.

This approach shares similarities with the training of LLMs like ChatGPT or Llama2, which involve multiple independent training steps, including self-supervised learning, fine-tuning, and reinforcement learning. This departure from optimizing in a one-shot end-to-end manner, known for its instability, has proven highly successful in NLP, and now in TAG demonstrated with our technique through extensive experiments.

Lastly, we believe that LLMs have disrupted the field of GNNs,but the current focus on end-to-end or joint LM+GNN training creates near-prohibitive scalability issues when applied to LLMs, making it harder to reach the potential of LLMs for graph learning. Our work takes a significant step by showing that LLM-to-LM interpreters can be used to embed useful knowledge from LLMs into GNN pipelines in a simple and efficient way, without sacrificing performance. As a matter of fact, since the introduction of our work, we have witnessed a surge in similar techniques published on ArXiv, many of which have achieved top rankings on the OGB leaderboard. This underscores the significance of our paper as the pioneer work in this line of research, which is poised to grow in the coming years.

Reviewer comment (Novelty):
The main concern is that the method has little to do with graph. It seems like an application attempt of LLMs on natural language tasks, and TAG is just a scenario. Thus the contribution of the method is limited.

Author response:
**Authors:** While it is true that the first part of our technique has no direct connection to graph topology, and focuses on node features, the overall methodology is strongly motivated towards addressing the challenges inherent in the TAG problem, an active research area propelled by the advent of language models. 

As advocated in our global response, our approach is deeply rooted in addressing the challenges associated with the joint training of GNNs and LMs. The task of representation learning on TAGs involves integrating the raw text with graphs in a robust and scalable way. Existing literature can be broadly categorized into two paradigms: 1) End-to-end or joint training of LM and GNN, exemplified by approaches like GLEM [1] and DRAGON [2]; and 2) Two-stage training, where LM is initially trained followed by GNN training. Our approach belongs to the second paradigm, inheriting its efficiency and ease of plugging into existing GNN pipelines, but incorporates an LLM-to-LM interpreter to extract and interpret the LLM's useful prior knowledge. 

The first approach is generally considered more attractive because the LM is trained to directly leverage graph structure. However, it often involves complex language and graph architectures, encounters training instabilities, and demands significant GPU resources, particularly challenging with LLMs. In this context, an important contribution of our work is the finding that with the help of our LLM-to-LM interpreter, the second paradigm of two-stage training can achieve equal or better performance than end-to-end or joint training, with much greater efficiency. In particular, our LLM-to-LM interpreter allows the valuable prior knowledge from LLMs to be embedded into GNN features, and our work shows that this does not sacrifice performance but greatly improves efficiency, compared to joint training.

Therefore, our work should be viewed as a strategic response aimed at overcoming these challenges. It goes beyond being a mere application of LLMs to TAG; rather, it represents a novel, efficient, and robust approach to addressing tasks that involve both text and graphs.

We acknowledge that we may not have sufficiently conveyed the important message that our work aims to resolve the intricate issues tied to LM+GNN training for TAG. We are committed to amending the paper to provide greater clarity on this aspect for the reader.

Reviewer comment (Experiments):
Since the Debera need fully fine-tuning, training the method cost much more memory than pure GNN methods.

Author response:
**Authors:** We appreciate the reviewer's observation regarding the increased memory usage in our approach compared to pure GNN methods, as illustrated in Figure 2. It is important to note that learning on TAG inherently demands the integration of LMs and GNNs. Established methods like GIANT [5], GLEM [1], GraphFormers [3], and Patton [4], among others, all involve LM in their processes. Therefore, a meaningful comparison of memory cost should be drawn against these baselines rather than pure GNNs.

Recognizing the inherent tradeoff between cost and accuracy, we highlight that despite the increased resource requirements, our method TAPE significantly boosts the accuracy of the pure GNN method from 70.83% to 77.50%. We firmly believe that the substantial performance improvement justifies the accompanying rise in memory costs. Notably, these elevated resource demands are manageable, even with relatively cost-effective academic GPU resources like our A5000 GPU (24GB).
We greatly appreciate the reviewer's observation regarding memory usage, which prompted us to conduct a more in-depth analysis, as outlined in Table 14 (in blue), showing that TAPE provides both best performance, and comparable or better running time than other LM-based methods.