Paper: GraphLLM: Boosting Graph Reasoning Ability of Large Language Model (ICLR-2024)

Reviewer comment (Evaluation):
One major advantage of using LLMs compared to GNNs is explainability. Therefore, I would recommend conducting some analyses in this aspect. For example, if the proposed method cannot produce reasons for its prediction, it is likely that the proposed method captures some spurious correlations and does not truly “solve” the problem.

Author response:
Thank you for your inquiry regarding the explainability of GraphLLM. In addressing the concern about providing explanations for the output answer, we conduct additional experiments on Shortest Path, tasked the LLM to simultaneously generate the optimal path as an explanation for the answer.

Under the division of 6000/2000/2000 graph instances for training/validation/test, GraphLLM (LLaMA-7B-2) can output valid and optimal paths for 94.55% of the test graph instances while the LoRA fine-tuning of LLaMA-7B-2 achieved only 57.70%. This outcome provides preliminary evidence of GraphLLM's ability to articulate explanations for its predictions.

Reviewer comment (Experiments):
I also wonder whether the authors have considered directly comparing with graph Transformers or GNNs, to see whether LLMs have advanced the field.

Author response:
We respectfully disagree and would like to stress that graph neural network-based methods are not a meaningful baseline for the issue explored in this paper.

Firstly, our paper's primary aim is to enhance the graph reasoning abilities of LLMs, a goal that does not necessitate a comparison with non-LLM methods. Previous benchmarking work[1] has also not considered graph neural networks as a type of baseline. As corroborated by previous benchmarking work[1], graph neural networks have not been considered a standard baseline in this context. Our focus is not on outperforming graph neural network-based methods in graph reasoning tasks, such as shortest path problems, which can be precisely solved with simple algorithms with a few lines of code. Instead, the pivotal question we aim to explore is: in the trend where generative LLMs are increasingly seen as a universal agent due to their powerful reasoning capabilities, how these LLMs fare in graph reasoning. Considering this, the performance metrics of graph neural network-based methods do not contribute additional insights to our research question.

Even if we were to attempt a comparison with graph neural networks, please note that the graph reasoning task we address (as outlined in Definition 2.2) is intrinsically a natural language generation task, necessitating inputs and outputs in human language. For instance, in a substructure counting task, the output is expected to be a natural language statement like 'There is 1 [substructure name]'. If we do not transform the task or adapt the graph neural network-based methods, even achieving a comparison becomes impossible. The question is how to make a fair comparison between a generative LLMs that take natural language for input and output, and a graph neural network?  Transforming natural language outputs into some kind of classification category seems a possible solution, but we should note that an inherent advantage of LLMs is their ability to interact and reason in human language, making such a comparison evidently meaningless.

Reviewer comment (Reproducibility):
Based on my understanding, one major advantage of the Graph2Text pipeline is being able to utilize close-sourced LLMs such as GPT-3.5 or GPT-4 without any fine-tuning step, since we only need to input data and related information (prompts, queries, etc.) to APIs. In comparison, as the proposed method needs to be trained end-to-end, it increases the computation cost and can only operate with open-sourced LLMs (in order to obtain gradients), which limits the applicability.

Author response:
Imagine this application scenario: users requires an LLM (like LLaMA 2 7B) to address needs that can be abstracted as a shortest path problem. It is found that the outcomes of zero/few-shot prompting do not satisfy (Exact Match Accuracy: 0.1089). In such situations, the most common practice for users is to perform fine-tuning on this task. The point is, in this scenario, fine-tuning is a necessary cost. While the graph2text approach bears this cost, it does not yield satisfying outcomes. This, we argue, adequately showcases the superiority of our method in applicability.

We wish to emphasize that our method indeed cannot be used with closed-source models, a limitation that should be attributed to the practice of closed-sourcing itself, rather than our method. We would like to remind that in many scenarios, using APIs from closed-source models is impractical, due to concerns such as data security or cost.

Reviewer comment (Novelty):
All things considered, the technical contribution of the paper is okay but not too novel, considering that all three components are heavily based on the existing literature, i.e., node understanding is a standard Transformer, structure understanding is essentially equivalent to Ma et al., 2023, and prefix tuning follows the common practice of LLMs. It would make the paper stronger if these components could be customized to further enhance the performance.

Author response:
We wish to emphasize that the paper's principal novelty lies in the idea of collaborating graph learning models and LLMs as a unified system, a first in this domain. Furthermore, even combining graph transformers with prefix tuning is non-trivial in its implementation. We would like to point out that we have indeed made customizations. For example, the most critical issue is how to make the graph transformer generate prefixes of specific dimensions. Our initial experimental attempts indicate that simply applying a dimensional transformation to the output of the graph transformer does not work. Therefore, the graph transformer's input is designed to be derived from a query of the same dimension as the prefix, a design we propose for the first time.

Reviewer comment (Experiments):
The main weaknesses of this paper are in its experimental evaluation. The architecture seems potentially powerful but under explored.

Author response:
We fully agree that conducting experiments on more complex, real-world datasets would add significant value to this paper, and our considerations for using simple artificial tasks are as follows:
- The use of simple artificial tasks for benchmarking the reasoning abilities of LLMs is a well-established practice in this field. An example of this is the GSM8K dataset[1], which is widely employed to evaluate LLM's mathematical reasoning abilities.
- Somewhat paradoxically, the established real-world benchmarks like ogbn-arXiv (with textual feature) might pose lower demands on LLMs' graph reasoning abilities compared to the tasks presented in our paper. For LLMs, determining the arXiv category of a paper based on its abstract is a very simple task. (LLMs can even continue writing the paper or provide review comments based on the abstract.) As a result, measuring the benefits of graph reasoning abilities for this task has become quite challenging (see Table 16 & 17 in [2]). In the era of LLMs, building a real-world benchmark that truly accurately measures the graph reasoning capabilities of these models requires the effort of the entire community. After careful consideration, at this early stage, we've opted for artificial datasets to secure somewhat precise measurements.

We are currently trying to further apply our architecture to some more exciting real-world applications.

Reviewer comment (Experiments):
LLMs are general purpose reasoners over text and graph2text takes advantage of that, whereas GraphLLM is purposely built and trained for these specific graph reasoning tasks. It would be surprising if GraphLLM did not perform better in this setting. As a consequence, most of the results are not "interesting". Graph2text is a single representation that can be used for all four tasks, while the prefixes from GraphLLM are tailored to each task individually. Although LLM fine-tuning is a more comparable setting, it is also less interesting, as once we are fine-tuning we could be using other architectures (e.g., GNN). A more interesting question would be whether GraphLLM can be used as a general graph-to-prefix encoder enabling the LLM to perform different fundamental tasks better.

Author response:
We must admit that our framework's ability to perform basic graphic tasks is indeed not very exciting. In evaluation, the more valuable aspect might be that finetuning on Graph2Text cannot achieve basic graph reasoning, such as the shortest path example. This suggests that, at least for now, the idea that 'natural language is all a graph needs' is still not true.

We also attempted to use GraphLLM as a general framework to perform different fundamental tasks. When joint training Substructure Counting and Maximum Triplet Sum with the same setting in the paper, GraphLLM can simultaneously achieve accuracies of 0.9946±0.0021 on Substructure Counting and 0.8472±0.0210 on Maximum Triplet Sum, indicating its preliminary general capability.

Reviewer comment (Experiments):
The experiments do not show that GraphLLM is "useful". The tasks tested do not require machine learning and can all be solved explicitly. This raises the question of why one would use a complex ML architecture to replace a simple program. Although the architecture is smart with many avenues to demonstrate utility, these experiments do not provide that demonstration. They do show that GraphLLM is capable of performing graph reasoning tasks, which has value, but there would be more value in showing more general capabilities.

Author response:
We must admit that our framework's ability to perform basic graphic tasks is indeed not very exciting. In evaluation, the more valuable aspect might be that finetuning on Graph2Text cannot achieve basic graph reasoning, such as the shortest path example. This suggests that, at least for now, the idea that 'natural language is all a graph needs' is still not true.

We also attempted to use GraphLLM as a general framework to perform different fundamental tasks. When joint training Substructure Counting and Maximum Triplet Sum with the same setting in the paper, GraphLLM can simultaneously achieve accuracies of 0.9946±0.0021 on Substructure Counting and 0.8472±0.0210 on Maximum Triplet Sum, indicating its preliminary general capability.

Reviewer comment (Experiments):
The comparison of input tokens does not seem to account for the text input to the node understanding encoder. While this may matter less because the encoder is much smaller, the main issue is that the design of graph2text seems intentionally aimed to maximize the number of tokens. For example, in the max triplet sum task, irrelevant descriptive content like "[Cornelia Brooks's] petite frame and delicate features give her a dainty and ethereal presence" is included (Appendix D). Additionally, the text uses "connected with" to describe edges while the question itself uses "friends," which could result in poor showcasing of the LLM baselines. Even for humans, reading the shortest path text is confusing due to irrelevant details such as distance from earth or "activating a wormhole." The only description of the task is "Starting from wormhole 1, How much dark matter we’ll need at the minimum to reach Wormhole 2?" which appears unclear.

Author response:
Also thanks for the concerns regarding the Graph2Text prompts. In Maximum Triplet Sum, the node text feature includes the key information about the person's age, and other descriptions generated by GPT-3.5-Turbo. This design aims to assess the node understanding capabilities of both GraphLLM and Graph2Text-based LLMs, specifically their ability to extract essential details from seemingly unrelated text. And according to your other concerns, we have revised the prompt and assessed Graph2Text with GPT-4 few-shot CoT. In Maximum Triplet Sum, we describe the edge between people using "friendship", and the accuracy remains 93.33%. In Shortest Path, we modify the background to cities connected with roads, where it incurs a cost to pass through a city. However, the accuracy decreases from 86.67% to 83.33%.

Reviewer comment (Experiments):
The paper evaluates GraphLLM on four graph reasoning tasks, including substructure counting, maximum triplet sum, shortest path, and bipartite graph matching. Although these tasks cover basic graph reasoning abilities, they are still relatively simple, which might overestimate how GraphLLM performs on noisier graphs with complex relational patterns. More complex graph reasoning tasks could better demonstrate the capabilities and limitations of GraphLLM. For example, tasks requiring multi-hop reasoning on large graphs might be more challenging.

Author response:
We wish to emphasize that the difficulty of our task is similar to that of the tasks designed in the benchmark study[1] on graph reasoning abilities of LLMs, which was recently accepted by NeurIPS 23. 

A primary aim of this paper is to attempt to answer the question: What hinders the graph reasoning abilities of LLMs? A natural thought is that, to investigate this issue, we should start with the most basic graph reasoning tasks. Surprisingly, we find that current LLMs even struggle with these simple tasks.  We agree that for further applications, dealing with noise graphs and more complex graph reasoning is important. However, for this article, we want to first focus on the primary question.

Reviewer comment (Experiments):
The datasets used in the paper are all small with fewer than 20 nodes on average. Thus, the ability of GraphLLM to handle large, real-world graphs is unclear. Evaluations on larger datasets would be informative.

Author response:
Thank you for your insightful observation regarding the scalability of large graphs. In our study, inspired by the previous work[1], we intentionally mirrored the graph size of ~20 in the graph reasoning problems from NLGBench to maintain consistency and focus. Surprisingly, LLMs, known for their general reasoning abilities, struggle with these basic graph reasoning tasks. Our primary goal is to explore and potentially resolve this apparent paradox. We fully acknowledge and agree with your point on the importance of handling larger graphs for practical applications. However, such an extension, while valuable, falls outside the scope of our current research focus.  We will include a discussion on scalability in the revised version of the paper.

Reviewer comment (Experiments):
The tasks only use textual node features. Testing on graphs with non-textual node attributes would make the approach more broadly applicable.

Author response:
What we want to clarify is that, rather than designing models applicable to various real-world graphs, the main purpose of this paper is to investigate the reasons hindering the graph reasoning abilities of LLMs and to identify potential solutions. Non-textual node features requiring additional alignment with LLMs could cause unnecessary distractions in our investigation.

Reviewer comment (Experiments):
The paper only compares GraphLLM against methods that convert the graph to text (Graph2Text). However, established graph neural networks like GNNs, GCNs, and graph transformers have been optimized specifically for graph reasoning. Comparing to these dedicated graph reasoning models could better highlight if GraphLLM actually improves over the state-of-the-art in graph representation learning. For example, how does GraphLLM compare to a graph transformer without the attached LLM on the tested graph reasoning tasks? This could isolate the benefits of the LLM integration. Similarly, comparisons to GNN variants on standard benchmarks could reveal where GraphLLM excels or lags behind existing graph-specific architectures. The improvements over Graph2Text may simply be because LLMs struggle with learning from text descriptions of graphs. Graph models designed for relational reasoning may be more competitive. In essence, without comparing to specialized graph reasoning models, it is hard to discern if GraphLLM's integration of LLMs and graph networks is truly advancing state-of-the-art performance. Adding these comparisons would give a clearer sense of GraphLLM's capabilities relative to the field of graph representation learning overall.

Author response:
We respectfully disagree and would like to stress that graph neural network-based methods are not a meaningful baseline for the issue explored in this paper.

Firstly, our paper's primary aim is to enhance the graph reasoning abilities of LLMs, a goal that does not necessitate a comparison with non-LLM methods. Previous benchmarking work[1] has also not considered graph neural networks as a type of baseline. As corroborated by previous benchmarking work[1], graph neural networks have not been considered a standard baseline in this context. Our focus is not on outperforming graph neural network-based methods in graph reasoning tasks, such as shortest path problems, which can be precisely solved with simple algorithms with a few lines of code. Instead, the pivotal question we aim to explore is: in the trend where generative LLMs are increasingly seen as a universal agent due to their powerful reasoning capabilities, how does the graph reasoning ability of LLMs perform, being one of the key reasoning skills for solving many problems like path planning, and towards what future developments might we be moving? Considering this, the performance metrics of graph neural network-based methods do not contribute additional insights to our research question.

Even if we were to attempt a comparison with graph neural networks, please note that the graph reasoning task we address (as outlined in Definition 2.2) is intrinsically a natural language generation task, necessitating inputs and outputs in human language. For instance, in a substructure counting task, the expected output is a natural language statement like 'There is 1 [substructure name]'. If we do not transform the task or adapt the graph neural network-based methods, even achieving a comparison becomes impossible. The essential question is how to make a fair comparison between generative LLMs that take natural language for input and output, and a graph neural network?  Transforming natural language outputs into some kind of classification category seems a possible solution, but we should note that an inherent advantage of LLMs is their ability to interact and reason in human language, making such a comparison meaningless.