Paper: Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR-2024)

Reviewer comment (Evaluation):
I am concerned that ToG would bring too many intermediate steps and cause high latency in reasoning and expensive deployment, especially when using API-based black-box models like GPT-4.

Author response:
Please refer to our submitted official comment "Response to All reviewers about the concerns on the Efficiency of ToG", where we elaborate on the maximum cost of answering a question needed by our method and the average cost on CWQ. We also propose some feasible solutions on how to improve the efficiency of ToG (some have been empirically verified in experiments). In particular, we show in this official comment that the latency in inference is NOT as high as what people may imagined using improved ToG algorithms. In most cases, the number of LLM calling by ToG is only single digit (even <5 times in most cases).

Reviewer comment (Experiments):
It would be interesting to see the performance of ToG when incorporating the KGs of low quality (sparsity, noisy, etc), since most KGs are sparse and out of date to some extent. So investigating the impact of KG quality would enhance the understanding of ToG method and its limitations.

Author response:
(1)	We agree with the reviewer that quality of knowledge graphs may have significant impact on the performance of ToG. We are doing experiments to test how the qualify of a KG affect ToG's performance. Specifically, we artificially add different amount of noise/errors and/or delete different amount of triples existing in original wikidata, and evaluate how much ToG's performance drops down. Unfortunately, the experiments are still in progress due to limited computational and LLM token resource. We are trying our best to show the experimental results asap.
 
(2)	Another important task we are doing is to research potential solutions for alleviating the negative effect of low-quality KG in ToG inference. Firstly, in Section 3.3, we have illustrated how ToG could correct the wrong information in KG with the help of LLMs and users (identifying potential errors in KG by LLM and determining the correctness by human). More concretely, the LLM can suggest to the users which knowledge in the reasoning paths may be wrong, and the users reply in the system on whether execute a correction action. Secondly, we are studying how to improve the quality of KG by fusing diverse external information source such as variant LLMs, search engine and many others.

Reviewer comment (Experiments):
I would encourage authors to present experiment results on small LLMs, such as 7B and 13B.

Author response:
Following the reviewer's suggestion, we tested the performance using LLaMA2-7b and LLaMA2-13b, two very small LLMs, on WebQSP, and the results are summarized below. 

| Method | LLaMA2-7b | LLaMA2-13b |
| --- | --- | --- |
| CoT | 7.1%| 8.6% |
| ToG | 12.6% | 25.1% |

Using exact match as the evaluation metric, the CoT prompting with LLaMA-7B and LLaMA-13B can only achieve 7.1% and 8.6%, respectively, on WebQSP. Meanwhile, ToG with LLaMA2-7B and LLaMA2-13B can achieve 12.6% and 25.1%, respectively. Consistent with our expectation, ToG performs better than CoT with these two very small LLMs. However, the actual accuracies for CoT and ToG are indeed far below from our expectation. After careful investigation, we found the main factor of the bad performance is: since both LLaMA2-7B and LLaMA2-13B are base models which have not been trained with supervised fine tuning and RLHF, the format of their outputs are really difficult to manage and difficult to exactly match the format required for a "correct" answer. For example, we require the answers from LLM should be put between { and }, but these two small LLMs always violate this rule and make the evaluator difficult to locate the answer, leading its answer misclassified to "wrong". In another example, we ask LLM to prune and return top 3 relations from candidates (A, B, C, D, E), but LLaMA2-7B and LLaMA2-13B return irrelevant relations, e.g., W, S, T etc.  In some cases, LLaMA2 models restate the input questions or repeat the text over and over again without responding to the question.

Reviewer comment (Evaluation):
The computational cost of ToG is relatively high. The searching process of ToG involves multiple LLM calls, which may be costly and limit its practical applicability in some settings.

Author response:
**Reply**:  Please refer to our submitted official comment "Response to All reviewers about the concerns on the Efficiency of ToG", where we elaborate on the maximum cost of answering a question needed by our method and the average cost on CWQ. We also propose some feasible solutions on how to improve the efficiency of ToG (some have been empirically verified in experiments).

Reviewer comment (Reproducibility):
Some details are inconsistent in the paper. For example, in the approach introduction section, ToG selects the next step triples/relations based on current expended reasoning paths. However, in the prompt illustrated in G.3, I cannot find the current reasoning paths used for the pruning process.

Author response:
**Reply**: First of all, we are sorry for the confusion about what (whole path or only triple?) ToG really gives LLM's prompt in pruning. We clarify it here: ToG selects the next relations/entities based only on the literal information of candidate entities/relations and the given question (as mentioned in Section 2.1.2), rather than the current whole expanded reasoning paths. We would revise the description in the manuscript to make it clearer for readers. In addition, in future work, we are going to study whether the pruning process might be improved by inputting current expanded reasoning paths instead.

Reviewer comment (Experiments):
I have concerns that ToG might not understand the meaning of relations well and generalize to different KGs. The relations defined in KGs are usually in diverse formats. For example, the relations in Freebase are defined in a hierarchical format, while the relations in Yago have clearer semantics. If the relations do not reveal the clear semantics, the searching process of ToG might be misled.

Author response:
**Reply**: We understand the reviewer's concern whether LLM in ToG can correctly understand complex and diverse relations in various KGs. Conceptually, although the format of relations in Freebase is different from natural language, the relation names still consist of words and phrases understandable by LLMs (e.g., people.person.occupation, sports.mascot.team, the last word shows the meaning of relation). In the case of Wikidata, most relations are quite similar to natural language (e.g., instance of, part of, founded by). Empirically, it can be seen from our experimental results that LLMs and light-weight pruning models can perform relation pruning well on both Freebase and Wikidata. In addition, the reviewer's concern leads us to study whether ToG can be improved either by cleaning the relation names in KGs before inputting them to LLMs or by incorporating the specific relation structure of a KG. We would study this problem in the future.

Reviewer comment (Reproducibility):
Can the authors discuss the cost of ToG in detail? What is the average number of calls for the reasoning? What is the overall price for the ChatGPT/GPT-4 API calls?

Author response:
**Reply**: Please refer to our submitted official comment "Response to All reviewers about the concerns on the Efficiency of ToG", where we elaborate on the maximum cost of answering a question needed by our method and the average cost on CWQ. We also propose some feasible solutions on how to improve the efficiency of ToG (some have been empirically verified in experiments).

Reviewer comment (Writing):
Can the authors explain the inconsistency I discussed above and present a clear illustration of the whole reasoning process?

Author response:
**Reply**: Since this question is similar to Comment 2, please refer to the reply to Comment 2.

Reviewer comment (Theory):
Can the authors explain how ToG can generalize to different KGs? How LLMs in ToG understand the meaning of relations in different KGs without additional training?

Author response:
**Reply**: This question is similar to Comment 3. Please refer to the reply to reviewer's comment 3.

Reviewer comment (Experiments):
Can other search algorithms be used for ToG? For example, depth-first search, breadth-first search, A* search, etc.

Author response:
**Reply**: Yes, theoretically, the search algorithm in ToG can be replaced to many other search algorithms, depending on the trade-off between efficiency and sufficiency. For example, beam search can be considered as a constrained breadth-first search (BFS), in order to improve search efficiency by appropriately sacrificing sufficiency. Likewise, constrained DFS and A* algorithm can also be used for searching optimal reasoning paths of ToG. In the future work, we are studying which search algorithms are better and how to automatically select the best search algorithm for ToG according to specific task and specific KG structure. In addition, we are going to study new search algorithms to optimize the performance and efficiency of ToG.

Reviewer comment (Experiments):
Can ToG be used to solve complex reasoning problems that cannot be solved by path-based reasoning? For example, in GrailQA, there are questions like: "How many TV programs has Bob Boyett created?". This question needs to count the KG structures to get the answer. Besides, there are other complex questions requiring intersection, union, and negation operations. Can ToG be used to solve these kinds of questions? Maybe a limitations section can be added to the paper.

Author response:
**Reply**: Although vanilla ToG may not solve these complex reasoning problems as mentioned by the reviewer, a slight variant of ToG can solve all these complex problems mentioned above. Specifically, if we swap the order of pruning and reasoning in each iteration of vanilla ToG, the new ToG algorithm (call new-ToG) can solve all above complex reasoning problems. For example, when new-ToG answers the question "How many TV programs has Bob Boyett created?", new-ToG searches all neighbors of "Bob Boyett" related to "TV programs" and generates $M$ reasoning paths:  {Bob Boyett-producer-program1, Bob Boyett-producer-program2, Bob Boyett-producer-program3... Bob Boyett-producer-programM}. LLMs can "count" the number of neighbors during the reasoning step and output the answer $M$. Note that vanilla ToG can't finish this task because these neighbors are pruned in each iteration and only top $N$ (width of beam search) are kept in next iteration, so it can't count all neighbors. 
For intersection questions, such as "How many TV programs have Bob Boyett and somebody created together?" According to Bob Boyett - productor - {program1, program2, program3... programN} and someone - productor - {program1, program2, program3... programM} , LLM reasoner can output the count of intersections. Similarly, union and negation operations can be implemented by LLM reasoner if a complete neighbor list (before pruning) can be provided.

Reviewer comment (Writing):
The writing of the paper can be largely improved.

Author response:
We sincerely thank the reviewer's suggestions on paper writing. Firstly, we are improving the manuscript writing to make it easier to read. Secondly, we follow your suggestions to add a table explaining all notations in this paper and to refine all notations and equations to be clearer. All these revisions will be added to the final version of the manuscript.

Reviewer comment (Writing):
The mathematical notations and equations can be improved to be clearer.

Author response:
We sincerely thank the reviewer's suggestions on paper writing. Firstly, we are improving the manuscript writing to make it easier to read. Secondly, we follow your suggestions to add a table explaining all notations in this paper and to refine all notations and equations to be clearer. All these revisions will be added to the final version of the manuscript.

Reviewer comment (Writing):
It would be better to summarize the frequently used notations in one table or sentence.

Author response:
We sincerely thank the reviewer's suggestions on paper writing. Firstly, we are improving the manuscript writing to make it easier to read. Secondly, we follow your suggestions to add a table explaining all notations in this paper and to refine all notations and equations to be clearer. All these revisions will be added to the final version of the manuscript.

Reviewer comment (Experiments):
The paper mentions the hallucination problem many times and uses it as the motivation of ToG. However, the hallucination problem is not sufficiently studied; there is only one preliminary analysis of error provided in Appendix D.2.

Author response:
In this article, we don't aim to figure out the whole hallucination problem. Rather, we aim to solve a subset of the hallucination problem: accurate deep-reasoning problem, which account for a large amount of hallucination in LLM reasoning. From the results in Table 1, Table 2 and tables in appendix, we can see ToG brings significant accuracy improvement in deep-reasoning problems compared with LLM-only methods （CoT etc.).

Reviewer comment (Theory):
The paper is empirically driven and lacks in-depth analysis, whether from methodological or theoretical perspectives.

Author response:
Along with the algorithmic research work of ToG, we are studying a theoretical framework for ToG, and even more general, for knowledge-driven LLM. We will add preliminary results of theoretical analysis in the camera-ready version of this manuscript (appendix) if this paper is accepted and a more comprehensive theoretical framework and mathematical analysis will be published in future articles.
   
We briefly highlight the main results of theoretical analysis as follows. A more detailed version of theoretical description can be found at this anonymous document: https://imgbox.com/hI5D5FSu

(1) We reformulate the knowledge-driven LLM problem (where ToG is a special case) as a two-level nested Markov decision process (and thus can be analyzed via RL theory), where the outer loop is the interaction between LLM and human user and the inner loop is the interaction between LLM and external knowledge base such as KG. An illustrative figure can be found at https://imgbox.com/hYXybh91. 

(2) In the inner loop, the reasoner LLM first checks the contents in memory buffer (which we denote as information state) and generate a reasoning action consisting of both knowledge base (KB, instantiated as KG in ToG) query (correspond to the `search` step in ToG) and information filtering (correspond to the `pruning` step in ToG). The KB, serving as the environment, takes the query action and updates the memory buffer content (correspond to transition function in Markov decision process (MDP)). The memory buffer content is also updated deterministically via `pruning`. A judge (instantiated as LLM in ToG) then takes memory buffer content and evaluate its informativeness in terms of answering user question (correspond to the `reasoning` step in ToG), serving as the reward function in MDP. The (information, action, reward) tuple is updated to the memory buffer, and the accumulated memory buffer content serves as information state for the next iteration.

(3) The goal of this MDP (RL problem) is to find a reasoning policy that maximizes its value function: $$V^\pi_\theta(s) = \mathbb{E}\bigl[\sum_{t=0}^\infty \gamma^t r_\theta(s_t, a_t) | s_0=s\bigr]$$
where $\pi$ is the reasoning policy induced by LLM inference. Notably, LLM inference can be considered as performing implicit Bayesian inference [1], where LLM infers the hidden 'concept' contained in the prompts a posteriori, and then generates output based on the concept and prompts. Formally, this process can be written as:
$$ p(y|x) = \int_\theta p(y|\theta, x) p(\theta|x) d\theta,$$ where $\theta$ parametrizes the knowledge environment (i.e. $\theta$ is the underlying knowledge).
For our task of LLM reasoning, the `concept` corresponds to the underlying knowledge that can be inferred from knowledge prompts. In this sense, the induced policy $\pi$ can be considered as Bayesian model-based planning (ToG belongs to this type as a special case), where the LLM reasoner first implicitly estimates the posterior distribution of knowledge ($\theta$) relevant to user problem (based on prior obtained via pretraining and data obtained via KB interaction), and then uses $\theta$ to implicitly parameterize its internal world model and performs model based planning. In this sense, the `pruning` step in ToG can be regarded as a tree search planning algorithm with 0 lookahead step.

(4) Assuming no bias between the LLM prior ($p(\theta)$) and the knowledge prior ($p(\theta^*)$), then the gap between an LLM-induced reasoning policy $\pi$ and an optimal reasoning policy, results only from the uncertainty in posterior estimation of the model parameter, $p(\theta|s_t)$. Since the uncertainty gradually decreases as information accumulates, we can use Bayesian regret [3], R(T), to measure the asymptotic performance of the reasoning policy $\pi$:
$ \mathcal{R}(T) = \mathbb{E}\_{\theta \sim p(\theta)} \bigl[\sum_{t=1}^T V^{\pi^*}\_\theta (s_t) - V^{\pi^t}\_\theta (s_t) \bigr]$
Utilizing proofs in [2] we can guarantee that this regret for LLM x KG is sublinear to the number of reasoning steps $T$ (so $\lim_{T \rightarrow \infty} \frac{R(T)}{T} = 0$), indicating that the reasoning will eventually converge to the correct information/answer due to the continuous uncertainty reduction in posterior inference $p(\theta|s_t)$ thanks to the iterative information gain from KB interaction.

(5) The analysis of the outer user feedback loop can be conducted in a similar way, and the analysis suggests that the response precision can also improve with iterative user feedback, as demonstrated in Figure. 4 of this submission.

Reviewer comment (Evaluation):
What is the running-time efficiency of ToG?

Author response:
Please refer to our submitted official comment "Response to All reviewers about the concerns on the Efficiency of ToG".

Reviewer comment (Experiments):
How do the KGs used in the experiment part solve the limitation of out-of-date knowledge?

Author response:
Firstly, the knowledge in KG is much easier to be updated than the knowledge in LLM. Specifically, the knowledge in KGs is stored on entities and relations (in the form of triples) which are usually physically stored on graph databases, and can be added, deleted and updated immediately. On the contrary, updating knowledge in LLM needs training with new samples which is usually time-consuming. Secondly, in practical scenarios, many KGs are updated frequently or even in real-time by adding new knowledge and deleting or revising out-of-date knowledge. The knowledge updating process can be automatically performed by a knowledge extraction system or manually done with crowdsourcing. For example, the knowledge stored in Wikidata is extracted automatically from Wikipedia and updated regularly according to the changes in Wikipedia content edited by human. Therefore, the knowledge stored in KGs such as Wikidata often has better timeliness than the inner knowledge of LLMs. Thirdly, the version of Wikidata used in this manuscript was released in Jan. 2023 and the latest version was released in Nov. 2023 where the newest knowledge is updated to Wikipedia. In comparison, the cut-off date of the knowledge of ChatGPT is September 2021, about two years ago.