Paper: Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning (ICLR-2024)

Reviewer comment (Theory):
From equation 1, I understand the authors want to decompose the prediction task into two parts: getting hidden variable z and then predicting the answer with the hidden variable. However, it's not clear whether equations 2-6 are necessary. Could you provide more intuition behind for explanation?

Author response:
We thank the reviewer for the question and for pointing out the issue in Equation 3. 

In equation 1, we want to illustrate the general purpose of RoG, which first generates a set of plans (planning module) and then conducts the reasoning on KGs with the plans (retrieval-reasoning module). However, directly optimizing Equation 1 is intractable due to the combination of hidden variable $z$ [1]. Therefore, we derive the evidence lower bound (ELBO) of Equation 1 in Equation 2-6 and come up with the viable optimization objective in Equation 7.

To clarify the misunderstanding, we want to correct Equation 3 in our paper as follows:
$$
\begin{equation}
    Q(z) \simeq Q(z|a,q,\mathcal{G}) = \begin{cases}
    & \frac{1}{|\mathcal{Z}|}, \exists w_z(e_q,e_a)\in \mathcal{G},\\\\
    & 0, \text{else},
    \end{cases}
\end{equation}
$$
where we assume there are multiple plans $z\in\mathcal{Z}$ which obey a uniform distribution [1].

[1] Singh, D., Reddy, S., Hamilton, W., Dyer, C., & Yogatama, D. (2021). End-to-end training of multi-document reader and retriever for open-domain question answering. NeurIPS 2021, 34, 25968-25981.

Reviewer comment (Theory):
Even if motivation exists, I'm not sure why equation 3 holds. Why is there only one path / relation path and it's faithful? For example, there could be multiple solutions/paths; if so, equation 3 might not hold. The authors should provide more justification for the equation 3.

Author response:
We thank the reviewer for the question and for pointing out the issue in Equation 3. 

In equation 1, we want to illustrate the general purpose of RoG, which first generates a set of plans (planning module) and then conducts the reasoning on KGs with the plans (retrieval-reasoning module). However, directly optimizing Equation 1 is intractable due to the combination of hidden variable $z$ [1]. Therefore, we derive the evidence lower bound (ELBO) of Equation 1 in Equation 2-6 and come up with the viable optimization objective in Equation 7.

To clarify the misunderstanding, we want to correct Equation 3 in our paper as follows:
$$
\begin{equation}
    Q(z) \simeq Q(z|a,q,\mathcal{G}) = \begin{cases}
    & \frac{1}{|\mathcal{Z}|}, \exists w_z(e_q,e_a)\in \mathcal{G},\\\\
    & 0, \text{else},
    \end{cases}
\end{equation}
$$
where we assume there are multiple plans $z\in\mathcal{Z}$ which obey a uniform distribution [1].

[1] Singh, D., Reddy, S., Hamilton, W., Dyer, C., & Yogatama, D. (2021). End-to-end training of multi-document reader and retriever for open-domain question answering. NeurIPS 2021, 34, 25968-25981.

Reviewer comment (Experiments):
One of the key points in the framework is the finetuned LLM (i.e., \theta). However, in the main paper, it's not clear how the LLM is finetuned. It seems the objective functions follow the equation 7. However, these two modules are not evaluated individually and there is no validation loss provided. Only the final framework performance can prove that these modules work as expected, which is not sufficient for readers to know why they can work.

Author response:
We thank the reviewer for the question. We have revised the paper to clarify the training process of RoG in equation 7 (based on the comments of Reviewer 1bmQ). The objective in equation 7 contains two terms: retrieval-reasoning optimization and planning optimization.  We optimize the objective in Equation 7 by maximizing the likelihood of predicting the correct answer and generating valid plans. 

To demonstrate the performance of each module individually, we conduct ablation studies in Table 3, where we remove the planning and reasoning modules, respectively. From the results, we can see that both these modules are helpful for the performance of RoG.

Ablation studies of RoG.

| Method            | WebQSP (F1) | CWQ (F1) |
| ----------------- | ----------- | -------- |
| RoG               | 70.81       | 56.17    |
| RoG w/o planning  | 49.69       | 33.76    |
| RoG w/o reasoning | 49.56       | 22.26    |

Reviewer comment (Evaluation):
Over-claim sentences. The authors claim that their RoG framework can address the hallucination issue and lack of knowledge issue. However, studies with several cases are definitely not sufficient to prove them. I would suggest adding more comprehensive results or changing the claim.

Author response:
Thank you for the comments. We have revised the paper by changing the claim to "RoG can **alleviate** the hallucination issue and lack of knowledge issue".

Reviewer comment (Experiments):
For the KG, is that constructed by yourself and used for all tasks, or is it provided in the dataset?

Author response:
We use the KGs preprocessed by previous works [2].

[2] He, G., Lan, Y., Jiang, J., Zhao, W. X., & Wen, J. R. (2021, March). Improving multi-hop knowledge base question answering by learning intermediate supervision signals. WSDM 2021

Reviewer comment (Theory):
In equation 4, the final equality, the constant term is missing.

Author response:
Thanks for pointing it out, we have revised Equation 4. We omit the const in the final objective since it makes no contributions to the optimization. The detailed derivation can be found in Appendix A.1 and in our response to Reviewer 1bmQ.
$$
\begin{align}
    \mathcal{L}\_{\text{plan}} = D\_{KL}(Q(z)||P\_\theta(z|q)) &= D\_{KL}(Q(z|a, q, \mathcal{G})||P\_\theta(z|q)), \\\\
    &\simeq - \frac{1}{|\mathcal{Z^*}|}\sum\_{z\in\mathcal{Z^*}} \log P\_\theta(z|q),
\end{align}
$$

Reviewer comment (Experiments):
In Table 2, RoG is finetuned LM, but it's compared with LLMs under the zero/few-shot setting. Is that a fair comparison?

Author response:
Thanks for the questions, we have revised our paper by providing the performance of LLM finetuned on the training set of WebQSP and CWQ. From the results, we can see that RoG still outperforms the LLMs finetuned on the training set of WebQSP and CWQ.  We have revised the paper by providing the results in Appendix A.7.3.

Performance comparison with various finetuned LLMs.

| Method                       | WebQSP (Hits@1) | CWQ (Hits@1) |
| ---------------------------- | --------------- | ------------ |
| Alpaca-7B  (Zero Shot)         | 51.78           | 27.44        |
| LLaMA2-Chat-7B  (Zero Shot) | 64.37           | 34.60        |
| Alpaca-7B (Finetuned)           | 74.57           | 55.98        |
| LLaMA2-Chat-7B  (Finetuned)  | 73.89           | 53.49        |
| RoG                          | 85.75           | 62.65        |

Reviewer comment (Theory):
There is some nonrigorous math in the paper. e.g. in equation 4, the expectation and Q should not coexist. It's either $\mathbb{E}_Q \log P$ or $\sum_z Q \log P$. In the next line, $z \in Q$ does not make sense as $Q$ is a probability distribution. Also, the equality does not make sense as there is a CONST in equation 4. Also, I don't think it's a good idea to use equality for an approximation. Similar nonregorousness happens in equation 6. The marginalization in equation 10 does not make sense, as the authors are marginalizing over the conditions. The correctness of the final training objective needs to be double-checked.

Author response:
We truly thank the reviewer for pointing out these issues! We have carefully revised the paper to fix the nonrigorous math equations.

The revised Equation 4 in our paper is shown here. 
$$
\begin{align}
    \mathcal{L}\_{\text{plan}} = D\_{KL}(Q(z)||P\_\theta(z|q)) &= D\_{KL}(Q(z|a, q, \mathcal{G})||P\_\theta(z|q)),\tag{1} \\\\
    &\simeq - \frac{1}{|\mathcal{Z^*}|}\sum\_{z\in\mathcal{Z^*}} \log P\_\theta(z|q),\tag{2}
\end{align}
$$
where $\mathcal{Z}^*\subseteq\mathcal{Z}$ denotes the set of shortest paths between $e_q$ and $e_a$ in KGs, and $\mathcal{Z}$ denotes all valid relation paths in KGs.

Due to the limitation of space, we put the detailed derivation in Appendix A.1, which is also shown as follows.

The posterior distribution $Q(z)$ can be approximated by $Q(z|a, q, \mathcal{G})$, given as
$$
\begin{equation}
    \tag{3}
    Q(z) \simeq Q(z|a,q,\mathcal{G}) = \begin{cases}
    & \frac{1}{|\mathcal{Z}|}, \exists w_z(e_q,e_a)\in \mathcal{G},\\\\
    & 0, \text{else},
    \end{cases}
\end{equation}
$$
where we assume a uniform distribution over all valid relation paths $\mathcal{Z}$ [1].

Thus, the KL divergence can be calculated as
$$
\begin{align}
\mathcal{L}\_{\text{plan}} = D\_{KL}(Q(z)||P\_\theta(z|q)) &= D\_{KL}(Q(z|a, q, \mathcal{G})||P\_\theta(z|q)), \tag{4}\\\\
&=\mathbb{E}\_{z\sim Q(z|a, q, \mathcal{G})}[\log Q(z|a, q, \mathcal{G})-\log P\_\theta(z|q)], \tag{5}\\\\
&= - \mathbb{E}\_{z\sim Q(z|a, q, \mathcal{G})}\log P\_\theta(z|q) + \text{CONST}, \tag{6}
\end{align}
$$
where the expectations cannot be computed exactly because of the large number of valid relation paths $\mathcal{Z}$, so we approximate it by using the set of shortest paths $\mathcal{Z}^*\subseteq\mathcal{Z}$ between $e_q$ and $e_a$ in KGs. This can be formulated as
$$
\begin{align}
\mathcal{L}\_{\text{plan}} = - \sum\_{z\in\mathcal{Z^*}} Q(z|a, q, \mathcal{G}) \log P\_\theta(z|q) + \text{CONST}. \tag{7}
\end{align}
$$
By assuming a uniform distribution over the set of shortest paths $\mathcal{Z}^*$, we can rewrite Eq. (7) as
$$
\begin{align}
\mathcal{L}\_{\text{plan}} & = - \frac{1}{|\mathcal{Z^*}|}\sum\_{z\in\mathcal{Z^*}} \log P\_\theta(z|q) + \text{CONST}.\tag{8}
\end{align}
$$
We keep the CONST in the equation. However, we omit it in the final objective since it makes no contributions to the optimization. 

The revised Equation 6 in our paper is presented here.
$$
\begin{align}
    \mathcal{L}\_{\text{reason}} &= \mathbb{E}\_{z\sim Q(z|a, q, \mathcal{G})}[\log P\_\theta(a|q,z,\mathcal{G})],\tag{9}\\\\ 
    &= \sum\_{z\in\mathcal{Z}^*\_K} \log P\_\theta(a|q,z,\mathcal{G}), \tag{10}\\\\ 
    &=  \log P\_\theta(a|q,\mathcal{Z}^*\_{K},\mathcal{G}). \tag{11}
\end{align}
$$
The expectation is approximated by sampling $K$ plans from the $\mathcal{Z^*}$, denoted as $\mathcal{Z}^*_K\subseteq \mathcal{Z}^*$. The conversion from Eq. (9) to Eq. (10) is based on the FiD framework [1], where we can simultaneously use multiple plans for reasoning. The FiD framework is introduced in Equation 5 of our paper, which is also shown here.

The FiD framework assumes each $z\in \mathcal{Z}$ contributes independently. The probability $P_\theta(a|q,\mathcal{Z},\mathcal{G})$ can be approximated as the product of each $P_\theta(a|q,z,\mathcal{G})$, which is formulated as
$$
\begin{equation}
    P\_\theta(a|q,\mathcal{Z},\mathcal{G}) = \prod\_{z\in\mathcal{Z}} P\_\theta(a|q,z,\mathcal{G}). \tag{12}
\end{equation}
$$

We also want to clarify the marginalization in Equation 10 of our paper is reasonable based on the FiD framework shown in Eq. (12), which can be derived as

$$
\begin{align*}
&\text{Because:} \\\\
& P\_\theta(a|q,\mathcal{Z},\mathcal{G}) = \prod\_{z\in\mathcal{Z}} P\_\theta(a|q,z,\mathcal{G}), \\\\
&\Rightarrow \\\\
&\log P\_\theta(a|q,\mathcal{Z}^*\_K, \mathcal{G}) = \log \sum\_{z\in\mathcal{Z}^*\_K} P\_\theta(a|q,z,\mathcal{G}), \\\\
&\Rightarrow \\\\
&\underset{\theta}{\operatorname{arg max}}\log P\_\theta(a|q,\mathcal{Z}^*\_K, \mathcal{G}) = \underset{\theta}{\operatorname{arg max}} \log \sum\_{z\in\mathcal{Z}^*\_K} P\_\theta(a|q,z,\mathcal{G}), \\\\
&= \underset{\theta}{\operatorname{arg max}} \log \sum\_{z\in\mathcal{Z}^*\_K}\sum\_{w\_z\in\mathcal{W}\_z}\prod\_{t=1}^{|a|}P\_\theta(t_i|t_{<i}, q, w_z), \\\\
&\text{Q.E.D.}
\end{align*}
$$

[1] Singh, D., Reddy, S., Hamilton, W., Dyer, C., & Yogatama, D. (2021). End-to-end training of multi-document reader and retriever for open-domain question answering. Advances in Neural Information Processing Systems, 34, 25968-25981.

Reviewer comment (Experiments):
More datasets to showcase the effectiveness of the proposed method would be great, as there are currently only two in the paper. Would the fine-tuned LLM generalize to other QA datasets, in addition to the datasets that it is fine-tuned on?

Author response:
In Appendix A.7.1, we evaluate the transferability of RoG to other KGs. We select the MetaQA-3hop dataset, which is based on Wiki-Movies KGs. The experiment results show that RoG trained on Freebase KGs can be effectively transferred to Wiki-Movies KGs with only 1000 samples. The performance of transferring is better than training from scratch. This indicates that RoG empowers the reasoning on graph ability via training, which can be transferred to new KGs to improve the performance. We also present the training and transferring time here. The transferring time is much less than the training time.

Performance on MetaQA-3hop (F1)

|  F1   | Only Train on Wiki-Movies | Transferring from Freebase |
| :---: | :-----------------------: | :------------------------: |
|  RoG  |           41.32           |           50.68            |

Training time comparison.

| Time | Training on Freebase | Transferring to Wiki-Movies |
| :--: | :------------------: | :-------------------------: |
| RoG  |        38 hours        |            2 hours            |

Reviewer comment (Evaluation):
About RoG as a planning module for other LLMs: I understand that the fine-tuned LLM can also be combined with other LLMs, and improve the performance of these not fine-tuned LLMs. However, according to Table 4, even combining with a stronger LLM (e.g. ChatGPT) cannot improve upon the original fine-tuned LLM. I don't see the usefulness of having this sort of integrability.

Author response:
The reason why other LLMs, when integrated with the planning module of RoG cannot outperform the full RoG is that these LLMs are not trained to understand the reasoning paths and conduct reasoning, as mentioned in Section 4.2. This also demonstrates the importance of reasoning optimization.

However, even without training, This experiment shows that the plans generated by the planning module still exhibit great interpretability and can be used by other LLMs to improve their performance. This further opens the possibility of separating the reasoning and generation modules of RoG in the future. For example, we can use a relatively simple LLM to generate the plans and train a larger and more powerful LLM to conduct reasoning based on the generated plans. This experiment shows the flexibility of RoG in actual practices.

Reviewer comment (Reproducibility):
The evaluation against ChatGPT appears to use 3.5-turbo. Please clarify, including the dates of the evaluations -- the implementation of ChatGPT changes over time.

Author response:
We greatly appreciate the reviewer’s recognition of our work. We want to clarify that we use ChatGPT-3.5-turbo in our experiments. Experiments are conducted with models released between July. to Sept., 2023. We have revised the paper (Appendix A.5) to clarify this point.

Reviewer comment (Reproducibility):
Can you clarify which version of ChatGPT was used?

Author response:
We greatly appreciate the reviewer’s recognition of our work. We want to clarify that we use ChatGPT-3.5-turbo in our experiments. Experiments are conducted with models released between July. to Sept., 2023. We have revised the paper (Appendix A.5) to clarify this point.

Reviewer comment (Presentation):
The approach section was hard to read. More specifically, the order of explanation was a bit hard to follow. Before explaining the optimization, explaining the flow of information step-by-step would be helpful when pointing to Figure 3 in the beginning.

Author response:
Thank you for your comment. We have revised the approach section to improve its readability. Regarding the framework, we have introduced the components of our method at the beginning and described them in the caption of Figure 2, where we provide a step-by-step description of our approach. 

Additionally, regarding the issue of what ground-truth supervision is, we would like to clarify that we use the shortest paths that connect the questions and answer entities in the KG as the supervision signals. We have revised Equation 4 in our paper to clarify this point. The detailed derivation can be found in Appendix A.1 and our responses to Reviewer 1bmQ.

Reviewer comment (Writing):
In the optimization part, explaining what kind of ground-truth supervision is used was not very explicit. Using the retrieved paths from the KG as a source of supervision could be made clear earlier in the approach.

Author response:
Thank you for your comment. We have revised the approach section to improve its readability. Regarding the framework, we have introduced the components of our method at the beginning and described them in the caption of Figure 2, where we provide a step-by-step description of our approach. 

Additionally, regarding the issue of what ground-truth supervision is, we would like to clarify that we use the shortest paths that connect the questions and answer entities in the KG as the supervision signals. We have revised Equation 4 in our paper to clarify this point. The detailed derivation can be found in Appendix A.1 and our responses to Reviewer 1bmQ.

Reviewer comment (Evaluation):
If trained with a specific KG, the results will improve in answering questions from that specific KG — which is indeed the scope of this work — however, it is unclear if this helps the LLM's general QA capability, including addressing issues such as hallucination and lack of general knowledge.

Author response:
Admittedly, the training process of our method could be costly. However, we want to emphasize that after training, our method enables the generalizability of transferring to new KGs and QA datasets with only a few steps of finetuning. 

In Appendix A.7.1, we evaluate the transferability of RoG to other KGs. We select the MetaQA-3hop dataset, which is based on Wiki-Movies KGs. The experiment results show that RoG trained on Freebase KGs can be effectively transferred to Wiki-Movies KGs with only 1000 samples. The performance of transferring is better than training from scratch. This indicates that RoG empowers the reasoning on graph ability via training, which can be transferred to new KGs to improve the model's performance on this new KG. We also present the training and transferring time below (also in Table 12). As can be seen, the transferring time is much less than the training time.

Performance on MetaQA-3hop (F1)

|  Method   | Only Train on Wiki-Movies | Transferring from Freebase |
| :---: | :-----------------------: | :------------------------: |
|  RoG  |           41.32           |           50.68            |

Training time comparison.
| Time | Training on Freebase | Transferring to Wiki-Movies |
| :--: | :------------------: | :-------------------------: |
| RoG  |        38 hours        |            2 hours            |

Reviewer comment (Reproducibility):
It was unclear whether, when referring to the retrieval and reasoning/planning modules of ROG, those are the outcome of instruction-tuning of a specific large language model.

Author response:
We use LLaMA2-Chat-7B as the LLMs in our experiments. When we combine RoG with other language models, we use the planning module to generate plans (relation paths). The plans are executed on KGs to retrieve the reasoning paths. The retrieved paths are fed into different LLMs during inference using the reasoning prompts template shown in Appendix A.10. This enables them to be combined with RoG without retraining. We have revised the paper by adding more details to the implementation settings in Appendix A.6.

Reviewer comment (Experiments):
When combining ROG with other language models in Table 4, it was not clear which one was used again in the planning module.

Author response:
We use LLaMA2-Chat-7B as the LLMs in our experiments. When we combine RoG with other language models, we use the planning module to generate plans (relation paths). The plans are executed on KGs to retrieve the reasoning paths. The retrieved paths are fed into different LLMs during inference using the reasoning prompts template shown in Appendix A.10. This enables them to be combined with RoG without retraining. We have revised the paper by adding more details to the implementation settings in Appendix A.6.