Paper: Keqing: Knowledge-based Question Answering is A Nature Chain-of-Thought mentor of LLMs (ICLR-2024)

Reviewer comment (Novelty):
Recent developments in question answering also consider utilizing graph neural network methods e.g., Question-Answer Sentence Graph for Joint Modeling Answer Selection. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 968–979, Dubrovnik, Croatia. Association for Computational Linguistics.

Author response:
Thanks for notification, and we have cited this paper in our revision already.
We know there are a lot of outstanding KBQA framework built on Graph Neural Network or some other sophisticated networks. However, compared to the performance, the explainability and generalizability should be also considered as the metric of measuring KBQA system. Compared to traditional KBQA systems, Keqing can intuitively explain the logic of generating multi-hop answers, leading to a trustworthy LLM-based KBQA system, which is currently a popular research direction. Meanwhile, Keqing also demonstrated strong generalization abilities by leveraging the reasoning capabilities of LLMs, and we believe its performance can be further improved as the increase of capabilities of LLMs."

Reviewer comment (Novelty):
I have a concern regarding the novelty of the approach. This work simply uses RoBERTa-based similarity scores and DPR-based knowledge augmentation, both works which have already been proposed before. The authors need to better highlight their contributions and why they consider it original work.

Author response:
We acknowledge that question decomposition and candidate reasoning are not fresh ideas in the Q&A domain, however, to the best of our knowledge, this is indeed the inaugural endeavor to build a pipeline integrating these functions through the magic of LLMs. With respect to novelty, we would like to emphasize that we have developed an effective framework that can automate the complete process of knowledge-based question answering with as little extra effort as possible. Note that our pipeline can be training-free with least effort by using off-the-shelf LLMs to finish question decomposition and candidate reasoning while maintaining a decent performance, although a little fine-tuning effort can lead to better performance.
Distinct from traditional KBQA systems, 

In addition, our approach also differs from those recent KBQA systems based on LLMs that typically generate one-step symbolic expressions for the query question leveraging the in-context learning capabilities of LLMs, which is error-prone and lacks transparency.

Reviewer comment (Evaluation):
Can the authors also illustrate the runtime and memory complexity of their work, as it is highly dependent upon LLMs which incur large runtime for finetuning?

Author response:
Thanks for your valuable advice, we have included the additional information about the runtime and memory complexity of our method in Table.8 of Appendix. 
More details can found in Appendix D in the updated version of the paper, where the new added content has been marked in blue.

Specifically, although we can use off-the-shelf LLMs to complete Question Decomposition and Candidate Reasoning, we instead employed a fine-tuned LLM in our experiments to achieve better performance. In practice, we chose to finetune the LLaMA model with 7 billion parameters (LLaMA-7B) using a parameter-efficient fine-tuning technique, i.e., LoRA, which we found to achieve reasonably good performance, finished on two NVIDIA Quadro RTX 8000 graphics cards with 48G memory for each. Thus, the finetuning of Keqing won’t cost too much resources of computation and memory storage.

Reviewer comment (Presentation):
The title is somewhat misleading, obscuring the paper's main contribution. Given that the paper primarily centers on a novel framework integrating an IR module to derive structured data from a knowledge graph, the connection between this pipeline and CoT remains unclear.

Author response:
Thanks, it is correct that main contribution of work is to develop a novel framework to integrate an IR module to retrieve knowledge stored on graph structured datasets to answer the input user question. During the procedure of knowledge retrieval, we will first use LLM to decompose the user question into a series of sub-questions according to predefined question templates, and them as CoT to gradually answer the user question. Considering each question template in a CoT can be solved with logical chains on the knowledge graph, we propose the title ``KNOWLEDGE-BASED QUESTION ANSWERING IS A NATURE CHAIN-OF-THOUGHT MENTOR OF LLMS’’

Reviewer comment (Novelty):
The paper's novelty in comparison to traditional KBQA systems is ambiguous. While elements like question decomposition and candidate reasoning aren't new to the field, it's uncertain whether this is the first instance of such a pipeline being employed with LLMs.

Author response:
To the best of our knowledge, Keqing is indeed the first instance of KBQA framework to integrate the modules of Question Decomposition, Knowledge Retrieval, Candidate Reasoning, and Response Generation into one entity, which is purely deployed with LLMs.

Distinct from traditional KBQA systems that either rely on laborious rule-based algorithms to achieve question decomposition [1] or individually train a graph neural network to perform candidate reasoning [2], our framework is designed to automate the whole process with as little extra effort as possible. Note that our pipeline can be training-free with least effort by using off-the-shelf LLMs to finish question decomposition and candidate reasoning while maintaining a decent performance, although using finetuned LLMs perform better. In addition, our approach also differs from those recent KBQA systems based on LLMs that typically generate one-step symbolic expressions for the query question leveraging the in-context learning capabilities of LLMs, which is error-prone and lacks transparency.

Reviewer comment (Evaluation):
The model's performance in a few-shot scenario appears to lag behind state-of-the-art fine-tuned models, such as Decaf. It would be beneficial to pinpoint the reason for this shortfall or determine which step in the process contributes to this gap.

Author response:
Actually, our framework is a zero-shot KBQA system rather than based on few-shot learning. Because either in the question decomposition stage or in the candidate reasoning stage, our method do not rely on imitating a few demonstrations to generate responses, i.e., we do not follow the paradigm of in-context learning. And in this respect we are consistent with FlexKBQA[3] rather than KB-BINDER[4]. 

As for the performance gap with state-of-the-art models, we assume it is caused by the capability of generalizing of Llama during question decomposition, because the answers can hardly be correct as long as any part of generated CoT is wrong. And conviction is that the performance of Keqing can be further improved with more powerful pretrained LLMs.