Paper: ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
In section 4.3 and 4.4, words such as “somewhat” and “good generative ability” appears in the description yet I am concerned that even with beam search, only 77% of the result lists contain the ground truth logical forms. If the relationships and entities were replaced, how do we ensure that the plugged-in entities/relationships were the right one? In what percentage were the right entities/relationships being plugged in if no ground truth is available?

Author response:
Thank you for your suggestion. During the testing phase of ChatKBQA, our actual practice is as follows: First, we use beam search to generate b candidate logical form sets $C$, and then we try to execute these b logical forms in the order of their beam scores. The first logical form that can be executed is considered our desired logical form. If none can be executed, we proceed with entity retrieval to get a new candidate logical form set $\mathcal{C}^{\prime}$, and try execution again. If still unsuccessful, we conduct relationship retrieval to obtain $\mathcal{C}^{\prime\prime}$, and then find the answer. Therefore, with each entity and relationship retrieval, the number of test samples with executable SPARQL increases, ensuring that the insertion of entities and relationships doesn't turn a correct logical form into an incorrect one, only improving the results.

Since we select logical forms to execute and obtain KBQA answers based on whether they can be converted into executable SPARQL, some entities and relationships might be correctly retrieved but may not combine correctly. Therefore, we believe that calculating the percentage of correct entities/relationships being plugged in is not as meaningful as assessing the accuracy of the final executed answers. In Section 5.3, to validate entity retrieval (ER) and relation retrieval (RR) separately, we removed ER or RR from the framework and obtained three simplified variants (ChatKBQA w/o ER, ChatKBQA w/o RR, and ChatKBQA w/o ER,RR) at four different beam sizes for comparison. As shown in Figure 4, ER and RR improve about 15 and 5 percentage points on average over no oracle entity linking in the F1 score at different beam sizes, respectively. This proves the effectiveness of ER and RR, and combined with ER, RR makes KBQA perform at its best.

Reviewer comment (Evaluation):
In section 4.5, the authors claim that Graph-Query-of-Thoughts are a way to improve QA’s interpretability and avoid LLM’s hallucinations, which has no evidence support in the result/analysis section. This seems to be an exaggerated claim and I am not convinced.

Author response:
Thank you for pointing out this issue. We proposed Graph-Query-of-Thoughts to improve QA’s interpretability and avoid LLM’s hallucinations, intending to illustrate a new paradigm combining LLM and KG, namely using fine-tuned LLMs to generate graph database query languages, then executing them. This replaces LLM reasoning with graph database queries on the knowledge graph, aiming to improve interpretability and avoid hallucinations in QA with strict interpretability requirements. This is a concept and also the direction of our future work.

To avoid misunderstandings, we have moved this content to Appendix F, Future Directions, in the updated paper.

Reviewer comment (Presentation):
Presentation of the paper needs improvement. Multiple grammatical errors, and the description of the method is confusing. Explanation of methods like QLora.etc can be moved to related work, since now it is interrupting the flow of the writing.

Author response:
Thank you for pointing out these issues. We have comprehensively revised the entire article, checking and correcting colloquialisms and grammatical errors based on feedback from all reviewers. Regarding explanations of methods like QLora, etc., we have moved the description of efficient fine-tuning methods in Section 4.2 to related work and Appendix B.2 to improve readability and coherence.

Reviewer comment (Experiments):
In section 4.2, why does chatKBQA converts the SPARQL corresponding to the test set in the KBQA dataset into logical form? Why test set?

Author response:
Thank you for pointing out this issue. The term 'test set' should be changed to 'train set', and we have made this correction in the updated paper.

Reviewer comment (Experiments):
How do you compare against chatGPT and GPT-4? The paper says that chatGPT and GPT-4 failed to generate logical form, is this zero-shot or few-shot?

Author response:
During the generation phase of ChatKBQA, we replaced our fine-tuned open-source LLMs with prompts to have ChatGPT and GPT-4 generate SPARQL in a zero-shot manner. As mentioned in the updated Section 5.5, ChatGPT and GPT-4, despite their large parametric capacities, struggle to generate standard logical forms effectively because they are not open-source and thus cannot be fine-tuned. They can produce SPARQL language, but constructing the correct query skeleton, entities, and relations is challenging. This difficulty arises from their limited ability to perceive the complex structure of the external knowledge base (KB) through designed prompts within a restricted context length.

Reviewer comment (Experiments):
ChatKBQA seems to rely on “ground truth logical forms”, which might be a rare resource. The datasets used are outdated since the task is now a less popular task, and I am curious of how chatKBQA will be useful in an open-domain QA era.

Author response:
We added Appendix F to discuss the insights and future directions of ChatKBQA. ChatKBQA is the first method to fine-tune open-source LLMs using unsupervised retrieval methods for the KBQA task, achieving state-of-the-art results. Therefore, its effectiveness depends on the quality of the dataset used for mapping natural language to logical forms. In future work, we plan to enhance the training set by extracting computation graphs from the knowledge graph using graph sampling, then translating these into natural language. This approach aims to explore methods to achieve maximum training effectiveness with the least amount of training data.

Furthermore, there is a growing demand for precise knowledge question-answering in various fields, including open-domain, medicine, finance, and telecommunications. We can first use technologies like UIE or LLM information extraction to build a knowledge graph. Then, by fine-tuning ChatKBQA to understand the structure of the knowledge graph, we aim to achieve interpretable knowledge question-answering in both open and specific domains.

Reviewer comment (Evaluation):
Lacking analysis of the failure examples, I am curious which module caused the error when the proposed framework did not get the correct answer.

Author response:
Thank you for your suggestion. We have added Section 5.8, expanding on the error analysis of ChatKBQA. The errors can be divided into four categories: Logical form skeleton error (40.10%), Entity retrieval error (27.17%), Relation retrieval error (19.48%), and SPARQL conversion error (13.26%).

The majority of the errors are caused by ChatKBQA failing to provide the correct logical form skeleton for the problem, which is due to the limited representation of certain complex skeletons in the training set. Errors in entity and relation retrieval are due to our use of unsupervised retrieval methods. Finally, a small proportion of the remaining errors arise from the fact that, although the generated logical form is consistent with the ground truth, it fails to execute or the answers are inconsistent when converted to SPARQL, which may be caused by the loss of the conversion from logical form to SPARQL or possibly changes in the KB. Therefore, further addressing these errors is the direction of our future efforts.

Reviewer comment (Experiments):
It would be better to list the detailed statistics of the datasets, *e.g.,* number of the skeletons of logical forms involved in the training set and the test set, etc.

Author response:
Thank you for your suggestion. We have added Appendix C to provide statistics on the datasets used in the experiments. This includes the number of questions, types of logical form skeletons, entities and relations involved, the scale of the training, validation, and test sets, as well as information on the databases relied upon.

Reviewer comment (Evaluation):
Why does the beam size have such an influence on the performance?

Author response:
We addressed this question in the updated Section 5.3 (ABLATION STUDY), second finding and analysis. Beam search is a heuristic algorithm usually used in sequence generation tasks, which expands the search space by generating multiple highly probable logical forms instead of only one. As shown in Figure 4, an increase in beam size enhances the likelihood of executing SPARQL queries based on candidate logical forms, improving the KBQA performance.

Reviewer comment (Evaluation):
During the test phase, when the fine-tuned LLMs generate logical forms, can they correctly generate relations not seen in the training set? I guess they can generate the right entities even though those entities have not appeared in the training set before, as seed entities are typically included in the natural language questions. But it is not necessarily for relations, this is why I'm curious about the result that ChatKBQA w/o RR is better than ChatKBQA w/o ER.

Author response:
We addressed this question in the updated Section 5.6 (ANALYSIS OF EFFICIENCY OF RETRIEVAL IN RETRIEVAL PHASE), second finding and analysis. As shown in Figure 6, RR enhances F1 score by an average of 5% across various beam sizes in ablation experiments. Although relations are rarely directly present in natural language problems, the number of thousand-level relations in the KB is still small compared to the tens of millions of entities, and the LLM perceives relational information well during fine-tuning. Thus, RR does not improve performance as much as ER, but combined with ER, RR makes KBQA perform at its best. And also, on this basis, for unseen relations, they can be obtained through the generation helped with semantic retrieval.

Reviewer comment (Novelty):
The proposed method is similar to semantic parsing-based methods, which focus on translating questions into logical forms executable against KBs, such as SPARQL, query graph, and S-expression, as discussed in the related work.

Author response:
As mentioned in the Related Work section, ChatKBQA can be categorized as an SP-based (Semantic Parsing-based) KBQA method. However, unlike previous SP-based methods, ChatKBQA introduces a new approach of generating and then retrieving to separately enhance the effects of generation and retrieval. In the generation phase, it uses a fine-tuned large language model to generate a logical form. In the retrieval phase, it employs an unsupervised method for entity and relation retrieval, achieving state-of-the-art results in the KBQA task. Additionally, ChatKBQA provides new insights into combining LLMs with KGs for interpretable knowledge question-answering.

Reviewer comment (Experiments):
Considering the authors replaced the backbone model with LLMs, it is hard to identify which parts play the key role in performance improvement. It would be better to have more ablation studies. For example, replacing LLM with the pre-trained model in baseline, such as T5, or fixing LLM and replacing the parsing and IR component with the baseline method.

Author response:
We believe that ChatKBQA consists of two stages: generation and retrieval. In the updated paper, we conducted ablation analyses on four key aspects: fine-tuning of the large model, beam search, entity retrieval, and relation retrieval, as detailed in Section 5.3. The results demonstrate that each of these aspects contributes to the improvement of the KBQA performance.

To further analyze the enhancements in the generation and retrieval stages of ChatKBQA, in the updated Section 5.5, we replaced the LLM component with models like ChatGPT and T5. This validates the advantages of fine-tuning open-source LLMs like Llama-2 in ChatKBQA. In Section 5.6, we changed the previous pre-retrieval to post-retrieval to verify improvements in retrieval efficiency during the retrieval phase.

Reviewer comment (Presentation):
Efficient fine-tuning methods seem not related to the claims in the paper. There's no need to discuss it in the methodology section.

Author response:
Thank you for your suggestion. We have moved the content on efficient fine-tuning methods to Appendix B.2 in the latest version of the paper.

Reviewer comment (Experiments):
As mentioned in the weaknesses, which part plays a crucial role in model improvement?

Author response:
We believe that, based on our experimental analysis, fine-tuning the large model, beam search, entity retrieval, and relation retrieval each play a significant role in enhancement. Their combined effect has enabled ChatKBQA to achieve state-of-the-art performance in the KBQA task.