C hat KBQA : A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models (2024)

Abstract: Knowledge Base Question Answering (KBQA) aims to answer natural language questions over large-scale knowledge bases (KBs), which can be summarized into two crucial steps: knowledge retrieval and semantic parsing. However, three core challenges remain: inefficient knowledge retrieval, mistakes of retrieval adversely impacting semantic parsing, and the complexity of previous KBQA methods. To tackle these challenges, we introduce ChatKBQA, a novel and simple generate-then-retrieve KBQA framework, which proposes first generating the logical form with fine-tuned LLMs, then retrieving and replacing entities and relations with an unsupervised retrieval method, to improve both generation and retrieval more directly. Experimental results show that ChatKBQA achieves new state-of-the-art performance on standard KBQA datasets, WebQSP, and CWQ. This work can also be regarded as a new paradigm for combining LLMs with knowledge graphs (KGs) for interpretable and knowledge-required question answering.

## introduction knowledge base question answering (kbqa) is a classical nlp task to answer natural language questions based on facts over a large-scale knowledge base (kb), such as freebase @xcite , wikidata @xcite , and dbpedia @xcite , which are composed of structured knowledge graphs (kgs) built from triples consisting of (head entity, relation, tail entity). previous kbqa methods primarily addressed two core issues: knowledge retrieval @xcite and semantic parsing @xcite . knowledge retrieval mainly aims to locate the most relevant entities, relations, or triples according to the question from kb, to narrow the ## preliminaries in this section, we define two basic concepts of our work: the knowledge base and the logical form, followed by the problem statement for kbqa task. definition 1: knowledge base (kb). a kb @xmath0, r, o)|@xmath1, @xmath2, @xmath3, r, o) where s is an entity, r is a relation , and o can be an entity or a literal. each entity @xmath4, e.g., e.@xmath5.0fm2h", which can be queried to get its label as e.@xmath6. each relation @xmath7, e.g. @xmath8. government_position_held. appointed_by". definition 2: logical form. a logical form is a structured representation of a natural language question. taking the s-expression as an example, a logical form usually consists of projection and various operators. projection operation represents a one-hop query of a triple (s, r, o) on s or o, where, (?, r, o) is denoted as (join r o), while (s, r, ?) is denoted as (join (r r) s). other operators, e.g. "and", "count", and "argmax", are introduced in appendix a. problem statement. for kbqa task, given a natural language question q, and a knowledge base k, we need to first convert q into a logical form @xmath9, where sp(.) is a semantic parsing function. then convert f to the equivalent sparql query @xmath10, where convert(.) is the fixed conversion function. finally, the final set of answers @xmath11, where execute(.) is the query execution function. prefix ns: <http ://rdf.freebase.com/ns/ > select distinct ?x where { filter (?x != ns:m.06w2sn5) filter (!isliteral(?x) or lang(?x) = '' or langmatches(lang(?x), 'en')) ns:m.06w2sn5 ns:people.person.sibling_s ?y . ?y ns:people.sibling_relationship.sibling ?x . ?x ns:people.person.gender ns:m.05zppz . } 44 45 46 efficient fine-tuning on open-source llms unsupervised retrieval for entities & relations fine-tuned llms for semantic parsing beam search open-source llms candidate logical forms updated candidate logical forms sparql query answers input nl question generation phase retrieval phase relation set of kb candidate relation tuple:( [ tv , regular tv appearance , actor ], [ tv , regular tv appearance , character ], [ tv , tv program , regular cast ] ) r 1 r 2 r 3 probability tv.regular_tv_ appearance.actor tv.regular_tv_ appearance.character tv.tv_program. regular_caset 95.72% tv.regular_tv_ appearance.actor tv.regular_tv_ appearance.actor tv.tv_program. regular_caset 78.16% relation retrieval ## methodology in this section, we first present an overview of the chatkbqa framework as shown in figure 3 , and introduce efficient fine-tuning on large language models (llms), logical form generation by finetuned llms, unsupervised retrieval for entities and relations, and interpretable query execution. ## experiments this section presents the experimental setup, results, and analysis. we answer the following research questions (rqs): rq1: does chatkbqa outperform other kbqa methods? rq2: does the main components of chatkbqa work? rq3: why use generate-then-retrieve method instead of retrieve-then-generate method? rq4: why use fine-tuned open-source llms instead of calling chatgpt or training traditional t5 models? rq5: does generate-then-retrieve method improve retrieval efficiency? rq6: does chatkbqa has plug-and-play characteristics? ## conclusion in this work, we introduce chatkbqa, a generatethen-retrieve kbqa framework that utilizes advanced fine-tuned llms, which overcomes traditional challenges like retrieval inefficiencies, semantic parsing errors, and complexity of kbqa methods. experimental results on webqsp and cwq benchmarks show that chatkbqa achieves a new state-of-the-art kbqa performance. its simplicity, flexibility, and plug-and-play make it an effective approach for combining llm and kg in interpretable knowledge-required kbqa tasks.