Interactive- KBQA : Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models (2024)

Abstract: This study explores the realm of knowledge base question answering (KBQA). KBQA is considered a challenging task, particularly in parsing intricate questions into executable logical forms. Traditional semantic parsing (SP)-based methods require extensive data annotations, which result in significant costs. Recently, the advent of few-shot in-context learning, powered by large language models (LLMs), has showcased promising capabilities. Yet, fully leveraging LLMs to parse questions into logical forms in low-resource scenarios poses a substantial challenge. To tackle these hurdles, we introduce Interactive-KBQA, a framework designed to generate logical forms through direct interaction with knowledge bases (KBs). Within this framework, we have developed three generic APIs for KB interaction. For each category of complex question, we devised exemplars to guide LLMs through the reasoning processes. Our method achieves competitive results on the WebQuestionsSP, ComplexWebQuestions, KQA Pro, and MetaQA datasets with a minimal number of examples (shots). Importantly, our approach supports manual intervention, allowing for the iterative refinement of LLM outputs. By annotating a dataset with step-wise reasoning processes, we showcase our model’s adaptability and highlight its potential for contributing significant enhancements to the field.

## introduction knowledge base question answering (kbqa) is an increasingly significant research area that leverages structured knowledge bases (kbs) to provide precise answers to natural language (nl) questions. a kbqa system aims to harness the vast, structured information residing in kbs, such as freebase @xcite , wikidata (vrandečić and krötzsch, 2014), or domain-specific databases like the movie kb @xcite . accurately interpreting and responding to user inquiries with data from knowledge bases has potent applications across numerous sectors, making it a focal point of both academic research and industrial innovation. recent advancements in kbqa research can generally be classified into two primary approaches: information retrieval (ir)-based methods and semantic parsing (sp)-based methods. ir-based methods focus on understanding the query, retrieving pertinent subgraphs from kbs related to the question, and extracting the answer from these subgraphs @xcite @xcite . conversely, sp-based approaches convert nl questions into executable logical forms, leveraging pre-trained generative models to interact with the knowledge base and generate answers @xcite @xcite . the emergence of large language models (llms), such as chatgpt @xcite and @xcite , has opened new avenues for enhancing kbqa systems. these models have demonstrated promising results in reasoning @xcite and few-shot learning ca-pabilities (chen, 2023), setting new benchmarks in the kbqa domain @xcite @xcite @xcite . despite these advancements, kbqa systems face several pressing challenges. ## related work knowledge base question answering (kbqa) methods can be broadly classified into two distinct categories: information retrieval (ir)-based methods and semantic parsing (sp)-based methods. these approaches address the challenge of effectively mapping natural language (nl) queries to the structured formats of knowledge bases (kbs). ir-based methods aim to extract a questionspecific subgraph from the kb and employ ranking algorithms to select the top entities or directly generate answers using text decoders @xcite . to bridge the gap between unstructured texts and structured @xcite proposed a structured knowledge-aware pre-training method to enhance complex subgraph representation learning. emphasizing the accuracy of subgraph retrieval, @xcite developed a trainable, decoupled subgraph retriever that boosts the performance of subgraph-oriented kbqa models. further, @xcite introduced unikgqa, a unified framework that integrates retrieval and reasoning across architectures and learning parameters. however, ir-based methods typically rely on identifying entities within a query as an initial step for retrieving the relevant subgraph. this reliance poses a notable limitation, especially when handling complex queries. sp-based methods parse questions into executable logical forms and perform queries against the kb to retrieve answers. initial works translated questions into intermediate logical forms before execution. @xcite defined a query graph that resembles subgraphs of the kb and can be directly mapped to a logical form. hu et al. (2018); lan and jiang (2020) in recent years, generative models have increasingly been used to directly generate logical forms. chen et al. (2021a) designed a retriever to fetch relevant kb items and utilized a grammar-based decoder based on lstm for generating s-expressions @xcite . @xcite employed case-based reasoning for the kbqa task, using big bird @xcite 2023 ) involved initially retrieving basic elements from the kb, such as entities, relations, subgraphs, or texts as supplementary information, before directly generating logical forms using the t5 model @xcite . however, this approach requires substantial amounts of training data and suffers from a lack of transparency in the reasoning process. large language models (llms) for kbqa leverage the potent few-shot learning capabilities inherent in llms. recent studies have shown that llms can significantly enhance reasoning over kbs. by capitalizing on the powerful few-shot learning capabilities, these methods have demonstrated marked improvements in both the accuracy and efficiency of information retrieval from kbs. in the context of few-shot scenarios, @xcite proposed that llms should prioritize evaluating the plausibility of agent plans over directly generating answers. concurrently, @xcite advocated for the generation of logical forms as initial drafts, which are subsequently refined into executable queries using the kb. from the agentenvironment perspective, @xcite developed two specialized interfaces for accessing the kb, while @xcite and @xcite designed seven tools to facilitate this process. @xcite introduced a novel approach that enables llms to iteratively employ beam search reasoning on a kb. furthermore, @xcite proposed assigning three distinct roles to the agent for addressing different kbqa subtasks. @xcite constructed an instruction dataset based on existing kbqa datasets. however, despite employing llms, these methods lack the exploration of more complex questions. ## experiment we examine interactive-kbqa across a variety of complex question types and diverse databases (dbs). ## conclusion interactive-kbqa introduces a kbqa approach which utilizes an llm as an agent for performing semantic parsing through multi-round interactions with a kb. initially, we developed a unified tool and an interaction methodology tailored to various db schemas. subsequently, by categorizing complex questions and annotating a minimal set of exemplars, we employed a few-shot learning strategy that guides the llm to incrementally generate sparql queries. moreover, we introduced a lowresource dataset that demonstrates superior performance when fine-tuned with open-source llms.