Augmenting Reasoning Capabilities of LLM s with Graph Structures in Knowledge Base Question Answering (2024)

Abstract: Recently, significant progress has been made in employing Large Language Models (LLMs) for semantic parsing to address Knowledge Base Question Answering (KBQA) tasks. Previous work utilize LLMs to generate query statements on Knowledge Bases (KBs) for retrieving answers. However, LLMs often generate incorrect query statements due to the lack of relevant knowledge in the previous methods. To address this, we propose a framework called Augmenting Reasoning Capabilities of LLMs with Graph Structures in Knowledge Base Question Answering (ARG-KBQA), which retrieves question-related graph structures to improve the performance of LLMs. Unlike other methods that directly retrieve relations or triples from KBs, we introduce an unsupervised two-stage ranker to perform multi-hop beam search on KBs, which could provide LLMs with more relevant information to the questions. Experimental results demonstrate that ARG-KBQA sets a new state-of-the-art on GrailQA and WebQSP under the few-shot setting. Additionally, ARG-KBQA significantly outperforms previous few-shot methods on questions with unseen query statement in the training data.

## introduction knowledge bases (kbs), such as freebase @xcite , wikidata @xcite , and dbpedia @xcite , storing a tremendous amount of world knowledge in the way of triples, is crucial in many domains as it can provide highly accurate information. knowledge base question answering (kbqa) @xcite , which retrieves answers from knowledge bases in response to natural language questions, has been widely studied in recent years. as the scale of kb continues to (b) our method. figure 1 : the difference between our method and previous icl methods. recently, large language models (llms) have demonstrated the ability to complete complex reasoning tasks, like generating executable code, with only minimal labeled data. this is due to their in-context learning (icl) capabilities @xcite . following the icl paradigm, @xcite and @xcite use a few (question, logical form) example pairs to guide llms in generating executable logical forms for retrieving answers from kbs. since llms do not have direct access to kbs, these examples can provide helpful information for generating correct logical forms. nonetheless, when the logical form involved in the test question has not appeared in the training set, and the examples provided in a fewshot scenario all come from the training set, they cannot offer directly useful information for the current problem. for example, as shown in figure 1a , when the llms lack information relevant to the question, it may generate a logical form that is incorrect. therefore, it is crucial to equip llms with comprehensive and relevant information to enhance their performance. some works @xcite retrieve relations or triples from kbs to provide more useful information. however, there are many questions requiring multi-hop reasoning on the knowledge base (kb) to get the answers. therefore, merely providing relations or triples is still insufficient to solve the problem, as shown in figure 1a . to address this challenge, we propose an innovative method called augmenting reasoning capabilities of llms with graph structures in knowledge base question answering (arg-kbqa). our method aims to provide llms with information more helpful for generating logical forms, specifically in the zero-shot generalization sceario mentioned above. unlike previous works which retrieve relations or triples from kbs as supply information, we leverage the graph structures within the kb to enhance the performance of llms as shown in figure 1b . to efficiently retrieve questionrelated graph structures from the kb, we design an unsupervised two-stage ranker that performs multi-hop beam search on the kb. in this process, we perform a two-stage ranking at each hop using a pre-trained language model without additional training, enhancing the efficiency and effectiveness of retrieving related graph structures (paths) from the kb. furthermore, to facilitate the understanding of llms, we convert the extracted graph structures (paths) into logical forms that match the target format. we conduct extensive experiments on two datasets, grailqa and webqsp. arg-kbqa sets a new state-of-the-art (sota) in the few-shot setting, demonstrating that retrieved graph structures provide llms with sufficient information. when it comes to the questions unseen in training set of grailqa, arg-kbqa significantly surpasses the existing sota, with improvements of 8.1 in em and 7.4 in f1. these results suggest that related graph structures can enhance llms' reasoning ca-pabilities, especially when the kb schema in the correct logical form is absent from the training set. our contributions can be summarized as follows: ## preliminaries in this section, we introduce two crucial concepts: knowledge base (kb) and logical form. ## methodology overall, our proposed arg-kbqa model, as illustrated in figure 2 , aims to generate a logical form for a given question to access the kb and retrieve the answers. the arg-kbqa model consists of three modules: multi-hop path enumeration, target logical form generation, and alignment and answer extraction. next, we will introduce these three modules separately. ## experiments in this section, we demonstrate the experimental validation of arg-kbqa, including dataset descriptions, implementation details, results, and a detailed analysis. ## conclusion in conclusion, our proposed framework, arg-kbqa, significantly enhances the reasoning capabilities of large language models (llms) for knowledge base question answering (kbqa). by leveraging graph structures within knowledge bases (kbs) through an innovative unsupervised two-stage ranker, arg-kbqa effectively retrieves relevant information for accurate logical form generation. experimental results on grailqa and we-bqsp datasets highlight the framework's effectiveness, setting new state-of-the-art benchmarks in the few-shot setting and excelling in zero-shot generalization scenarios. these findings underscore the importance of integrating graph structures to augment llm reasoning capabilities, especially for unseen query statements in training data.