Improving LLM -based KGQA for multi-hop Question Answering with implicit reasoning in few-shot examples (2024)

Abstract: Large language models (LLMs) have shown remarkable capabilities in generating natural language texts for various tasks. However, using LLMs for question answering on knowledge graphs still remains a challenge, especially for questions requiring multi-hop reasoning. In this paper, we present a novel planned query guidance approach that improves large language model (LLM) performance in multi-hop question answering on knowledge graphs (KGQA). We do this by designing few-shot examples that implicitly demonstrate a systematic reasoning methodology to answer multi-hop questions. We evaluate our approach for two graph query languages, Cypher and SPARQL, and show that the queries generated using our strategy outperform the queries generated using a baseline LLM and typical few-shot examples by up to 24.66% and 7.7% in execution match accuracy for the MetaQA and the Spider benchmarks respectively. We also conduct an ablation study to analyze the incremental effects of the different techniques of designing few-shot examples. Our results suggest that our approach enables the LLM to effectively leverage the few-shot examples to generate queries for multi-hop KGQA.

## introduction question answering on knowledge graphs (kgqa) is a challenging task that requires understanding the natural language query, mapping it to the kg schema, and generating a graph query that can retrieve the correct answer from the kg. we focus on two graph query languages in this work, namely cypher foot_0 , a well-known graph query language developed by neo4j foot_1 , and sparql, a popular language for querying rdf foot_2 databases. in this study, we focus on the task of answering a question from a knowledge graph by generating cypher and sparql queries to query the knowledge graph. large language models (llms), such as @xcite , have shown remarkable capabilities in generating natural language texts for various tasks. recent studies have explored the capability of llms in cypher generation @xcite @xcite as well as sparql generation @xcite @xcite . however, using llms for multi-hop kgqa still remains a challenge, as they need to generate queries that can capture the multi-hop reasoning logic. furthermore, models are limited by the availability of labeled data for kgqa, which is costly and timeconsuming to obtain. therefore, it is desirable to leverage the fewshot learning ability of llms, which allows them to adapt to new tasks with only a few examples, and design effective few-shot examples that can guide the llm to generate more accurate queries for multi-hop kgqa. the utilization of few-shot learning in llms has shown promise in various domains to address the limitations of data scarcity and improve model generalization. several studies demonstrated the value of few-shot learning in various domains for improving the performance of llms @xcite @xcite . however, to the best of our knowledge, the influence of few-shot examples design in kgqa, particularly for generating cypher and sparql queries, has not been extensively studied. existing kgqa models like transfernet @xcite , which excels in multi-hop reasoning over relation graphs, unikgqa @xcite , known for its unified retrieval and reasoning framework, and nutrea @xcite , which leverages neural tree search for context-rich embeddings, outperform our proposed llm-based approach, achieving up to 100% in the hits@1 metric. however, these methods incur higher complexity and cost, as they require extensive training on specific knowledge graphs. in contrast, simple llm-based methods can achieve competitive performance with a well-designed few-shot example set, bypassing the need for exhaustive training or customization. this efficiency makes the study of few-shot example design for llm-based kgqa a crucial research area, promising swift adaptability and innovation in question-answering systems. in this paper, we propose a novel approach to improve the performance of llm-based cypher and sparql generation for multi-hop kgqa. we do this by designing few-shot examples that implicitly demonstrate a systematic reasoning methodology to answer multi-hop questions. this guides the llm to follow a similar reasoning process for new questions, without explicitly specifying the steps. we hypothesize that such few-shot examples can enhance the llm's understanding of the question, the kg schema, and the syntax of the graph query language, enabling it to generate more accurate queries for multi-hop kgqa. we evaluate our approach on two popular benchmark datasets, metaqa @xcite and spider @xcite , both of which feature natural language questions across various levels of difficulty for multi-hop querying. we start by conducting an ablation study to analyze the effects of different components of our few-shot examples design on cypher. we then show how our methodology transfers to sparql. our results demonstrate that this strategy can enhance execution match accuracy over that of conventional methods used in few-shot examples. ## methodology this work focuses on the methodology of crafting few-shot examples for improved performance of llms for the task of cypher and sparql generation for kgqa. for this task, a few-shot example is composed of a natural language question, accompanied by an expected response of a cypher or sparql query that can be run on an associated kg to answer the natural language question. we propose a method for designing cypher and sparql queries for few-shot examples that clearly demonstrates to the llm the reasoning required to answer multi-hop questions. techniques like chain-of-thought prompting @xcite use textual explanations to teach step-by-step reasoning to llms. our methodology employs a code style that implicitly shows how to take each hop stepby-step. figure 2 is an example of a cypher query written in such a style to be used as a few-shot example. contrast the query in figure 2 with a typical or conventional style foot_3 used by developers to write cypher queries in figure 3 . certain prevalent practices characterize this conventional style of crafting such graph queries. these include the utilization of succinct and non-descriptive variable names, the consolidation of all traversal hops into a single chain in a single match clause, and the immediate specification of string literals for entity matching within the variable declaration itself in the match clause (e.g., {name: "dil chahta hai"}). our proposed approach outlines practical methods for crafting few-shot examples to generate graph queries, like cypher and sparql: ## discussion our findings demonstrate the effectiveness of our proposed approach in improving the performance of llm-based kgqa systems, particularly in addressing the challenge of multi-hop reasoning. by designing few-shot examples that implicitly demonstrate systematic reasoning to guide llms in generating cypher and sparql queries, we have shown enhancements in accuracy, thereby highlighting the potential of this methodology for advancing the field of kgqa. future research directions include testing our proposed approach on knowledge graphs with increasingly complex schemas, addressing challenges such as accurate attribute selection, aggregations and function usage in these graph query languages, and assessing the efficacy of using this few-shot example design in more graph languages and code generation tasks. additionally, there is potential to develop techniques that automatically generate few-shot examples for a broad range of llms, streamlining the creation process and enhancing adaptability across various domains. help me rewrite the following query to make it more readable and understandable. make sure that: please help me rewrite the following query in the style discussed above.