S ym KGQA : Few-Shot Knowledge Graph Question Answering via Symbolic Program Generation and Execution (2024)

Abstract: Semantic Parsing of natural language questions into their executable logical form (LF) has shown state-of-the-art (SOTA) performance for Knowledge Graph Question Answering (KGQA). However, these methods are not applicable for real-world applications, due to lack of KG-specific training data. Recent advances in the capabilities of Large Language Models (LLMs) has led towards generating low-level LFs such as SPARQL and S-Expression in a few-shot setting. Unfortunately, these methods: (1) are limited to the knowledge of underlying LLM about the LF, (2) performs inferior for the harder complex benchmarks such as KQA Pro, (3) suffers while grounding the generated LF to a specific Knowledge Graph. Recently, a new LF called KoPL has been introduced that explicitly models complex reasoning process step-by-step in a symbolic manner and has shown SOTA on KQA Pro in fully-supervised setting. Inspired by this, we propose SymKGQA framework that generates step-by-step Symbolic LF i.e., KoPL in a few-shot in-context learning setting using LLM. Our framework is not dependent on pre-trained information of LLM about KoPL. We further build a Retrieval-Augmented Generation based Question-Aware Contextual KoPL (QUACK) resolver to ground the generated LF. Our experiments with different LLMs and few-shot settings demonstrate that SymKGQA outperforms all other few-shot and even many of the fully-supervised KGQA approaches.

## introduction knowledge graph question answering (kgqa) has received a lot of research interest in recent years @xcite @xcite @xcite . it aims at answering a natural language question using a structured knowledge graph (kg) by producing a logical form (lf) such as sparql, s-expression, λ-dcs, etc. that is executed on the kg to retrieve the answer(s). when such techniques are used in applications such as intelligent assists, the annotated training data is generally not available at cold-start, thus, making it difficult to use these techniques without significant investment in building a large, generalizable (to the application) dataset to train the models on. with recent advancements in large language models (llms), the research is moving towards leveraging the reasoning capability of llms to obtain the lf for a given natural language question (nlq) in both fine-tuning @xcite @xcite and few-shot settings @xcite . for few-shot setting, these works have used in-context learning (icl) paradigm with various state-of-the-art llms such as @xcite variants to generate lf. however, these methods suffer from the following limitations: (1) their lf generative capabilities are limited to the pre-trained knowledge of the underlying llm about the grammar and semantics of the lf (sparql or s-expression). (2) this also results in their inferior performance for harder complex kgqa benchmarks such as kqa pro @xcite . these benchmarks require explicit modeling of kg complexities such as concepts, attributes, qualifiers, etc. to perform joint compositional and numerical reasoning that may not be seen by the llm. (3) their kg grounding formulation is independent of the context of the question, which limits their performance. due to the recent surge in the complexity of questions that users pose to question answering (qa) systems, complex kgqa has particularly gained attention @xcite . complex questions require joint compositional and numerical reasoning on kg to answer them. some of the works have demonstrated that step-by-step chainof-thought (cot) decomposition of the reasoning process is one of the key factors in enhancing the reasoning capability of llms for qa tasks @xcite @xcite . however, a recent shift towards decomposing the problem with symbolic formalization i.e., chainof-symbol (cos) @xcite has emerged as another key factor to observe further enhancements in reasoning @xcite . additionally, some of the works @xcite @xcite have shown that using the same underlying model with lfs having high to low formalization i.e., dissimilarity to natural language has shown an inverse trend in the performance for kgqa. recently, @xcite , a symbolic lf has been introduced that defines various functions (operations) on kg catering to various complexities. it reflects the cos reasoning process as compared to other lfs that have to be used as a whole query @xcite . its modular functions and their inputs make it suitable for complex compositional and numerical reasoning while being interpretable. figure 1 shows an example of a complex question and its corresponding lfs with high (top) to low (bottom) level of formalization. as shown, kopl is most decomposable, interpretable lf while having symbolic formalization to solve complex questions. another challenge that most of the existing llmbased techniques face while generating lf for kgqa is -that llm picks kg items (entities, relations, etc.) based on its inherent knowledge or picks an appropriate phrase from the input question. these items have to be grounded/resolved for a kg before the generated lf can be executed. to illustrate this, consider the generated kopl in figure 1 . the functional input of step 3 is occupy that does not exist in the kg and the closest relations in the kg are {mass, area, etc.}. the existing methods pick the closest relation based on the similarity between occupy and the candidates {mass (0.91 2 ), area (0.84)}. hence, they would pick mass due to its higher similarity than area. however, if we consider an additional context of the input question while choosing the closest relation, area should be chosen (desired relation). existing methods do 2 bert similarity score not take into consideration this context of the input question and hence, suffer in grounding the lf. addressing the above-discussed challenges, the contributions of our work are as follows: popular kgqa datasets such as metaqa @xcite , webqsp @xcite , etc. has sparse coverage of complex questions. on the other hand, a recently introduced kqa pro dataset (cao et al., 2022a) contains much harder complex qa pairs that require compositional and numerical reasoning as compared to other datasets such as grailqa @xcite . thus, this work utilizes kqa pro as the core benchmark dataset for experiments. our extensive experiments demonstrate that symkgqa outperforms all other llm-based fewshot and even most of the sota fully-supervised kgqa approaches by a significant margin. to the best of our knowledge, symkgqa is the first fewshot kgqa framework that uses a cos approach (using kopls) with rag-based quack resolver. ## proposed framework: symkgqa we first briefly define the problem statement and then introduce the details of symkgqa framework shown in figure 2 , which consists of two stages: ## results and analysis we discuss the results of symkgqa on each dataset in detail. qualitative analysis examples are provided in appendix a.5. ## conclusion we propose symkgqa, that generates and executes symbolic programs (kopl) in a few-shot setting. to the best of our knowledge, it is the first one to generate kopl lf using llms for complex kgqa. we provide a method for llm prompt generation with manual and dynamic demonstration selection for few-shot icl. we further propose a rag based question-aware contextual kopl (quack) resolver that grounds the generated kopl for a kg. our extensive experiments shows that symkgqa surpass all few-shot and most of the fully supervised sota baselines, setting a new benchmark for various complex kgqa datasets. our approach being independent of underlying llm knowledge, makes it applicable for any other symbolic lf while being interpretable.