Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answering (2024)

Abstract: Knowledge graph question answering (KGQA) involves answering natural language questions by leveraging structured information stored in a knowledge graph. Typically, KGQA initially retrieve a targeted subgraph from a large-scale knowledge graph, which serves as the basis for reasoning models to address queries. However, the retrieved subgraph inevitably brings distraction information for knowledge utilization, impeding the model’s ability to perform accurate reasoning. To address this issue, we propose a Question-guided Knowledge Graph Re-scoring method (Q-KGR) to eliminate noisy pathways for the input question, thereby focusing specifically on pertinent factual knowledge.Moreover, we introduce Knowformer, a parameter-efficient method for injecting the re-scored knowledge graph into large language models to enhance their ability to perform factual reasoning.Extensive experiments on multiple KGQA benchmarks demonstrate the superiority of our method over existing systems.

## introduction knowledge graph question answering (kgqa) aims to provide factual answers to natural language questions by leveraging information from knowledge graphs @xcite @xcite . sitting at the intersection between graphs and texts, this task can further facilitate the applicability of knowledge graphs in more downstream tasks, such as recommendation systems @xcite , information extraction @xcite , time series forecasting @xcite @xcite . recent work on kgqa attempts to synergize large language models (llms) with knowledge graphs (kgs) by utilizing retrieved knowledge from kgs to assist llms in conducting factual reasoning @xcite @xcite . a prevailing approach is to heuristically retrieve a subgraph from a large-scale kg using topic entities (kg entities mentioned in the given question and candidate answers) and their few-hop neighbors, and then use this subgraph as the foundation for llms to perform reasoning. despite the promising performance, this approach heavily rely on the retrieved graph which inevitably includes distracting information. this might leads the llm to focus on irrelevant knowledge, thereby hindering its ability to conduct accurate reasoning. figure 1 depicts an example from the riddlesence dataset @xcite . in the retrieved subgraph, the question entity bank connects a distracting answer entity repository with two noisy pathways: (i) @xmath0. directly incorporating knowledge from these paths into the llm would significantly distort its reasoning and judgment. however, these noisy paths can be easily identified and excluded by considering and analysing the input question. for example, we can exclude the noisy path (i) by considering that the question is irrelevant to "river". in addition, the noisy path (ii) can be excluded by taking into account the semantic meaning more deeply such as a specific restriction phrase, "where children keep their assets". building on the insights from above analyses, this paper introduces a question-guided knowledge graph re-scoring method (q-kgr) to eliminate the negative effect of noisy pathways for specific questions, thereby focusing attention on precise factual knowledge. specifically, we develop an edge re-scorer to assign a relevance score to each edge by computing the semantic similarity between the specific question and each edge triplet to obtain a re-scored knowledge graph. and then we extend the graph attention network to incorporate edge scores during feature aggregation. in addition, we design knowformer, a customized transformer where a knowledge projector is utilized to align and inject the information of knowledge graphs into the standard transformer architecture, thereby enhancing its reasoning capabilities. furthermore, knowformer can naturally adapt to lora @xcite , and the combined use can effectively achieve task adaptation and knowledge injection. we conduct extensive experiments on four kgqa benchmarks, and the experimental results demonstrate that our method successfully eliminate noisy pathways and inject knowledge into llm to perform better factual reasoning. our main contributions are summarized as follows: ## methodology the overall framework of the proposed method is illustrated in figure 2 . firstly, we employ an edge re-scorer to assign a relevance score to each edge based on its relevance to the question ( §4.1). additionally, we modify gat to encode the rescored subgraph, obtaining structured knowledge representations at each layer ( §4.2). furthermore, we develop a layer-wised knowledge injection method to inject knowledge from last m layers in gnns into corresponding knowformer layers ( §4.3). finally, we process the question text sequentially through n layers of the transformer and m layers of knowformer to predict the probability distribution of the candidate answers. ## experiments in this section, we conduct extensive experiments on four kgqa benchmarks including openbookqa (obqa) @xcite , ai2 reasoning challenge (arc) @xcite , riddlesense (riddle) @xcite and physical interaction question answering (piqa) @xcite to compare the performances of different methods. we use conceptnet @xcite , a general-domain knowledge graph as our structured knowledge source g. a detailed explanation of the datasets and the knowledge graph is provided in the appendix a. ## analyses and discussions we conduct comprehensive quantitative and qualitative analysis from various perspectives to validate the effectiveness of knowledge graph rescoring and parameterized knowledge injection. ## conclusion in this paper, we propose a question-guided knowledge graph re-scoring method to eliminate noisy pathways for specific problem, directing attention towards precise factual knowledge. figure 6: as the depth of the layers increases, the injected knowledge gradually integrates into the distribution of the ffn parameters. obqa riddle figure 7: performance by adjusting the layers of knowledge injection, as well as the number of layers in the gat encoder on the riddle and obqa datasets. furthermore, we design knowformer, a customized transformer, through which we align and inject structured knowledge into the parameter space of llm. extensive experiments on multiple kgqa datasets across different llm architectures, sizes and setups demonstrate the superiority of our method. and in-depth analysis experiments indictates that our model is particularly effective in handling complex questions characterized with significant noisy pathways in external knowledge. we also illustrate that our approach achieves effective knowledge alignment and injection for llm.