Knowledge Graph-augmented Language Models for Complex Question Answering (2023)

Abstract: Large language models have shown impressive abilities to reason over input text, however, they are prone to hallucinations. On the other hand, end-to-end knowledge graph question answering (KGQA) models output responses grounded in facts, but they still struggle with complex reasoning, such as comparison or ordinal questions. In this paper, we propose a new method for complex question answering where we combine a knowledge graph retriever based on an end-to-end KGQA model with a language model that reasons over the retrieved facts to return an answer. We observe that augmenting language model prompts with retrieved KG facts improves performance over using a language model alone by an average of 83%. In particular, we see improvements on complex questions requiring count, intersection, or multi-hop reasoning operations.

## introduction large language models (lms) have shown great promise in a variety of nlp tasks, including question-answering (qa) @xcite @xcite . as language models scale, they achieve impressive results on standard qa benchmarks, such as squad @xcite , and can answer questions either few-shot or zero-shot using only knowledge stored within the model parameters @xcite . language models have also been shown to solve complex reasoning tasks by outputting step-by-step instructions from question to answer @xcite . despite these successes, language models are still prone to hallucinations and can return answers that are incorrect, out-of-date, and not grounded in verified knowledge sources, making them an unsafe choice for a factual question answering service. additionally, step-by-step reasoning can be computationally expensive as it requires multiple calls to a language model. alternatively, knowledge graph-based question answering (kgqa) models @xcite are trained to traverse knowledge graph (kg) facts to return answers to questions. these models are faithful and grounded to facts stored in a kg. however kgqa models are often limited in the types of reasoning that they can perform. most end-to-end kgqa models can perform relation following for single or multiple hops @xcite , and some models have been trained for set intersection @xcite , union, or difference @xcite , however expanding to general reasoning capabilities remains an open challenge. kgqa models are also restricted to using facts stored in a knowledge graph and can not leverage common world knowledge. in this paper, we propose a novel method for complex question answering using a kgqa model retriever with a language model reasoner. our approach harnesses both the ability to traverse over verified facts with a kgqa model, and the ability to reason over text with an lm. for our kgqa retriever, we train an end-toend kgqa model based on reifkb @xcite and the rigel family of models @xcite . we use this model to return a weighted set of facts from the knowledge graph that could be useful for answering a given question. we then prompt an lm with the question and the top-k facts retrieved, in a zero-shot setting, and the language model returns a natural language answer. in our experiments over four qa datasets, we show that our approach can outperform using an lm alone by an average of 83%. ## method we propose a new method for question answering using a kgqa model to retrieve facts from a knowledge graph, and a language model to reason over the question and facts to return an answer. ## results we evaluate our method on four complex questionanswering datasets using seven language models. we compare against two baselines. the results are reported in these results show that in almost all cases, a language model using facts retrieved from our rigel model outperforms no knowledge and random facts. for smaller models such as flan-t5, using rigel facts improves performance by up to 300%. larger models, such as alexatm, start with higher baselines using no knowledge, but still see an average of 47% improvement across datasets. exceptions are opt on mintaka and flan-t5 xxl on lc-quad, where random facts outperform rigel. we observe that in many of the questions where augmenting with random facts performs better than rigel facts, neither provide useful information. interestingly, however, random facts still encourage the lm to output the correct answer. the no knowledge results show that models can answer questions without additional facts. larger models with no facts can even outperform smaller models with rigel facts, for example, alexatm vs. flan-t5 on mintaka (28.48 vs. 20.58). nevertheless, it is promising to see smaller models become more competitive with the help of a retriever. finally, in ## conclusion in this paper, we show how facts from a kgqa based retriever can be combined with a language model to help answer complex questions. our results show improvements over calling a language model directly over four datasets, and in particular on complexity types such as multi-hop and count questions. we present our method as a promising way to leverage a knowledge graph, which con-tains verified and up-to-date facts, with a single call to a language model. in future work, we plan to improve performance across more complexity types and aim to explore ways to update the training of our kgqa retriever with feedback from the language model.