HOLMES : Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLM s (2024)

Abstract: Given unstructured text, Large Language Models (LLMs) are adept at answering simple (single-hop) questions. However, as the complexity of the questions increase, the performance of LLMs degrade. We believe this is due to the overhead associated with understanding the complex question followed by filtering and aggregating unstructured information in the raw text. Recent methods try to reduce this burden by integrating structured knowledge triples into the raw text, aiming to provide a structured overview that simplifies information processing. However, this simplistic approach is query-agnostic and the extracted facts are ambiguous as they lack context. To address these drawbacks and to enable LLMs to answer complex (multi-hop) questions with ease, we propose to use a knowledge graph (KG) that is context-aware and is distilled to contain query-relevant information. The use of our compressed distilled KG as input to the LLM results in our method utilizing up to 67% fewer tokens to represent the query relevant information present in the supporting documents, compared to the state-of-the-art (SoTA) method.Our experiments show consistent improvements over the SoTA across several metrics (EM, F1, BERTScore, and Human Eval) on two popular benchmark datasets (HotpotQA and MuSiQue).

## introduction multi-hop question answering (mhqa) is a field that presents unique challenges in the realm of natural language processing. to illustrate the challenges of mhqa, consider extracting information from data arising during board meetings. while current technologies (such as llms) are proficient at addressing simple (single-hop) questions, such as "how many board meetings were held in the last twelve months?", they falter when confronted with figure 1 : multi-dimensional improvements: our method (with gpt-4 as reader llm) achieves sota results on several datasets and multiple multi-hop qa metrics. em: exact-match with the gold answer, self-aware em: confidence-aware em, bertscore @xcite : semantic similarity between predicted and gold answer; query info efficiency: efficiency of representing query-relevant information in the supporting documents -inversely proportional to the input token count for the reader llm. complex (multi-hop) questions. an example of a multi-hop question is "for the board meeting with the most divided votes in the last twelve months, what was the agenda, who voted against it, and by what margin did it pass or fail?". answering this question requires a series of interconnected steps: first, identifying the meeting with the most divided votes; next, determining the main agenda of that meeting; then, listing the members who voted against it; and finally, calculating the margin by which it was approved or rejected. each of these steps, or "hops", demands not only the retrieval of additional information but also a nuanced understanding of the context and the relationships between various entities. this complexity underscores the challenge of mhqa, where the goal is to navigate through layers of information to arrive at the answer. our work is aimed at tackling these multifaceted questions. llms have emerged as a powerful set of tools for various nlp tasks @xcite ## preliminaries hyper-relational knowledge graph -a hyperrelational kg h is an enriched form of a traditional kg. it allows for the representation of multiple relationships between entities. let a denote a collection of attribute sets whose elements represent different types of attribute sets. for example, a @xmath0, while a @xmath1, that serve as additional attributes. we then define our hyper relational kg h as: ## methodology the key idea of our method is to identify the subset of documents that contain the answer to the multihop query, and subsequently extract context-aware structured information from them (where context comes from the documents). we further perform a refinement step to retain query-relevant information. below, we begin with a brief description of the problem statement in section 4.1, followed by an explanation of our method in section 4.2. the overview of our approach can be seen in figure 3 . ## results and analysis here, we study the performance of our method by investigating several key dimensions of multi-hop question answering (especially in the era of llms): multi-hop reasoning: (a) evaluation using automatic metrics -in dataset hotpotqa methods h-eval (↑) b-score (↑) base 86.5 87.00 structqa 87.5 85.20 our method 89.0 89.05 we observe that our method achieves a 1-2 point improvement on both the metrics w.r.t. the baselines, confirming the efficacy of our method. ## acknowledgement we would like to thank shivangana rawat and other members of the ai lab at fujitsu research india for providing valuable feedback on the manuscript. we also thank the anonymous arr reviewers, the meta-reviewer, and the acl program chairs for their comments and feedback, which helped improve our draft.