DIVKNOWQA : Assessing the Reasoning Ability of LLM s via Open-Domain Question Answering over Knowledge Base and Text (2024)

Abstract: Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when solely relying on their internal knowledge, especially when answering questions that require less commonly known information. Retrievalaugmented LLMs have emerged as a potential solution to ground LLMs in external knowledge. Nonetheless, recent approaches have primarily emphasized retrieval from unstructured text corpora, owing to its seamless integration into prompts. When using structured data such as knowledge graphs, most methods simplify it into natural text, neglecting the underlying structures. Moreover, a significant gap in the current landscape is the absence of a realistic benchmark for evaluating the effectiveness of grounding LLMs on heterogeneous knowledge sources (e.g., knowledge base and text). To fill this gap, we have curated a comprehensive dataset that poses two unique challenges: (1) Two-hop multi-source questions that require retrieving information from both open-domain structured and unstructured knowledge sources; retrieving information from structured knowledge sources is a critical component in correctly answering the questions. (2) Generation of symbolic queries (e.g., SPARQL for Wikidata) is a key requirement, which adds another layer of challenge. Our dataset is created using a combination of automatic generation through predefined reasoning chains and human annotation. We also introduce a novel approach that leverages multiple retrieval tools, including text passage retrieval and symbolic language-assisted retrieval. Our model outperforms previous approaches by a significant margin, demonstrating its effectiveness in addressing the above-mentioned reasoning challenges.

## introduction llms have shown exceptional performance in multi-hop question-answering (qa) tasks over text (textqa) @xcite et @xcite @xcite @xcite , tables (tableqa) @xcite @xcite , and knowledge-bases (kbqa) @xcite @xcite , where the supporting fact is contained in a single knowledge source -structured or unstructured. however, in many real-world scenarios, a qa system may need to retrieve information from both unstructured and structured knowledge sources; failing to do so results in insufficient information to address user queries. while existing qa benchmarks provide diverse perspectives for evaluating models ( dataset kb text table openr human ei sgt hybridqa @xcite ✗ ✓ ✓ ✗ ✓ ✓ ✗ ott-qa @xcite ✗ ✓ ✓ ✓ ✓ ✓ ✗ nq-tables @xcite ✗ ✓ ✓ ✗ ✗ ✗ ✗ tat-qa @xcite ✗ ✓ ✓ ✗ ✓ ✓ ✗ multimodelqa @xcite ✗ ✓ ✓ ✗ ✓ ✗ ✗ manymodelqa @xcite ✗ ✓ ✓ ✗ ✓ ✗ ✗ finqa @xcite ✗ ✓ ✓ ✗ ✓ ✓ ✗ hetpqa @xcite ✗ ✓ ✓ ✗ ✓ ✗ ✗ compmix @xcite ✗ ✓ ✓ ✓ ✓ ✗ ✗ wikimovies-10k @xcite ) ✓ ✓ ✗ ✓ ✓ ✗ ✗ metaqa (zhang et al., 2018) ✓ ✓ ✗ ✓ ✓ ✗ ✗ divknowqa (ours) ✓ ✓ ✗ ✓ ✓ ✓ ✓ aspects: (1) closed-book qa: closed-book questions do not accurately reflect the real-world setting where individuals generally have access to diverse knowledge sources on the internet; (2) automatically generated data: the lack of human verification results in erroneous data; in this work, we propose divknowqa, a novel fact-centric multi-hop qa benchmark that requires models to utilize heterogeneous knowledge sources equitably in order to answer a question. we perform the first study to assess the reasoning ability of llms, via jointly exploiting open-domain qa over heterogeneous knowledge sources. in particular, we have chosen knowledge base (kb) as our primary case study for the structured source, and we have created a dataset comprising 940 human-annotated examples. additionally, each entry in our dataset includes a corresponding symbolic sparql query to facilitate the retrieval of information from the kb. to generate the questions, we construct a question collection pipeline comprising three key steps: text-based qa sampling, kb question generation, and question composition, all while minimizing the need for human annotation efforts. to set up a baseline, in addition to benchmarking on standard and tool-augmented llms, we propose a diverse retrieval tool augmented llm (detllm) to address the challenges posed by divknowqa. detllm decomposes a multihop question into multiple single-hop questions, and adopts two novel strategies: (1) symbolic query generation to retrieve supportive text from a kb by transforming a single-hop natural question into a sparql query, and (2) retrieval tool design, which includes a textual retriever and a symbolic query generation tool to recall relevant evidence from heterogeneous knowledge sources. our method shows improvements of up to 4.2% when compared to existing methods. ## detllm: diverse retrieval tool augmented llm we now introduce our diverse retrieval tool augmented llm (detllm) and show its promising capability on the proposed divknowqa benchmark by unifying the retrieval ability from the structured and unstructured knowledge sources. to tackle a complex question, we follow the chain-of-thought (cot) framework @xcite to decompose a complex question into singlehop questions where each single-hop question is knowledge-intensive, requiring supportive fact retrieval from a knowledge source. we design a retrieval tool capable of retrieving from heterogeneous knowledge sources. for unstructured text knowledge, a dense passage retriever @xcite is employed to retrieve relevant passages. for structured knowledge, we consider two modalities of structured knowledge to maximize the relevant information coverage. first, we transform the structured data into text passages by linearizing the relation triplets into passages in which case a sparse text retriever can be used to detect similar sources. second, we propose a symbolic query generation module to map a natural language query to a structured query (e.g., sparql) to directly query against the kb (e.g., wikidata). the benefits are twofold: (1) pinpointing precise knowledge, and (2) leveraging the compositionality of the query language and reducing the mere reliance on the language model's reasoning responsibility. figure 3 shows the detllm flow for querying an llm. ## conclusion we introduce the divknowqa, designed to evaluate the proficiency of question-answering systems, especially those enhanced by retrieval tools, in addressing knowledge-intensive questions with a strong emphasis on multi-hop multi-source retrieval. this dataset is constructed through automated data generation and subsequent human verification, minimizing manual effort. our evaluation encompasses both standard llms and llms augmented with retrieval tools. notably, we identify that this task presents a new challenge for state-of-the-art models due to the demand for structured knowledge retrieval and the inherent lack of prior knowledge in this context. to tackle this challenge, we propose the detllm, which incorporates diverse retrieval tools including innovative symbolic query generation for retrieving information from the structured knowledge source. in the future, we are keen on enhancing llms' capabilities in understanding and generating symbolic language, as well as exploring methods to improve performance on knowledge-intensive and complex question-answering tasks.