U ni K - QA : Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering (2022)

Abstract: We study open-domain question answering with structured, unstructured and semi-structured knowledge sources, including text, tables, lists and knowledge bases. Departing from prior work, we propose a unifying approach that homogenizes all sources by reducing them to text and applies the retriever-reader model which has so far been limited to text sources only. Our approach greatly improves the results on knowledge-base QA tasks by 11 points, compared to latest graph-based methods. More importantly, we demonstrate that our unified knowledge (UniK-QA) model is a simple and yet effective way to combine heterogeneous sources of knowledge, advancing the state-of-the-art results on two popular question answering benchmarks, NaturalQuestions and WebQuestions, by 3.5 and 2.6 points, respectively. The code of UniK-QA is available at: https://github.com/facebookresearch/UniK-QA .

## introduction answering factual questions has long been an inspirational challenge to information retrieval and artificial intelligence researchers @xcite . in its most general form, users can ask about any topic and the answer may be found in any information source. defined as such, the challenge of open domain question answering is extremely broad and complex. though there have been successful undertakings which embrace this complexity (notably @xcite , most recent works make simplifying assumptions as to the source of answers, which fall largely in two categories: structured data and unstructured text. a long line of research aims to answer user questions using a structured knowledge base (kb) @xcite , known as kbqa. typically, a kb can be viewed as a knowledge graph consisting of entities, properties, and a predefined set of relations between them. a question can be answered, provided that it can be expressed within the language of relations and objects present in the knowledge graph. with a high-quality, carefully curated kb, answers can be extracted with fairly high precision. kbqa, however, struggles with low answer coverage due to the cost of curating an extensive kb, as well as the fact that many questions simply cannot be answered using a kb if the answers are not entities. a second line of work targets a large collection of unstructured text (such as wikipedia) @xcite as the source of answers. thanks to the latest advances in machine reading comprehension and text retrieval, substantial progress has been made for open-domain question answering from text (textqa) in just the past couple years @xcite @xcite @xcite . on the other hand, semi-structured tables and structured kbs can be valuable knowledge sources, yet textqa methods are restricted in taking only unstructured text as input, missing the opportunity of using these complementary sources of information to answer more questions. when it comes to answering questions using both structured and unstructured information, a straightforward solution is combining specialized textqa and kbqa systems. the input question is sent to multiple sub-systems, and one of them is selected to output the final answer. while this approach may take advantage of the state-of-the-art models designed for different information sources, the whole end-to-end system becomes fairly complex. it is also difficult to handle questions that require reasoning with information from multiple sources. having a more integrated system design that covers heterogeneous information sources has proven to be difficult. one main reason is that techniques used for kbqa and textqa are drastically different. the former exploits the graph structure and/or semantic parsing to convert the question into a structured query, while textqa has mostly settled on the retriever-reader architecture powered by pre-trained transformers. recent work on multisource qa has tried to incorporate free text into graph nodes @xcite to make texts amenable to kbqa methods, but the performance remains unconvincing. in this work, we propose a novel unified knowledge representation (unik-qa) approach for open-domain question answering with heterogeneous information sources. instead of having multiple specialized sub-systems or incorporating text into knowledge graphs, we flatten the structured data and apply textqa methods. our main motivation for doing so is to make the powerful machinary of pre-trained transformers available for structured qa. in addition, this approach opens the door to a simple and unified architecture. we can easily support semi-structured sources such as lists and tables, as well as fully structured knowledge bases. moreover, there is no need to specially handle the schema or ontology that defines the structure of the kb, making it straightforward to support multiple kbs. our unik-qa model incorporates some 27 million passages composed of text and lists, 455,907 wikipedia tables, and 3 billion relations from two knowledge bases (freebase and wikidata) in a single, unified open-domain qa model. we first validate our approach by modeling kbqa as a pure textqa task. we represent all relations in the kb with their textual surface form, and train a retriever-reader model on them as if they were text documents. this simple approach works incredibly well, improving the exact match score on the webqsp dataset by 11% over previous state of the art. this result further justifies our choice of unifying multi-source qa under the textqa framework as it can improve kbqa performance per se. for our multi-source qa experiments, we consider lists, tables, and knowledge bases as sources of structured information, and convert each of them to text using simple heuristics. we model various combinations of structured sources with text, and evaluate on four popular open-domain qa datasets, ranging from entity-heavy kbqa benchmarks to those targeting free-form text sources. our results indicate that our multi-source unik-qa approach, unlike existing efforts on combining kbqa and textqa, consistently improves over strong textqa baselines in all cases. we obtain new state-of-theart results for two datasets, advancing the published art on naturalquestions by 3.5 points and on we-bquestions by 2.6 points. in addition, we consider the realistic setting in which the source of questions is not known a priori, as would be the case for a practical system. we train a single multi-dataset model on a combined dataset from several benchmarks, and show that it outperforms all single-source baselines across this diverse set of questions. ## kbqa as textqa: a motivating experiment in this section, we present a motivating experiment showing that our unik-qa approach not only provides a natural pathway to multi-source open-domain qa, but also improves kbqa per se. in particular, we evaluate our approach on a widely-used kbqa dataset, webqsp @xcite , in the single-source setting. we use freebase as the knowledge source, and re-use pre-computed stagg entity linking results and 2-hop neighborhoods as provided by @xcite for fair comparisons. we convert kb relations in the 2-hop neighborhood into text, retrieve the most relevant ones using dpr to form 100 context passages, and feed them into the t5 fid reader as described in section 3.2. we use the original dpr checkpoint from @xcite for retrieval, and train fid using the training questions in webqsp and the dpr-retrieved contexts with default hyperparameters (see §7). the results are shown in we see that our kbqa method outperforms previous state-of-the-art methods by a wide margin, improving exact match accuracy to 79.1%. since we adopt the exact same kb setup and pre-processing procedure from previous work, this improvement can be attributed purely to our unik-qa model. we take this result as strong evidence for our claim that powerful textqa methods generalize well to structured data, and offer a natural new framework for unifying structured and unstructured information sources. ## multi-source qa experiments we now present our main experiments on unified multi-source question answering. ## analysis having demonstrated that combining information sources does improve answer accuracy, we now provide more analysis on how this is achieved by inspecting both retriever and reader closely. retriever one natural assumption is that adding more data increases the coverage of relevant contexts that can be used to answer the input questions, thereby improving the end-to-end performance. we verify this by examining the retrieval results of different models using the nq development set, where a context is considered relevant if it contains the correct answer string. when more knowledge sources are added, our system is able to improve retrieval recall ( reader although including additional information sources improves the chance of retrieving relevant contexts, it is not guaranteed that the reader can leverage those contexts and output the correct answers. for instance, reader model training may benefit from diverse sources of contexts, and the end-to-end improvement of answer accuracy may simply be attributed to a reader model that per-model r@20 r@100 forms better on contexts from regular text. due to the nature of the fid generative reader, however, it is non-trivial to ascertain which input context(s) contribute the answer. as a proxy, we look at the correlation between the source of positive contexts (those which contain a correct answer string) feeding into the reader model and the performance change in the outcome. suppose we are comparing two reader models m u and m t , where m u uses additional sources of information compared to m t (e.g., m t uses text only and m u uses text and kb). let q be all the questions in our development set, q u ⊆ q and q t ⊆ q the subsets of questions answered correctly by m u and m t , respectively. the improvement set @xmath0. examining the source of the positive contexts for the questions in q can help shed some light on how m u performs better. for example, if more positive contexts are from kb rather than text, then the improvement is more likely due to additional information present at inference time. figure 3 plots the percentages of positive contexts originating from the additional sources for the questions in the full development set (q) vs those in the improvement set (q ) in two cases. the first one compares a baseline text-only model to a model with lists and tables added on nq, and the second compares a text+tables model with text+tables+kb on webq. in both cases, answers retrieved from the additional source correlate with a better outcome. to examine the effects of other indirect factors, such as the change of overall model quality due to the inclusion of varied sources or more training samples from the tables, we evaluate the text + tables model with text-only input. we find that this achieves similar performance (48.7 em) on the nq test set compared to a text-only model on the same input, suggesting that these other factors are not a major contributor and that the improved performance is primarily due to the added knowledge from structured sources. ## implementation details the code, data, and trained model checkpoints of unik-qa are available at: https://github. com/facebookresearch/unik-qa . ## discussion we demonstrated a powerful new approach, unik-qa, for unifying structured and unstructured information sources for open-domain question answering. we adopt the simple and general retriever-reader framework and show not only that it works for structured sources, but improves over traditional kbqa approaches by a wide margin. by combining sources in this way, we achieved new state-of-the-art results for two popular opendomain qa benchmarks. however, our model also has several shortcomings in its current form. as a result of flattening all sources into text, we lose some desirable features of structured knowledge bases: the ability to return all answers corresponding to a query, and the ability to infer multi-hop paths to answer more complex questions. in this work we have side-stepped the first issue by focusing on the exact match metric (equivalent to hits@1), which is standard in the open-domain qa literature, but largely ignores multiple answers. we were also able to ignore the second issue, since the datasets we evaluated on, while standard, are composed mostly of simple, natural user questions which can be answered from a single piece of information. we do believe these are important details and they can be addressed within the framework described here. for instance, outgoing edges of an entity with the same relation can easily be merged, thus encoding all answer entities into a single text representation. it is also possible to simply generate multiple answer candidates from the reader's decoder. for multi-hop question answering, there is recent work @xcite successfully extending dense retrieval to the multi-hop setting @xcite , which could naturally be applied within our framework. it remains to be seen how these approaches would compare to more traditional structured methods.