Open Domain Question Answering with A Unified Knowledge Interface (2022)

Abstract: The retriever-reader framework is popular for open-domain question answering (ODQA) due to its ability to use explicit knowledge. Although prior work has sought to increase the knowledge coverage by incorporating structured knowledge beyond text, accessing heterogeneous knowledge sources through a unified interface remains an open question. While data-to-text generation has the potential to serve as a universal interface for data and text, its feasibility for downstream tasks remains largely unknown. In this work, we bridge this gap and use the data-to-text method as a means for encoding structured knowledge for open-domain question answering. Specifically, we propose a verbalizer-retriever-reader framework for ODQA over data and text where verbalized tables from Wikipedia and graphs from Wikidata are used as augmented knowledge sources. We show that our Unified Data and Text QA, UDT-QA, can effectively benefit from the expanded knowledge index, leading to large gains over text-only baselines. Notably, our approach sets the single-model state-of-the-art on Natural Questions. Furthermore, our analyses indicate that verbalized knowledge is preferred for answer reasoning for both adapted and hot-swap settings.

## introduction pretrained language models @xcite have been shown to store certain knowledge (linguistic or factual) implicitly in parameters @xcite @xcite , partially explaining the superior generalization abilities over downstream tasks. however, besides the well-known hallucination issue, the implicit knowledge learned through language modeling objective over text struggles at reflecting up-to-date knowledge from text and structured data for answering open-domain questions. ## verbalizer: data-to-text generation here, we formally describe the data-to-text model developed in this paper, including the input format ( §3.1) and the adaptation for odqa ( §3.2). ## experiment setup in this section, we describe the data used for experiments and sources of structured knowledge. ## experiments: qa over data and text here we present our main experiments on odqa over data and text. for regular wikipedia text, we use the same index containing 21m passages as in @xcite . to augment text, two settings are considered, i.e. the single data setting and the hybrid data setting. in the single data setting for nq, we augment the text index with tables from the all-tables set ( §4.2). for comparison, we also experiment with the raw representations using a simple linearization of tables similar to @xcite . in single data setting for webq, we consider combining text with kb graphs from wd-graphs in the single data setting. different from @xcite where a separate entity-linking based retriever is used for kb, we use a single model over the text index with either linearization of raw kb graphs or our verbalized kb graphs. hence, in our case, both text and data (tables and kb graphs) can be handled by a unified retriever-reader pipeline. in the hybrid data setting for both nq and webq, we use text, all-tables and wd-graphs for retrieval. the statistics of our document index are shown in we create additional retriever training data from nq-table-q and webq-kb-q in a similar fashion as in the text-only setting, so that dpr can better handle additional knowledge. following @xcite , we also use the iterative training setup for retriever training. more training details can be found in appendix b. to evaluate the effectiveness of our udt-qa for odqa, we first include recent state-of-the-art odqa models using text as the only knowledge source, dpr and unitedqa. we also compare our udt-qa with recent models using additional structured knowledge, kealm and unitk-qa. following the literature, we report the exact match (em) score for evaluation. the results are in as we can see, models with additional structured knowledge achieve better performance than text-only models. this indicates that both kb graphs and tables contain complementary knowledge which is either absent in text or harder to be reasoned over. for nq, although we consider a significantly larger structured knowledge source which is likely to be more challenging, all our models substantially outperform unitk-qa. as for webq, our model achieves competitive performance, although worse than unitk-qa. we attribute this gap to two possible reasons. first, unitk-qa uses a separate entity-linking based retriever for kbs which might lead to higher retrieval recall. second, since webq is fully based on free-base, using wikidata only in our models likely suffers from mismatch @xcite . nevertheless, our verbalizer-based models achieve better performances than the corresponding raw format models on both datasets, indicating that the proposed verbalizer is highly effective for tables and kb graphs. ## analysis in this section, we present analyses over the impact of document index size, the use of additional structured knowledge in a hot-swap setting, comparison to a recent kb-only data-to-text approach in an end-to-end fashion, and manual exam of the verbalized/raw tables for their impact on odqa. how does the size of document index affect retriever and reader performance? more knowledge is likely to have better coverage of relevant information. on the other hand, larger and noisier index also increases the reasoning complexity. ## conclusion in this paper, we demonstrated that a unified verbalizer-retriever-reader framework, udt-qa, for open-domain qa over data and text. we proposed a novel data-to-text paradigm that can largely improve the verbalization effectiveness for downstream knowledge-intensive applications, i.e. opendomain qa, when attaining good intrinsic performances. with the verbalized knowledge, we achieved a new state-of-the-art result for nq. remarkably, we showed that simply augmenting the text index with the verbalized knowledge improve the performance without retraining the model. in addition to our method, there are many recently proposed approaches for open-domain qa that are orthogonal. for example, language models specifically optimized for dense retrieval @xcite , pretraining on large-scale qa data @xcite and hybrid system that consists of retriever, reranker, extractive reader and generative reader @xcite . incorporating those methods may further improve the performance for open-domain qa, and we leave that exploration for future work. lastly, instead of only considering a sanitized collection of knowledge sources, it is an interesting future direction to scale up the knowledge to web-scale @xcite .