Knowledge Transfer between Structured and Unstructured Sources for Complex Question Answering (2022)

Abstract: Multi-hop question answering (QA) combines multiple pieces of evidence to search for the correct answer. Reasoning over a text corpus (TextQA) and/or a knowledge base (KBQA) has been extensively studied and led to distinct system architectures. However, knowledge transfer between such two QA systems has been under-explored. Research questions like what knowledge is transferred or whether the transferred knowledge can help answer over one source using another one, are yet to be answered. In this paper, therefore, we study the knowledge transfer of multi-hop reasoning between structured and unstructured sources. We first propose a unified QA framework named SimultQA to enable knowledge transfer and bridge the distinct supervisions from KB and text sources. Then, we conduct extensive analyses to explore how knowledge is transferred by leveraging the pre-training and fine-tuning paradigm. We focus on the low-resource fine-tuning to show that pre-training SimultQA on one source can substantially improve its performance on the other source. More fine-grained analyses on transfer behaviors reveal the types of transferred knowledge and transfer patterns. We conclude with insights into how to construct better QA datasets and systems to exploit knowledge transfer for future work.

## introduction structured knowledge source, such as knowledge base (kb) and unstructured knowledge source, such as text corpus, are arguably the most popular sources for complex question answering (cqa). multi-hop kb based question answering (kbqa) systems translate questions to logical forms to be executed over a kb for finding answers @xcite @xcite @xcite figure 1 : to facilitate knowledge transfer between structured and unstructured sources, we develop a unified framework simultqa that can leverage supervisions from both sources to answer complex questions. @xcite , while text based qa (textqa) systems leverage large text corpora to retrieve paragraphs and extract answer spans for complex questions @xcite @xcite @xcite . however, despite the impressive performance of separate kbqa and textqa systems, it is not quite clear to the community whether a system trained on one source can be transferred and beneficial to question answering over another source. inspired by the general transfer learning in nlp by pre-trained language models (plms) @xcite @xcite , it is important to study this research problem systematically and thoroughly for the following reasons. first, given the heterogeneity of structured and unstructured sources, it is desirable to build a unified reasoning module to work on both text and kb and combine different source-specific supervisions. second, transfer learning has shown to boost the performance on low-resource domains, and it would be practically useful to leverage annotated datasets on one source for cqa on the other source, especially when human annotations are expensive to create new multi-hop qa datasets. third, it is also critical to investigate what kind of knowledge can be transferred, which can inspire future cqa dataset creation and system design. one major obstacle in such an investigation for knowledge transfer between structured and unstructured sources is the disparity of them and their specifically designed qa systems as we mentioned earlier. for instance, kb is highly structured and curated where complex query functions can be executed, while text data is unstructured and noisier, leading to quite distinct qa systems. one relevant line of research is hybridqa that tries to leverage multiple sources for qa @xcite @xcite @xcite . to operate their single model on both kb and text, these methods primarily convert distinct sources into similar data format, e.g., merge text and kb by entity linking, which sacrifices unique characteristics of each source to some extent and makes it harder to investigate knowledge transfer as sources are entangled together. thus, typical hybridqa methods are not suitable for studying knowledge transfer problem. in this paper, our first contribution is proposing a unified cqa framework to enable knowledge transfer between structured and unstructured sources. the proposed framework, simultqa, could perform multi-hop reasoning over text and kb simultaneously by collecting reasoning paths from either text or kb, then rerank paths to select the best one for generating the answer. there are several new and desirable properties of si-multqa. first, simultqa unifies the recent advanced kbqa @xcite and textqa @xcite systems seamlessly, which preserves their unique strengths maximally to handle various reasoning types. second, simultqa can utilize distinct supervisions from both sources, which has the potential to combine both kbqa and textqa datasets for a unified training. last but not least, since simultqa can be applied to any source, we can pre-train it on kb and fine-tune it on text and vice versa, which makes it easier to quantify transfer effect brought by the pre-training on a different source. in summary, despite the framework design looks straightforward, we are the first to unify two seemingly distinct cqa systems and study knowledge transfer between two sources for cqa. with simultqa that enables knowledge transfer, our second contribution is to systematically analyze the transfer behavior to help us deeply understand the nature of the multi-hop reasoning process in kb and text. we apply our methodology to cwq @xcite and hot-potqa @xcite , which are arguably the most popular dataset in kb and text source, and are representative enough to cover most of the reasoning types on kb and text. we first show that pre-training on one source can consistently improve the fine-tuning performance on the other one in the low-resource setting, indicating future data-hungry qa systems can be boosted by pre-training on another disparate source, especially when human annotation is expensive. more interestingly, further fine-grained analyses attempt to reveal sources of performance gain and find out what knowledge is transferred. we mainly investigate three aspects, reasoning types, reasoning hops and question similarity. we find that despite kb and text sources are quite disparate, simultqa still find ways to transfer knowledge by learning a shared semantic space for the reasoning and a high-level understanding beyond distinct surface forms of reasoning paths. in addition, we study a more challenging transfer setting where we seek to use text reasoning to answer kb-based questions foot_1 and vice versa. promising results are obtained by using text knowledge to help kb questions highlighting the expressiveness of text corpus. we conclude that knowledge transfer between structured and unstructured sources is an intriguing direction to combine the strengths of kbqa and textqa systems and to use data from one source to boost qa on the other. to the best of our knowledge, this paper is the first to study knowledge transfer between kb-and text-based cqa in a quantitative and systematic manner. ## simultqa framework simultqa is a unified framework for multi-hop reasoning to incorporate both kb and text sources. it consists of two stages, iteratively reasoning and final reranking, which can be trained with supervisions from both sources. ## knowledge transfer experiments we focus on investigating knowledge transfer between structured and unstructured sources in this paper, though the proposed simultqa can be applied to any open-domain cqa datasets. we seek to answer three research questions (rqs): • rq1: can the knowledge learned on one source help the qa performance on another one? ( §4.2) • rq2: what kind of knowledge has been transferred between kb and text? ( §4.3) • rq3: can knowledge transfer help answer questions by both sources? ( §4.4) ## discussion for future directions based on our findings of knowledge transfer for cqa in this paper, we discuss the following directions for future cqa datasets and systems. knowledge transfer for efficient cqa dataset annotations. when annotating new cqa datasets whether on text or kb, it would be beneficial to leverage pre-trained simultqa on other sources to discover high-quality reasoning paths for further annotating, which will save much annotation cost. diversity of reasoning types. both text and kb sources are dominant by relatively easy reasoning types, e.g., composition and conjunction. future cqa datasets should pursue more diverse and harder reasoning types, e.g., types with constraints and arithmetic operations @xcite . a universal reasoning module. investigating knowledge transfer between text and kb in this paper suggests that despite the discrepancy of surface forms in different sources, their underlying reasoning processes could be shared. this points out the possibility of learning a universal reasoning process from multiple sources and it is strongly desired to modularize such a reasoning process, which can be injected it into future qa systems. ## conclusion in this paper, we study cqa over structured and unstructured knowledge sources (i.e., kb and text particularly), and focus on studying the knowledge transfer between different knowledge sources. to facilitate the transfer, we first propose a unified cqa framework, simultqa to bridge kbqa and textqa systems. empirical results show that knowledge transfer enables substantial improvements on low-resource domains. more importantly, we conduct fine-grained analyses to shed more light on how knowledge is transferred to inspire future research on knowledge transfer between sources, and we conclude the paper with insights for future cqa datasets and systems.