Unsupervised Multi-hop Question Answering by Question Generation (2021)

Abstract: Obtaining training data for multi-hop question answering (QA) is time-consuming and resource-intensive. We explore the possibility to train a well-performed multi-hop QA model without referencing any human-labeled multi-hop question-answer pairs, i.e., unsupervised multi-hop QA. We propose MQA-QG, an unsupervised framework that can generate human-like multi-hop training data from both homogeneous and heterogeneous data sources. MQA-QG generates questions by first selecting/generating relevant information from each data source and then integrating the multiple information to form a multi-hop question. Using only generated training data, we can train a competent multi-hop QA which achieves 61% and 83% of the supervised learning performance for the HybridQA and the HotpotQA dataset, respectively. We also show that pretraining the QA system with the generated data would greatly reduce the demand for human-annotated training data. Our codes are publicly available at https://github.com/teacherpeterpan/Unsupervised-Multi-hop-QA .

## introduction extractive question answering (eqa) is the task of answering questions by selecting a span from the given context document. works on eqa can be divided into the single-hop @xcite @xcite and multihop cases @xcite @xcite . unlike single-hop qa, which assumes the question can be answered with a single sentence or document, multi-hop qa requires combining disjoint pieces of evidence to answer a question. though different well-designed neural models @xcite have achieved near-human performance on the multi-hop qa datasets @xcite , these approaches rely heavily on the figure 1 : an overview of our approach for generating bridge-type multi-hop questions from table and text. the full set of supported input types and question types are described in section 3.2. to address the above problem, we pursue a more realistic setting, i.e., unsupervised multi-hop qa, in which we assume no human-labeled multi-hop question is available for training, and we explore the possibility of generating human-like multi-hop question-answer pairs to train the qa model. we study multi-hop qa for both the homogeneous case where relevant evidence is in the textual forms @xcite and the heterogeneous case where evidence is manifest in both tabular and textual forms @xcite . though successful attempts have been made to generate single-hop question-answer pairs by style transfer @xcite or linguistic rules @xcite , these methods are not directly applicable to the multi-hop setting as: 1) they cannot integrate information from multiple data sources, and 2) they only handle free-form text but not heterogeneous sources as input contexts. we propose multi-hop question generator (mqa-qg), a simple yet general framework that decomposes the generation of a multi-hop question into two steps: 1) selecting relevant information from each data source, 2) integrating the multiple information to form a question. specifically, the model first defines a set of basic operators to retrieve / generate relevant information from each input source or to aggregate different information. afterwards, we define six reasoning graphs. each corresponds to one type of multi-hop question and is formulated as a computation graph built upon the operators. we generate multi-hop question-answer pairs by executing the reasoning graph. figure 1 shows an example of generating a table-to-text question: a) given the inputs of (table, text), the f indbridge operator locates a bridge entity that connects the contents between table and text. b) we generate a simple, single-hop question for the bridge entity from the text (qgwithent operator) and generate a sentence describing the bridge entity from the table (describeent operator). c) the bridgeblend operator blends the two generated contents to obtain the multi-hop question. we evaluate our method on two multi-hop qa datasets: hotpotqa @xcite and hy-bridqa @xcite . questions in hot-potqa reason over multiple texts (homogeneous data), while questions in hybridqa reason over both table and text (heterogeneous data). the experiments show that mqa-qg can generate highquality multi-hop questions for both datasets. without using any human-labeled examples, the generated questions alone can be used to train a surprisingly well qa model, reaching 61% and 83% of the f1 score achieved by the fully-supervised setting on the hybridqa and hotpotqa dataset, respectively. we also find that our method can be used in a few-shot learning setting. for example, after pretraining the qa model with our generated data, we can obtain 64.6 f1 with only 50 labeled examples in hotpotqa, compared with 21.6 f1 without the warm-up training. ## related work unsupervised question answering. to reduce the reliance on expensive data annotation, unsupervised / zero-shot qa has been proposed to train question answering models without any humanlabeled training data. @xcite proposed the first unsupervised qa model which generates synthetic (context, question, answer) triples to train the qa model using unsupervised machine translation. however, the generated questions are unlike human-written questions and tend to have a lot of lexical overlaps with the context. to address this, followup works utilized the wikipedia cited documents @xcite , predefined templates @xcite , or pretrained language model @xcite to produce more natural questions resembling the human-annotated ones. however, all the existing studies are focused on the squad @xcite dataset to answer single-hop and text-only questions. these methods do not generalize to multi-hop qa because they lack integrating and reasoning over disjoint pieces of evidence. furthermore, they are restricted to text-based qa without considering structured or semi-structured data sources such as kb and table . in contrast, we propose the first framework for unsupervised multi-hop qa, which can reason over disjoint structured or unstructured data to answer complex questions. multi-hop question generation. question generation (qg) aims to automatically generate questions from textual inputs @xcite . early work of question generation (qg) relied on syntax rules or templates to transform a piece of given text to questions @xcite . with the proliferation of deep learning, qg evolved to use supervised neural models, where most systems were trained to generate questions from (passage, answer) pairs in the squad dataset @xcite @xcite . with the advent of pretraining language models @xcite , the challenge of generating single-hop questions similar to squad group operator @xmath0, text d) or texts (d1, d2) → bridge entities e b select an entity e b that links the two input texts d1 and d2 (or links the table t and the text d) f indcoment text @xmath1, datetime, number, etc.). generation qgwithans (text d, answer a) → question q generate a single-hop question q with answer a from the input text d qgwithent (text d, entity e) → question q generate a single-hop question q that contains the given entity e from the input text d describeent (table t , entity e) → sentence s generate a sentence s that describes the given entity e based on the information of the table t quest osent question @xmath2, sentence s, bridge e b ) → bridge-type multi-hop question q b generate a bridge-type multi-hop question q b by fusing the single-hop question q and the sentence s given the entity e b as the bridge compblend (question q1, question q2) → comparative multi-hop question q c generate a comparison-type multi-hop question q c by fusing two single-hop questions ## methodology the setup of multi-hop qa is as follows. given a question q and a set of input contexts @xmath3, • • • , c n }, where each context c i can be a passage, table, image, etc., the qa model p θ (a|q, c) predicts the answer a for the question q by integrating and reasoning over information from c. in this paper, we consider two-hop questions and denote the required contexts as c i and c j . formally, each time our model takes as inputs c i , c j to generate a set of (q, a) pairs. we focus on two modalities: the heterogeneous case where c i , c j are table and text and the homogeneous case where c i , c j are both texts. however, the design of our framework is flexible enough to generalize to multihop qa for other modalities. our model mqa-qg consists of three compo-nents: operators, reasoning graphs, and question filtration. operators are atomic operations implemented by rules or off-the-shelf pretrained models to retrieve, generate, or fuse relevant information from input contexts (c i , c j ). different reasoning graphs define different types of reasoning chains for multi-hop qa with the operators as building blocks. training (q, a) pairs are generated by executing the reasoning graphs. question filtration removes irrelevant and unnatural (q, a) pairs to give the final training set d for multi-hop qa. ## experiments we evaluate our framework on two multi-hop qa datasets: hotpotqa @xcite and hy-bridqa @xcite . hotpotqa focuses on multi-hop qa over homogeneous inputs, while hybridqa deals with multi-hop qa over heterogeneous information. hotpotqa contains ∼100k crowd-sourced multi-hop questions, where each question requires reasoning over two supporting wikipedia documents to infer the answer. hy-bridqa contains ∼70k human-labeled multi-hop questions, where each question is aligned with a structured wikipedia table and multiple passages linked with the entities in the table. the questions split train dev test total hotpotqa bridge 72,991 5,918 -78,909 (81 %) comparison 17,456 1,487 -18,943 (19 %) total 90,447 7,405 -97,852 hybridqa in-passage 35,215 2,025 2,045 39,285 (56 %) in- ## conclusion and future works in this work, we study unsupervised multi-hop qa and propose a novel framework mqa-qg to generate multi-hop questions via composing reasoning graphs built upon basic operators. the experiments show that our model can generate human-like questions that help to train a well-performing multi-hop qa model in both the unsupervised and the fewshot learning setting. further work is required to include more flexible paraphrasing at the fusion stage. we can also design more reasoning graphs and operators to generate more complex questions and support more input modalities.