Generative Context Pair Selection for Multi-hop Question Answering (2021)

Abstract: Compositional reasoning tasks such as multi-hop question answering require models to learn how to make latent decisions using only weak supervision from the final answer. Crowdsourced datasets gathered for these tasks, however, often contain only a slice of the underlying task distribution, which can induce unanticipated biases such as shallow word overlap between the question and context. Recent works have shown that discriminative training results in models that exploit these underlying biases to achieve a better held-out performance, without learning the right way to reason. We propose a generative context selection model for multi-hop QA that reasons about how the given question could have been generated given a context pair and not just independent contexts. We show that on HotpotQA, while being comparable to the state-of-the-art answering performance, our proposed generative passage selection model has a better performance (4.9% higher than baseline) on adversarial held-out set which tests robustness of model’s multi-hop reasoning capabilities.

## introduction recently many reading comprehension datasets like hotpotqa @xcite and wiki-hop @xcite that require compositional reasoning over several disjoint passages have been introduced. this style of compositional reasoning, also referred to as multi-hop reasoning, first requires finding the correct set of passages relevant to the question and then finding the answer span in the selected set of passages. these dataset are often collected via crowdsourcing, which makes the training and evaluation of such models heavily reliant on the quality of the collected held-out sets. crowdsourced datasets, however, often present only a partial picture of the underlying data distribution. learning complex latent sequential decisions, like in multi-hop reasoning, to answer a given question under such circumstances is marred by numerous biases, such as annotator bias @xcite , label bias @xcite , survivorship bias @xcite , and ascertainment bias @xcite . as a result, testing model performance on such biased held-out sets becomes unreliable as the models exploit these biases and learn shortcuts to get the right answer but without learning the right way to reason. consider an example from hotpotqa in figure 1 , where the latent entity "virgina commonwealth university" can be used by the model @xcite to bridge the two relevant passages (highlighted in green) from the original dev set and correctly predict the answer "1838". however, upon adding an adversarial context (highlighted in pink) to the pool of contexts, the model prediction changes to "1938" implying that the model did not learn the right way to reason. this is because the discriminatively trained passage selector exploits lexical cues like "founded" in the second passage and does not pay attention to the complete question. the absence of such adversarial contexts at training allows the model to find incorrect reasoning paths. in this work, we propose a generative context pair selection model that reasons through the data generation process of how a specific question could have been generated given pair of passages. we show that our proposed passage selection module has a better performance on the original (+2.2%) and the adversarial dev set (+4.9%) that tests the model's reasoning abilities (unlike the original dev set which is marred by bias). we use a generic answering model and show that while being comparable in end-to-end performance with close to state-of-the-art systems on the original dev set, our model provides a better performance on the adversarial set. any advances in the answering model can be applied in a straightforward manner to a generative passage selector to further improve performance. ## generative passage selection given a set of contexts @xmath0, ..., c n }, the goal of multi-hop question answering is to combine information from c to an identify answer span a for a given question q. let ψ = {(c i , c j ) = c i j : c @xmath1, c @xmath2. existing models for multi-hop question answering @xcite consist of two components: a discriminative passage selection and an answering model. passage selection identifies which pairs of contexts are relevant for answering the given question, i.e., it estimates p(c i j | q, ψ). this is followed by the answering model to extract the answer span given a context pair and the question (p(a | q, c i j )). these are combined as follows: the discriminative passage selector learns to select a set of contexts by conditioning on the question representation. this learning process does not encourage the model to pay attention to the entire question, which can result in ignoring parts of the question, and thus, learning spurious correlations. for prediction, best context pair c * i j is used by the answering module to get the answer, a * = argmax p(a | q, c * i j ). as shown by @xcite , using the top scoring reasoning chain to answer the question is often sufficient and does not require marginalization over multiple chains. foot_0 ## related work many recent passage selection models for hot-potqa and wikihop's distractor style setup employ discriminative context selectors given the question @xcite @xcite . the high performance of such passage selectors can be attributed to existing bias in hot-potqa @xcite , which allows shallow lexical overlap of question with a single context to result in the correct answer. another more general line of work dynamically updates the working memory to re-rank the set of passage at each hop @xcite . with the release of datasets like searchqa @xcite , triviaqa @xcite , and naturalquestions @xcite , a lot of work has been done in open-domain passage retrieval, especially in the full wikipedia setting. however, these questions do not necessarily require multi-hop reasoning. a series of work has tried to match a document-level summarized embedding to the question @xcite @xcite for obtaining the relevant answers. in generative question answering, a few works @xcite have used a joint question answering approach on a single context. a large body of work has employed simple question generation for factoid answers in numerous cases, like answer verification @xcite , fact checking @xcite , data augmentation @xcite , pedagogical systems @xcite , and dialog systems @xcite etc. ## conclusion we proposed a generative formulation of context pair selection for multi-hop question answering. by encouraging this selection model to explain the entire question, it is less susceptible to bias, performing substantially better on adversarial data than existing discriminative methods. our proposed model is simple to implement and can be used with any existing (or future) answering model; we will release code to support this integration. since context pair selection scales quadratically with the number of contexts, it is not ideal for scenarios that involve a large number of possible contexts. however, it allows for deeper inter-document interaction as compared to other approaches that use summarized document representations. with more reasoning steps, selecting relevant documents given only the question becomes challenging, increasing the need for inter-document interaction. an easy way to reduce the computation cost is to consider only a set of top-k contexts and perform a two-stage coarse-to-fine passage selection. the generative story presented in the paper may not work for question types beyond the datasets considered, for eg., in case of multiple (>1) bridge entities or more than two contexts. however, we demonstrate that this idea works for most common reasoning types that are central in current multihop reasoning datasets. the code is available at https://github.com/ddua/jointqa