Complementary Evidence Identification in Open-Domain Question Answering (2021)

Abstract: This paper proposes a new problem of complementary evidence identification for open-domain question answering (QA). The problem aims to efficiently find a small set of passages that covers full evidence from multiple aspects as to answer a complex question. To this end, we proposes a method that learns vector representations of passages and models the sufficiency and diversity within the selected set, in addition to the relevance between the question and passages. Our experiments demonstrate that our method considers the dependence within the supporting evidence and significantly improves the accuracy of complementary evidence selection in QA domain.

## introduction in recent years, significant progress has been made in the field of open-domain question answering @xcite @xcite @xcite . very recently, some works turn to deal with a more challenging task of asking complex questions @xcite @xcite from the open-domain text corpus. in the open-domain scenario, one critical challenge raised by complex questions is that each question may require multiple pieces of evidence to get the right answer, while the evidence usually scatters in different passages. examples in figure 1 shows two types of questions that require evidence from multiple passages. to deal with the challenging multi-evidence questions, an open-domain qa system should be able to (1) efficiently retrieve a small number of passages that cover the full evidence; and (2) accurately extract the answer by jointly considering the candidate evidence passages. while there have been several prior works in the latter direction @xcite @xcite figure 1 : examples of complex questions involving two facts of a person. different facts are color-coded. p# are all relevant passages, while only the ones with solid-line boxes are the true supporting passages. @xcite , the solutions to the first problem still rely on traditional or neural information retrieval (ir) approaches, which solely measure the relevance between the question and each individual paragraph, and will highly possibly put the wrong evidence to the top. 1 for example in figure 1 (top), p1 and p2 are two candidate evidence passages that are closely related to the question but only cover the same unilateral fact required by the question, therefore leading us to the wrong answer newton. this paper formulates a new problem of complementary evidence identification for answering complex questions. the key idea is to consider the problem as measuring the properties of the selected passages, more than the individual relevance. specifically, we hope the selected passages can serve as a set of spanning bases that supports the question. the selected passage set thus should satisfy the properties of (1)relevancy, i.e., they should be closely related to the question; (2) diversity, i.e., they should cover diverse information given the coverage property is satisfied; (3) compactness, i.e., the number of passages to satisfy the above properties should be minimal. with these three defined properties, we hope to both improve the selective accuracy and encourage the interpretability of the evidence identification. note that complementary evidence identification in qa is different from search result diversification (srd) in ir on their requirement of compactness. the size of the selected set is constrained in qa tasks by the capability of downstream reasoning models and practically needs to be a small value, whereas it is not the case in srd. to achieve the above goals, a straightforward approach is to train a model that evaluates each subset of the candidate passages, e.g., by concatenating passages in any subsets. however, this approach is highly inefficient since it requires to encode o(k l ) passage subsets, where k is the total number of candidates and l is the maximum size of subsets. thus, a practical complementary evidence identification method needs to be computationally efficient. this is especially critical when we use heavy models like elmo @xcite and bert @xcite , where passage encoding is time and memory consuming. to this end, we propose an efficient method to select a set of spanning passages that is sufficient and diverse. the core idea is to represent questions and passages in a vector space and define the measures of our criterion in the vector space. for example, in the vector space, sufficiency can be defined as a similarity between the question vector and the sum of selected passage vectors, measured by a cosine function with a higher score indicating a closer similarity; and diversity can be defined as ℓ 1 distance between each pair of passages. by properly training the passage encoder with a loss function derived by the above terms, we expect the resulted vector space satisfies the property that the complementary evidence passages lead to large scores. in addition, our method only encodes each passage in the candidate set once, which is more efficient than the naive solution mentioned above. to evaluate the proposed method, we use the multi-hop qa dataset hotpotqa (the full wiki setting) since the ground-truth of evidence passages are provided. experiments show that our method significantly improves the accuracy of complementary evidence selection. ## conclusion in the paper, we propose a new problem of complementary evidence identification and define the criterion of complementary evidence in vector space. we further design an algorithm and a loss function to support efficient training and inference for complementary evidence selection. compared to the baseline, our approach improves more than 20% and remains to scale well to the computationally complex cases.