Mitigating False-Negative Contexts in Multi-document Question Answering with Retrieval Marginalization (2021)

Abstract: Question Answering (QA) tasks requiring information from multiple documents often rely on a retrieval model to identify relevant information for reasoning. The retrieval model is typically trained to maximize the likelihood of the labeled supporting evidence. However, when retrieving from large text corpora such as Wikipedia, the correct answer can often be obtained from multiple evidence candidates. Moreover, not all such candidates are labeled as positive during annotation, rendering the training signal weak and noisy. This problem is exacerbated when the questions are unanswerable or when the answers are Boolean, since the model cannot rely on lexical overlap to make a connection between the answer and supporting evidence. We develop a new parameterization of set-valued retrieval that handles unanswerable queries, and we show that marginalizing over this set during training allows a model to mitigate false negatives in supporting evidence annotations. We test our method on two multi-document QA datasets, IIRC and HotpotQA. On IIRC, we show that joint modeling with marginalization improves model performance by 5.5 F1 points and achieves a new state-of-the-art performance of 50.5 F1. We also show that retrieval marginalization results in 4.1 QA F1 improvement over a non-marginalized baseline on HotpotQA in the fullwiki setting.

## introduction multi-document question answering refers to the task of answering questions that require reading multiple documents, extracting relevant facts, and reasoning over them. systems built for this task typically involve retrieval and reasoning components that work in tandem. the retrieval component needs to extract information from the documents that is suitable for the reasoning model to perform the end-task effectively. recent advances in reading comprehension have resulted in models that have been shown to answer questions requiring complex reasoning types such as bridging, comparison @xcite or even arithmetic @xcite , given adequate context. however, when the context needs to be retrieved from a large text corpus (e.g., wikipedia), the performance of such reading comprehension models is greatly affected by the quality of the retrieval model. given supervision at all stages (i.e., document, supporting evidence and answer), it is common to build retrieval and reasoning models independently and connect them as a pipeline at test time. in this case, the retrieval and reasoning models are usually trained to maximize the likelihood of labeled supporting evidence snippets and the answer given the gold context respectively. however, in a multi-document qa setting, it is common to have some relevant snippets not marked as gold. two such examples are shown in figure 1 . in the first example, only snippet 2 is marked as gold evidence, and consequently snippets 1 and 3 are treated as negative examples during retrieval. this is problematic because unlike snippet 1 which is actually irrelevant, snippet 3 is not only useful, but provides an even more direct way to derive the correct answer since it does not require a subtraction. similarly, in the second example two evidence snippets from different documents contain the same information, thus at least two contexts can be used to answer this question, yet only one of them is labeled as being a positive example for the training objective. we define these contexts that contain non-gold snippets and can still be used to answer the questions as alternative contexts. alternative contexts are inevitable when datasets are created from large corpora, because it is prohibitively expensive to exhaustively annotate all possible contexts. these alternative contexts are false negatives during training and lead to a noisy and weak learning signal, even with this "fully-supervised" setup. we design a training procedure for handling these false negatives, as well as cases where retrieval should fail (i.e., when the question is unanswerable). specifically, we assign probabilities to documents, evidence candidates, and potential answers with parameterized models, and marginalize over a set of potential contexts by combining top retrieved evidence from each document, allowing the model to score false negatives highly. to make the marginalization feasible, we decompose the retrieval problem into document selection and evidence retrieval and show how we can still model contexts as sets. we evaluate our model on two multi-document qa datasets: iirc @xcite and @xcite . we see 2.8 and 4.8 f1 point improvement on iirc and hot-potqa respectively by jointly modeling our proposed set-valued retrieval and the reasoning steps, and a further 2.7 and 4.1 f1 point improvement re-spectively by using retrieval marginalization. our final result of 50.5 f1 on the test set of iirc represents a new state-of-the-art. ## multi-document qa here we formally describe the multi-document qa setting and highlight the two main challenges in this setting that our work attempts to address. problem definition multi-document question answering measures both the retrieval and reasoning abilities of a qa system. given a question q and a set of documents @xmath0, d foot_0 , ..., d n }, each document containing a set of evidence snippets ## learning with marginalization over retrieval to address these challenges, we decompose the retrieval problem into document selection and evidence retrieval, leaving the handling of unanswerable questions entirely to the evidence retrieval component. these two components together produce a probability distribution over sets of retrieved contexts, which we marginalize over during training to account for false negatives. our general framework is illustrated in figure 2 . ## main results on iirc ## results on hotpotqa the general effectiveness of retrieval marginalization and joint modeling on hotpotqa fullwiki is displayed in ## qualitative analysis as mentioned in section 5.1 and section 6, the reason for our model to achieve higher qa scores though having lower scores in matching annotated evidence is that our model locates false-negative evidence that can be used as alternatives for reasoning. in joint models for retrieval and reasoning more recent work such as orqa @xcite and dense passage retrieval @xcite focus on modeling retrieval for question answering. but their main focus is to develop efficient neural retrieval systems that help to scale up qa to a large corpus, which is orthogonal to our contribution in mitigating false-negative contexts. some recent works also investigate the possibility of modeling retrieval as a latent task and enabling end-to-end training. realm @xcite used a neural retriever to augment language models with external knowledge for question answering. while realm also uses a maximum marginal likelihood objective, it only marginalizes over retrieved documents. however, our model marginalizes over a set of contexts consisting of evidence snippets from different documents to account for set-valued retrieval of variable sizes, which is crucial for multidocument qa. rag @xcite ) adopted a similar method as realm, but for sequence generation tasks. rag can be adapted to perform multi-hop question answering (xiong et al., 2020), and we directly compared our results with multihop rag on hotpotqa fullwiki in the experiments. finally, our work leverages marginalization over latent variables to deal with weak and noisy supervision signals, which is reminiscent of using maximum marginal likelihood for training weakly supervised semantic parsers @xcite . ## conclusion we proposed a new probabilistic model for retrieving set-valued contexts for multi-document qa and show that training the qa model with marginalization over this set can help mitigate the false negatives in evidence annotations. experiments on iirc and hotpotqa fullwiki show that our proposed framework can learn to retrieve unlabeled alternative contexts and improves qa f1 by 5.5 on iirc and 8.9 on hotpotqa. failed modes in our case, failing to answer a user-issued question may result in incorrect or misleading information. thus we should be careful when put our systems into practical use. computation power our most expensive experiment takes about 1.5 days to run on two rtx 8000 (48gb) gpus or one a100 (40gb) gpu, while a typical experiment takes about half of that computing power. while conducing experiments, we made effort to take advantage of technologies such as mixed-precision training to shorten our training time under the same experiment setting and save power consumption.