M 3: A Multi-View Fusion and Multi-Decoding Network for Multi-Document Reading Comprehension (2022)

Abstract: Multi-document reading comprehension task requires collecting evidences from different documents for answering questions. Previous research works either use the extractive modeling method to naively integrate the scores from different documents on the encoder side or use the generative modeling method to collect the clues from different documents on the decoder side individually. However, any single modeling method cannot make full of the advantages of both. In this work, we propose a novel method that tries to employ a multi-view fusion and multi-decoding mechanism to achieve it. For one thing, our approach leverages question-centered fusion mechanism and cross-attention mechanism to gather fine-grained fusion of evidence clues from different documents in the encoder and decoder concurrently. For another, our method simultaneously employs both the extractive decoding approach and the generative decoding method to effectively guide the training process. Compared with existing methods, our method can perform both extractive decoding and generative decoding independently and optionally. Our experiments on two mainstream multi-document reading comprehension datasets (Natural Questions and TriviaQA) demonstrate that our method can provide consistent improvements over previous state-of-the-art methods.

## introduction open domain question answering (qa) aims to produce an answer for a given question using a large text corpus source such as wikipedia. one of the typical approaches to open domain qa follows the retriever-reader framework @xcite @xcite , where a retriever first identifies the most relevant documents, then a reader understands the retrieved documents and produces an answer. in this work, we focus on improving the effectiveness of the reader, whose goal is to efficiently aggregate and combine evidence from multiple documents for better answering questions, which is also known as multi-document reading comprehension @xcite . depending on the difference of decoding approaches, recent works on multi-document reading comprehension could be mainly divided into two categories: extractive approaches @xcite @xcite and generative approaches @xcite @xcite . to produce an answer, extractive approaches make predictions by extracting a contiguous span from the given evidence documents as answer. since it is not straightforward for extractive models to aggregate and combine evidence from multiple documents (or paragraphs), various techniques have been put forward to address this limitation @xcite @xcite . besides, another drawback of extractive models is that they can't produce an answer string if it does not contained in the given evidence documents. @xcite seek to address these issues in a generative way. in their method, they use a pre-trained generative model @xcite to perform evidence fusion in the decoder and sequentially generate an answer string, which is later wildly adopt in @xcite @xcite . recently, on multi-domain dialogue state tracking, @xcite achieve impressive performance by utilizing a generative model to generate slot values while using a classifier to discriminate whether the corresponding (domain, slot) pair is actually triggered. on open domain qa, @xcite show that ensembling the complementary results of generate and extractive approaches yields significant performance improvement. these hint that discriminative and generative objectives could cooperate with each other and work well. intuitively, to answer a question based on given evidence documents, people could effortless extract all the correct answer spans from the given documents and generate a valid answer if the evidence documents do not contain one. these inspire us to explore the joint advantage of using both the extractive modeling method and generative modeling method in multi-document reading comprehension. in this paper, to have the best of both the extractive modeling method and the generative modeling method, we present m3: a multi-view fusion and multi-decoding network for multi-document reading comprehension. unlike previous approaches, m3 not only performs evidence information fusion in the encoder and decoder simultaneously (multi-view fusion), but also is able to extract an answer or generate an answer at the same time (multi-decoding). more precisely, we take advantage of a question-centered fusion mechanism to gather question-related clues from different documents on the encoder side and meanwhile make use of a cross-attention mechanism @xcite to aggregate and combine evidence from multiple passages in the decoder. to aggregate useful training signals from both the encoder side and the decoder side, we equip m3 with an extractive reader in the encoder and a generative reader on the decoder side concurrently. we evaluate our proposed approach by experimenting on two commonly used open domain qa datasets: natural questions @xcite and triviaqa @xcite . experimental results show that m3 exhibits a consistent better generative performance for various generative models of comparable size on all datasets, outperforming recent generative approaches by about 2 exact match points. we also find that the extraction performance of m3 are superior to those of many typical extractive models that have similar or more model parameters. last but not least, based on careful ablation studies, we demonstrate that the proposed multi-view information integration and the proposed multi-decoding mechanism are the key elements that lead to performance improvement. ## model our proposed model m3 consists of three components: a document content extraction module, a cross-document information integration module and a multivariate heterogeneous decoding module. figure 1 gives an overview of the architecture of our model. in the following sections, we will describe each component and our training objective in detail. ## ablations in this section, we investigate design choices regarding the key elements of our method: the multiview evidence fusion and the multivariate heterogeneous decoding strategy. specifically, we mainly consider the following variants of m3: variant 1: this variant perform evidence fusion in the decoder only and is equipped with a generate reader only, which is a fusion-in-decoder model @xcite indeed. variant 2: this variant perform evidence fusion in the encoder only and is equipped with an extractive reader only. variant 3: this variant perform evidence fusion both in the encoder and decoder and is equipped with a generative reader only. variant 4: this variant is the same as m3 in model architecture, but it is only trained with filtered samples, in each of which at least one of the reference answers must appear in the given documents 3 . 3 for m3, we train it using all the available training samples. when none of the reference answers can be found in the given documents, we simply set the heterogeneous decoding weight defined in equation 25 to be zero and train it as usual. model em ( generative ## conclusion in this work, we propose m3, an effective multidocument reading comprehension models, which perform evidence information integration from the perspective of both the encoder and the decoder. without additional training, m3 could perform extractive decoding and generative decoding individually and optionally. experiment results on two mainstream open domain qa datasets show that the proposed model m3 achieves better generative performance than state-of-the-art generative methods and obtain competitive or better performance than previous typical extractive models.