F i E : Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering (2022)

Abstract: Generative models have recently started to outperform extractive models in Open Domain Question Answering, largely by leveraging their decoder to attend over multiple encoded passages and combining their information. However, generative models tend to be larger than extractive models due to the need for a decoder, run slower during inference due to auto-regressive decoder beam search, and their generated output often suffers from hallucinations. We propose to extend transformer encoders with the ability to fuse information from multiple passages, using global representation to provide cross-sample attention over all tokens across samples. Furthermore, we propose an alternative answer span probability calculation to better aggregate answer scores in the global space of all samples. Using our proposed method, we outperform the current state-of-the-art method by 2.5 Exact Match score on the Natural Question dataset while using only 25% of parameters and 35% of the latency during inference, and 4.4 Exact Match on WebQuestions dataset. When coupled with synthetic data augmentation, we outperform larger models on the TriviaQA dataset as well. The latency and parameter savings of our method make it particularly attractive for open-domain question answering, as these models are often compute-intensive.

## introduction open-domain question-answering is the task of answering an input question given a large external knowledge-base, such as the entire wikipedia. this problem is typically approached by leveraging a retriever model to first retrieve a set of relevant documents/passages using some ir method, which are then passed on to a reader model @xcite @xcite @xcite . the reader model then encodes all the passages through a transformer encoder separately, as transformers have quadratic computation with input sequence length. extractive span-based methods @xcite @xcite @xcite use these encoded representations for a per-passage selection of the answer span. generative models such as fid @xcite , and derivative models utilizing a similar reader, such as izacard and grave (2021a); @xcite ; @xcite ; @xcite use their decoder to attend to all the passages simultaneously by concatenating their encoded representations. the encoderdecoder cross attention enables the generative readers to fuse information globally across different passages. so far, the existing works have tried to perform global information fusion by adding a decoder and hence adopting the generative approach. we hypothesize that this is not efficient for extractive tasks where the answer exists in the provided context passages. it may be helpful to restrict the answer probability space to the given context instead of using the universal vocabulary, which might also lead to issues such as hallucinations @xcite @xcite . moreover, adding a decoder and performing auto-regressive answer generation increases the latency of the model. in this work, we propose to extend the transformer encoder of extractive models with the ability for early global fusion of information across input samples. we achieve this by adding global representation tokens to the input of the encoder, which can attend to all tokens in all passages with cross-sample attention. by doing so, not only do we remove the need for a decoder but also outperform existing generative approaches. extractive reader models typically @xcite @xcite passage encoder layer span classifier 𝑃𝑃 𝑠𝑠 (𝑠𝑠 𝐾𝐾 ) figure 1 : the overall architecture of our proposed model fie: fusion in encoder. the global representation tokens attend to all the tokens from all the passages, while all passages attend to themselves and to the global representation. we also adopt a similar approach to take the best advantage of the global information flow in our model. moreover, extractive models classify each token as the start/end of the answer span. the probability of a span is then the probability of its start token multiplied by its end token. this inherently assumes that the start and end probabilities of a span are independent of each other. we modify the probability space by directly calculating the score for each span rather than multiplying the start and end token scores. this approach enables us to calculate a separate score for each span and enables better aggregation of answer probability scores. we evaluate our proposed method on the most commonly used open-domain datasets -natural questions @xcite , triviaqa @xcite and webquestions @xcite . on natural questions, we outperform the previous best models having 4x more parameters by 2.5 exact match, achieving a new state-of-theart score of 58.4em. we also achieve an improvement in score over the previous best models by 4.4 exact match points on webquestions, and on triviaqa outperform the best performing models when coupled with data augmentation. our main contributions in this paper are -• fusing global information in the encoder by leveraging cross-sample attention, eliminating the need for a large and expensive decoder. ## conclusion we propose fie: fusion-in-encoder, where we extend transformer encoders with the ability to fuse information across multiple input samples using global representation via cross-sample attention and propose a global probability space for answer spans. facilitating this information flow across samples enables our proposed method to achieve state-of-the-art performance in 3 popular opendomain questions answering datasets by 2.5 em on nq, and 0.5 em on triviaqa, while simultaneously reducing parameter count and inference latency, and 4.4 em on webq. detailed ablations and analyses further demonstrate the effectiveness of our proposed method across multiple model sizes and the capability to fuse information from a large number of samples.