A Copy-Augmented Generative Model for Open-Domain Question Answering (2022)

Abstract: Open-domain question answering is a challenging task with a wide variety of practical applications. Existing modern approaches mostly follow a standard two-stage paradigm: retriever then reader. In this article, we focus on improving the effectiveness of the reader module and propose a novel copy-augmented generative approach that integrates the merits of both extractive and generative readers. In particular, our model is built upon the powerful generative model FiD (CITATION). We enhance the original generative reader by incorporating a pointer network to encourage the model to directly copy words from the retrieved passages. We conduct experiments on the two benchmark datasets, Natural Questions and TriviaQA, and the empirical results demonstrate the performance gains of our proposed approach.

## introduction open-domain question answering (odqa) focuses on providing highly precise answers to natural language questions from a large collection of unstructured text data @xcite . with the pioneering work of drqa @xcite , modern approaches to odqa commonly adopt a simple two-stage retriever-reader pipeline, that firstly retrieve a relatively small number of support passages @xcite @xcite , followed by the reader identifying the answer. the reader models can be broadly categorized into two classes: extractive @xcite @xcite and generative @xcite @xcite . recently, benefiting from the powerful ability of large-scale pre-trained encoderdecoder language models @xcite and the capability of aggregating information from multiple passages @xcite and grave, 2021b), generative approaches have achieved in general better performance than extractive methods. compared to extractive models, generative models generate text more freely, which makes it often suffer from the problem of producing hallucinated text that is factual inaccuracy or inconsistent to the input. this problem has been addressed in tasks like text summarization @xcite and machine translation @xcite . we found that the phenomenon also happens in odqa. as shown in inspired by the work of @xcite , we enhance the generative model with a pointer net- question + passage 1 question + passage 2 question + passage k answer (shifted) ... encoder encoder encoder decoder + + linear encoder hidden states context attention distribution concat vocabulary distribution p gen 1-p gen p gen linear final distribution work @xcite , that enables the model to directly copy text from the retrieved passages while retains the ability of generating new words when the true answers are not explicitly present in the input. to be more specific, our model fusion-in-decoder pointer-generator network (fid-pgn) is built upon the state-of-the-art model fid. we reuse the encoder-decoder attention scores as the copy distribution to reduce the computational cost. compared to fid, we achieve comparative or even better accuracy on the naturalquestions (nq) @xcite and triviaqa @xcite benchmarks, with less passages used in training. our experiments results show the effectiveness and efficiency of our model. ## method our model follows the standard two-stage retrieverreader framework with a focus on the enhancement of the reader module built upon the fid reader. we adopt the retriever results of fid-kd, where a dense retriever similar to dpr @xcite is used. a pointer network is integrated into the fid reader to facilitate copying words from the retrieved passages. the overall reader architecture is depicted in figure 1 . reader encoder. the reader encoder of our model is identical to the one of fid reader. we firstly concatenate the given question q with each retrieved passage p i as x @xmath0; p i ]. next, we pass each x i individually to the reader encoder, i.e., the encoder of t5 or bart model, and obtain the hidden representations h @xmath1,1 , h i,2 , . . . , h i,n ) of the questionpassage pair where h i,@xmath2. finally, we concatenate all the hidden representations of top-k passages {h 1 , . . . , h k } as input to the decoder. reader decoder. our approach mainly differs from fid reader in the decoder module by adding a pointer network. specifically, at each decoding step t, let e @xmath3, and denote s l @xmath4, then the probability of generation is given as follows, next, let v denote the vocabulary containing words for the generative model and |v| be the size of the vocabulary. then at step t, the probability distribution of words generation over the vocabulary is computed as, benefiting from the encoder-decoder attention layer in transformer architecture, we directly utilize the cross-attention score α l t of the last decoder layer l over the source tokens for the target token y t as copy distribution. then the probability of selecting y t in source sequence is calculated as, finally, put all the above together, the target token y t could both be generated from vocabulary with probability p gen , and copy from the source passages. the final prediction probability is defined as ## conclusion in this article, we propose a novel fid-pgn approach for the reader module of odqa under the standard retriever-reader framework. specifically, we integrate a pointer network into the fid reader to allow the model to directly select words from the retrieved passages. experimental results show that our model outperforms fid-kd on two benchmark datasets under the same setting, demonstrating the advantages of our method.