You Only Need One Model for Open-domain Question Answering (2022)

Abstract: Recent approaches to Open-domain Question Answering refer to an external knowledge base using a retriever model, optionally rerank passages with a separate reranker model and generate an answer using another reader model. Despite performing related tasks, the models have separate parameters and are weakly-coupled during training. We propose casting the retriever and the reranker as internal passage-wise attention mechanisms applied sequentially within the transformer architecture and feeding computed representations to the reader, with the hidden representations progressively refined at each stage. This allows us to use a single question answering model trained end-to-end, which is a more efficient use of model capacity and also leads to better gradient flow. We present a pre-training method to effectively train this architecture and evaluate our model on the Natural Questions and TriviaQA open datasets. For a fixed parameter budget, our model outperforms the previous state-of-the-art model by 1.0 and 0.7 exact match scores.

## introduction open-domain question answering (open qa) is a knowledge-intensive task that finds the answer for the given question from a large-scale knowledge corpus that can easily surpass millions of documents. thus, how to store and refer to the knowledge at such scales is important in terms of both performance and scalability for open qa systems. traditional systems rely on information retrieval engines such as lucene. these score the relevance of knowledge to a given query by lexical overlaps between them in a sparse representation space based on tf-idf or bm25 @xcite @xcite . however, recent advances in neural language modeling have enabled two new * work done while at samsung research lines of approach; 1) referring to internal knowledge parameterized in the model @xcite @xcite , and 2) referring to external knowledge retrieved by matching query and knowledge in dense representation spaces @xcite @xcite @xcite . despite the simplicity of the approach, parametric models have limitations such as a large number of model parameters that require large compute for both training and inference and nonexpandable knowledge without re-training. their implicit knowledge reference also makes it hard to find supporting knowledge and often results in hallucinations @xcite . the current dense retrieval models have advantages over parametric models on these issues @xcite @xcite . but most retrieval models only have a weak coupling between and separate parameters for the reader, reranker (if any), and retriever that limits these models from achieving optimal end-to-end training and efficient use of the total model capacity. in this paper, we propose a single language model yono (you only need one model) that can refer to external knowledge via its internal attention functions, which are trainable in a fully end-to-end manner. we achieve this by generalizing the retrieval and reranking as internal passagewise attentions. at the lower retrieval layers, the query and passages are separately encoded allowing pre-computation of all the passage representations. then passage-wise hard attention is applied to retrieve initial relevant passages from the entire knowledge base. while it would be optimal to retrieve passages based on cross-attention between the query and all the passages @xcite , it is computationally intractable. hence, we approximate this attention by a passagewise hard-attention layer using decoupled query and passage representations. the representations of the initial relevant passages are further encoded jointly with the query representation to compute more expressive coupled representations. these are used to select only the more relevant passages using another passage-wise hard-attention in the reranking layer. the representations of the final set of passages are then encoded by transformer encoders for deeper representations that are fused in the decoder to generate the answers. we train this architecture fully end-to-end by self-supervised pre-training and weakly supervised fine-tuning without passage labels. ## yono architecture as depicted in figure 1 , we propose a single encoder-decoder language model architecture consisting of 3 components: retrieval layer, reranking layer, and reading layer. ## conclusion in this paper, we propose a novel language model architecture that embeds the retriever and the reranker as internal passage-wise attention mechanisms and a training method to effectively train this model. this singular model architecture efficiently uses model capacity by cascading and sharing the representations from retriever to reranker to the reader leading to better gradient flow for end-toend training. we evaluate our model on natural questions and triviaqa open datasets and for a fixed parameter budget, our model outperforms the previous state-of-the-art model by 1.0 and 0.7 exact match scores. we show detailed ablations and analyses of each component of our approach. our future work is to conduct more experiments on various knowledge-intensive tasks and extend this model to match query and passage in multiple or hierarchical representation spaces.