A Survey for Efficient Open Domain Question Answering (2023)

Abstract: Open domain question answering (ODQA) is a longstanding task aimed at answering factual questions from a large knowledge corpus without any explicit evidence in natural language processing (NLP). Recent works have predominantly focused on improving the answering accuracy and have achieved promising progress. However, higher accuracy often requires more memory consumption and inference latency, which might not necessarily be efficient enough for direct deployment in the real world. Thus, a trade-off between accuracy, memory consumption and processing speed is pursued. In this paper, we will survey recent advancements in the efficiency of ODQA models and conclude core techniques for achieving efficiency. Additionally, we will provide a quantitative analysis of memory cost, query speed, accuracy, and overall performance comparison. Our goal is to keep scholars informed of the latest advancements and open challenges in ODQA efficiency research and contribute to the further development of ODQA efficiency.

## introduction open domain question answering @xcite is a longstanding task in natural language processing that can answer factoid questions, from a large knowledge corpus such as wikipedia @xcite or bookcorpus @xcite . traditional qa models rely on explicit evidence texts to locate the answer @xcite @xcite , while odqa models require the processing of large amounts of knowledge quickly to answer input questions. and compared to search engines, odqa models aim to enhance user-friendliness by presenting the final answer to a question directly, rather than returning a list of relevant snippets or hyperlinks @xcite . however, most general-purpose odqa models are computationally intensive, slow to infer, and expensive to train. one of the reasons is the huge index/document size. for example, @xcite processed an english wikipedia corpus including 26 million articles and built a dense index with a size of 65gb. besides, the majority of general-purpose odqa models are developed with large pre-trained language models, which often contain millions of parameters. for instance, the state-of-the-art odqa models on the natural question dataset, r2-d2 @xcite and unitedqa @xcite have 1.29 billion and 2.09 billion model parameters, respectively. storing the corpus index and pre-trained language models is memory-intensive @xcite while evidence retrieving and reading are memory and time consuming. these make general-purpose odqa models a big challenge for real-time use @xcite , such as on a mobile phone. towards this challenge, there are various tradeoffs in building odqa models that meet real-world application needs, such as the trade-offs among accuracy, memory consumption, inference speed, and so on @xcite @xcite . neurips 2020 organized an efficien-tqa competition @xcite , aiming to build odqa systems that can predict correct answers while also satisfying strict on-disk memory budgets. for this purpose, a line of work focused on building more efficient protocols. besides retriever-reader, retriever-only @xcite , generator-only @xcite are newly proposed protocols. see fig. 1 for more details. various efficiency techniques are also developed, such as index downsizing @xcite , fast searching @xcite , evidence retrieval or reading omitting @xcite @xcite and model size reducing @xcite etc. in this survey, we provide a comprehensive introduction to the broad range of methods that aim to improve efficiency with a focus on the odqa task. in section 2, we overview general-purpose odqa models and discuss their strategies and limitations in terms of efficiency. in section 3, we first walk through the key odqa models which concentrate on efficiency, then conclude the core techniques used. section 4 gives a quantitative analysis with an overall comparison of different frameworks and three specific aspects, i.e., memory cost, processing speed, and accuracy. finally, in section 5, we discuss the challenges reminded followed by the conclusion given in section 6. foot_0 2 overview of odqa models in this section, we summarize odqa models into three typical frameworks (see in fig. 1 ): retriever-reader, retriever-only, and generator-only. as described in section 1, retriever-reader models include two modules: a retriever and a reader. for retrievers, traditional non-neural methods, such as tf-idf @xcite and bm25 @xcite , use sparse representations to measure term matching between questions and passages. however, these approaches can only capture lexical information, limiting capabilities in matching questions and passages @xcite . differently, recent neural network-based dual-encoder retrievers @xcite encode questions and documents into a latent dense vector space where text semantics beyond terms can be adequately learned and measured. for readers, considering the way of obtaining answers, there exist two categories: extractive readers and generative readers. extractive readers normally answer the question using a span from the context and the goal is to classify the start and end positions of the answer in the retrieved evidence @xcite . and generative readers are not restricted to the input context and freely generate answers by autoregressively predicting tokens @xcite . distinctively, retriever-only models only use one retriever to extract answers directly from a phrase or qa-pair knowledge base. and generator-only models directly generate answers with the question, not involving evidence retrieval and reading @xcite . retriever-reader odqa methods generally obtain good performance. however, due to dense encoding for corpus passages and longer evidence for answer reasoning, they normally suffer from a larger index size and a slower processing speed. in addition, the dual-encoder retrievers like dpr, encoding for questions and documents independently, ignored interaction between them and limited the retrieval performance @xcite . in retriever-only odqa models, the omission of the reading/generating step greatly improves the speed of answering questions. but there are a few limitations for retriever-only odqa models: (1) lower performance on average compared to retriever-reader odqa models since less information is considered during answer inference; ## efficient odqa models and techniques in this section, we first walk through the key odqa models which concentrate on efficiency, and discuss their strengths and weaknesses as well as their unique characteristics in section 3.1. then we conclude the core techniques used in these models for improving the efficiency of odqa, from data and model perspectives, respectively, in section 3.2. before we start, we first take dpr on the natural questions (nq) test dataset as an example to show the time each module needs during inference and their detailed memory costs in fig. 2 . we can see the total processing time dpr needs is 0.91 seconds (s) 2 where the inference speed is mainly affected by evidence searching (74.79%) and reading (23.95%). the total memory cost of dpr is 79.32gb which is huge. the index takes up 81.95% of the memory, the raw corpus takes 16.39% space, and the remaining 1.66% are for the models where the retriever model is around twice the size of the reader model. based on these observations, how to improve the efficiency of odqa models focuses on the reduction of processing time and memory cost. to reduce processing time, we can accelerate evidence searching and reading. to reduce the memory cost, we can reduce the size of the index and model. besides, some emerging directions are also proposed, such as jumping the retrieval part to generate answers using questions directly or retrieving answers directly to omit evidence reading. we introduce the details below. ## quantitative analysis this section gives a quantitative analysis of the aforementioned odqa models. we first give an overall comparison of different frameworks and further discuss the methods quantitatively from three specific aspects: memory cost, processing speed, and accuracy foot_1 . at the end of the analysis, the following subsection summarizes and concludes what has been analyzed and discussed. ## discussion in this section, we summarize and illustrate the insights and future directions from the following aspects. we first summarize the key points to improve the effectiveness of odqa systems, from the two aspects of index and model respectively. in terms of index size, it is worth exploring deeper on generative models and the techniques of compacting embedding. in terms of model size, knowledge distillation is a promising direction to reduce model size while another direction is the application of lightweight models. in addition, one-stage odqa models are also worthy of research. additionally, we provide some advice on model recommendations under different requirements. for example, if we pursue real-time feedback, retriever-only systems should be good choices; if we are limited by computing resources, generator-only systems are suitable candidates; and if we need to trade off performance, memory cost and processing time, retriever-reader systems are relatively more appropriate. in general, for researchers who are interested in improving the state-of-the-art efficiency methods on odqa tasks, this survey can serve as an entry point to find opportunities for new research directions. however, some salient challenges need to be addressed in the way of odqa efficiency research. one of the worrisome things is that most odqa approaches are computation-heavy and energyexpensive. how can the odqa system be deployed in low-power devices with limited computing resources and mobile devices is still very challenging. another thing is that it seems to be inadequate to evaluate the efficiency of odqa models only on accuracy, memory, and processing time, due to many other factors that should be considered and traded off. for example, it is also important to establish what resource, e.g., money, time, and data for model training, power consumption, carbon emissions, etc. ## conclusion in this survey, we retrospected the typical literature according to three different frameworks of open domain question answering (odqa) systems. further, we provided a broad overview of existing methods to increase efficiency for odqa models and discussed their limitations. in addition, we performed a quantitative analysis in terms of efficiency and offered certain suggestions about method selections of open domain question answering. finally, we discussed possible open challenges and potential future directions of efficient odqa models.