F ast F i D : Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection (2024)

Abstract: Open Domain Question Answering (ODQA) has been advancing rapidly in recent times, driven by significant developments in dense passage retrieval and pretrained language models. State-of-the-art models typically incorporate the FiD framework, which is composed by a neural retriever alongside an encoder-decoder neural reader. In the answer generation process, the retriever will retrieve numerous passages (around 100 for instance), each of which is then individually encoded by the encoder. Subsequently, the decoder makes predictions based on these encoded passages. Nevertheless, this framework can be relatively time-consuming, particularly due to the extensive length of the gathered passages. To address this, we introduce FastFiD in this paper, a novel approach that executes sentence selection on the encoded passages. This aids in retaining valuable sentences while reducing the context length required for generating answers. Experiments on three commonly used datasets (Natural Questions, TriviaQA and ASQA) demonstrate that our method can enhance the inference speed by **2.3X-5.7X**, while simultaneously maintaining the model’s performance. Moreover, an in-depth analysis of the model’s attention reveals that the selected sentences indeed hold a substantial contribution towards the final answer. The codes are publicly available at https://github.com/thunlp/FastFiD.

## introduction open domain question answering(odqa) is a longstanding task in natural language processing that involves generating an answer solely based on a given question. recent advancements in this field have typically adopted the retriever-reader framework @xcite @xcite , which breaks down the task into two distinct stages. initially, a retriever retrieves a set of relevant passages from a high-quality collection of open domain documents, such as wikipedia. subsequently, a reader model generates an answer by considering the question and the retrieved passages. thanks to advancements in neural models, the retriever has transitioned from traditional search methods like tf-idf @xcite to dense passage retrieval @xcite , resulting in improved retrieval performance. furthermore, driven by the progress of pretrained language models (plms) @xcite @xcite , the reader has evolved from extracting answers from a single passage to generating answers from multiple passages @xcite . this approach enables the model to leverage information from various passages more effectively, thereby producing more accurate answers. a recently successful model is fuse-in-decoder (fid) @xcite , which utilizes dense passage retrieval and a generative reader based on t5 @xcite , an encoderdecoder model. fid is capable of encoding each retrieved passage independently and subsequently concatenating these encoded passages to form an extensive context. the concatenated context is then used by the decoder to generate a response. owing to its straightforward and extensible architecture, numerous subsequent works have introduced modifications based on this framework @xcite @xcite . however, as the decoder must generate a response based on all retrieved passages, it can be time-consuming to enhance performance through the retrieval of additional passages. moreover, in real-world scenarios, the latency in generating an answer is a significant factor. as larger language models continue to be developed and demonstrate superior performance, this issue may become more pronounced. to address this issue, we introduce fastfid, a novel approach that performs sentence selection post the encoder's output and maintains only the essential sentences as references for the decoder, thereby significantly reducing the inference time for each query. to demonstrate the effectiveness of our approach, we first carry out experiments to ascertain that the multi-task training, which involves sentence selection and answer generation, does not conflict with one another during the model's learning process. this is achieved by seamlessly incorporating a selection loss on the encoder outputs with a language modelling loss on answer generation, enabling the model to simultaneously handle both sentence selection and answer generation tasks. an in-depth analysis of the decoder's cross-attention reveals that tokens from the chosen sentences yield a higher average attention score compared to those unchosen. this finding provides compelling evidence that the selected sentences significantly contribute more to the model's predictions. guided by this insight, we execute a secondary training phase, obliging the model to solely anchor to the selected encoder outputs when making the final prediction. the experimental results obtained from two widely used odqa datasets, namely natural questions (nq) @xcite and trivi-aqa @xcite , along with a long-form qa dataset called asqa @xcite , demonstrate that fastfid can achieve performance metrics comparable to the original fid. notably, it can reduce the context length by up to 38x and accelerate the inference time by 2.3x-5.7x on different datasets. to validate the effectiveness of sentence selection, we also compare its performance with passage reranking after the encoder outputs. the results show that sentence selection yields better performance while maintaining a similar context length. this comparison indicates that sentence selection is a more effective strategy for compressing information across multiple passages. in summary, our contributions can be encapsulated within the following three key points: ## related work open domain question answering serves a crucial role in natural language processing, with its primary function being to respond to factoid questions. followed by @xcite , current odqa systems usually use a large collection of documents like wikipedia as the knowledge source to answer questions. since the document collection usually contains millions of documents, the system always adds a retriever to retrieve some most relevant passages for the reader to make predictions. to get better retriever performance, @xcite proposed a shift from sparse retrieval systems like tf-idf to dense retrieval to enhance the efficiency of the retriever. subsequent research @xcite @xcite has investigated the use of end-to-end training methodologies to further boost the performance of the retriever, bypassing the need for pair-wise questiondocument data. izacard and grave (2021a) demonstrated an improvement in performance through the distillation of knowledge from the reader to the retriever. the idea of pretraining both the retriever and the reader on a vast, unlabeled corpus has been explored by @xcite and @xcite . a different research trajectory has aimed to augment the reader's capacity to better utilize retrieved passages. with the advancement of plms, the reader has evolved from rnn-based models @xcite to bert-based extractive readers @xcite and t5 or bart-based generative readers @xcite . recent studies @xcite @xcite have pivoted towards a hybrid approach, exploring the integration of both generative and extractive readers to further enhance system performance. efficient odqa the majority of contemporary open-domain question answering (odqa) systems face efficiency challenges, primarily due to the large-scale document processing and the use of sizable pre-trained language models. these efficiency challenges arise in two stages. the first stage is retrieval efficiency. given the potentially massive number of passages, dense retrieval can be extremely slow. instead of relying solely on brute force search methods, alternative algorithms such as approximate nearest neighbor (ann) @xcite and hierarchical navigable small world (hnsw) @xcite ) can be employed to expedite the retrieval process. the second efficiency challenge lies in the reading process, which involves handling multiple passages for each query. to address this, @xcite propose fid-light, which limits the decoder's attention to the first k tokens of each passage to reduce the context length. @xcite explores reducing the number of cross attention layers in fid's decoder to increase efficiency, but this comes at the cost of re-pretraining the base model. other complementary strategies explore to identify and stop processing less relevant passages early on by utilizing adaptive computation @xcite or knowledge graph with gnn network @xcite . additionally, some research has focused on directly retrieving answers to questions without the need for passage processing @xcite @xcite , or using language models to generate answers directly by finetuning and fewshot prompting @xcite . answer sentence selection answer sentence selection (as2) is a long-standing task that has been extensively explored. dense neural networks (dnns) have been widely employed in this task @xcite . @xcite further advanced the field by utilizing transformer-based pre-trained language models (plms) to achieve better results. recent studies have investigated methods such as generating answer sentences @xcite and implementing complex ranking pipelines @xcite . unlike these approaches, our work aims to predict the exact answer span from retrieved passages, using answer sentence selection only for enhancing inference speed. ## methods in this section, we propose fastfid, which is based on fid @xcite to reduce its inference time and make it more efficient. fastfid contains a two-stage training procedure. initially, in the first stage, we introduce a multi-task training objective that allows for simultaneous training of sentence selection and answer generation (section 3.1). then, in the second stage, we use the model trained in the first stage as the base model and perform continuous training on generating answers with reference to the selected tokens. (section 3.2). finally, in the inference stage, the encoder transcodes each passage into context embed-dings and curates a selection of valuable sentences, which are then employed in the decoder generation process to expedite inference time (section 3.3). the overall framework is shown in figure 2 . ## further analysis in this section, we present additional experiments to demonstrate the effectiveness of our method. first, we compare our sentence selection method with the passage reranking method in section 5.1. second, we evaluate the performance of our method with varying numbers of selected sentences in section 5.2. third, we conduct an ablation study to verify the importance of our two-stage training approach in section 5.3. finally, we assess the effectiveness of our method on decoder-only models in section 5.4. ## conclusion in this paper, we present fastfid, a model based on the fid framework, designed to accelerate the inference process for odqa tasks. fastfid utilizes a two-stage training technique to enable the selection of valuable sentences and focus its predictions exclusively on these sentences. experimental results demonstrate that fastfid substantially improves inference speed while maintaining its original answer generation performance. and our ablation study confirms the effectiveness of the two-stage training approach, showing a decrease in final performance when any single training stage is omitted.