Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering (2021)

Abstract: In open-domain question answering (QA), retrieve-and-read mechanism has the inherent benefit of interpretability and the easiness of adding, removing, or editing knowledge compared to the parametric approaches of closed-book QA models. However, it is also known to suffer from its large storage footprint due to its document corpus and index. Here, we discuss several orthogonal strategies to drastically reduce the footprint of a retrieve-and-read open-domain QA system by up to 160x. Our results indicate that retrieve-and-read can be a viable option even in a highly constrained serving environment such as edge devices, as we show that it can achieve better accuracy than a purely parametric model with comparable docker-level system size.

## introduction open-domain question answering (qa) is the task of finding answers to generic factoid questions. in recent literature, the task is largely approached in two ways, namely retrieve & read and parametric. the former solves the problem by first retrieving documents relevant to the question from a large knowledge source and then reading the retrieved documents to find out the answer @xcite @xcite @xcite . the latter, also known as closed-book qa, generates the answer in a purely parametric end-to-end manner @xcite . while a parametric model enjoys the benefit in terms of system size that they do not require additional knowledge source like a retrieve & read system does, its fundamental limitations are that their predictions are not so interpretable and they are not suitable for dynamic knowledge source as ˚most of the work was done while the author was working at naver corp. end-to-end qa accuracy (em) dpr §2.1 passage filtering §2.2.1 128d embedding §2.2.1 mobilebert §2.2.2 retriever encoder sharing §2.2.3 unified retriever-reader through kd §2.2.4 iterative finetuning it is difficult to add, remove, or edit knowledge in the parametric model. these limitations are welladdressed by the retrieve & read mechanism, which makes it often more suitable for real-world products. however, it is known to suffer from its large storage footprint due to its document corpus and index, especially compared to the parametric model that only needs to store the parameters @xcite @xcite . building an interpretable and flexible opendomain qa system and reducing its system size are both important in real-world scenarios; the system must be able to quickly adapt to the changes of the world and be deployed in a highly constrained serving environment such as edge devices. hence, to get the best of both worlds, it is worthwhile to explore the trade-off between the storage budget and the accuracy of a retrieve & read system. well-known approaches for reducing the size of a neural network include pruning @xcite , quantization @xcite , and knowledge distillation @xcite . in this paper, we utilize some of these generic approaches and combine them with problem-specific techniques to size down a conventional retrieve & read system. we first train a passage filter and use it to reduce the corpus size (section 2.1). we further apply parameter sharing strategies and knowledge distillation to make a single-encoder lightweight model that can perform both retrieval and reading (section 2.2). in addition, we adopt multiple engineering tricks to make the whole system even smaller (section 2.3). we verify the effectiveness of our methods on the dev set and test set of efficientqa foot_0 @xcite . by applying our strategies to a recent extractive retrieve & read system, dpr @xcite , we reduce its size by 160x with little loss of accuracy, which is still higher than the performance of a purely parametric t5 @xcite baseline with a comparable docker-level storage footprint. in appendix a.5, we also report the performance on two more open-domain qa datasets, natural questions @xcite and triviaqa @xcite , to test the generalizability of our methods and suggest a future research direction. ## method in this section, we discuss three techniques for reducing the storage footprint of a generic retrieve & read system, namely passage filtering (section 2.1), unifying retriever and reader into a single model through parameter sharing (section 2.2), and posttraining compression (section 2.3). we assume that the initial system takes the conventional composition of a trainable (neural) retriever with a question encoder and a passage encoder that create dense vectors used for search, a neural extractive reader (possibly with passage ranking), and a text corpus and the corresponding index that serve as the knowledge source. figure 1 shows how we start from one such retrieve & read system and apply each of the methods, in the order they are introduced in this section, to successively reduce its system footprint without sacrificing much accuracy. ## experiments experimental setup we apply our storage reduction methods to a recent extractive retrieve & read system, dpr @xcite , which consists of three different bert-base encoders: question encoder of the retriever, passage encoder of the retriever, and encoder of the reader with a ranker. all experiments are done on naver smart machine learning (nsml) platform @xcite . the training and evaluation details are in appendix a.1, a.2, and a.3. ## conclusion we discuss several orthogonal approaches to reduce the system footprint of a retrieve-and-read-based open-domain qa system. the methods together reduce the size of a reference system (dpr) by 160 times with an accuracy drop of 2.45% and 4% on efficientqa dev and test, respectively. we hope that the presented strategies and results can be helpful for designing future retrieve-and-read systems under a storage-constrained serving environment. nako sung, minkyu kim, hyunwoo jo, youngil yang, jingwoong kim, leonard lausen, youngkwan kim, gayoung lee, donghyun kwak, jung-woo ha, and sunghun kim. 2017. nsml: a machine learning platform that enables you to focus on your models. in neurips systems for ml workshop. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, rémi louf, morgan funtowicz, joe davison, sam shleifer, patrick von platen, clara ma, yacine jernite, julien plu, canwen xu, teven le scao, sylvain gugger, mariama drame, quentin lhoest, and alexander m. rush. 2019. huggingface's transformers: state-of-theart natural language processing. arxiv preprint arxiv:1910.03771. lee xiong, chenyan xiong, ye li, kwok-fung tang, jialin liu, paul bennett, junaid ahmed, and arnold overwijk. 2021. approximate nearest neighbor negative contrastive learning for dense text retrieval. in iclr. sohee yang and minjoon seo. 2020. is retriever merely an approximator of reader? arxiv preprint arxiv:2010.10999. ofir zafrir, guy boudoukh, peter izsak, and moshe wasserblat. 2019. q8bert: quantized 8bit bert. in neurips emc2 workshop.