Bridging the Training-Inference Gap for Dense Phrase Retrieval (2022)

Abstract: Building dense retrievers requires a series of standard procedures, including training and validating neural models and creating indexes for efficient search. However, these procedures are often misaligned in that training objectives do not exactly reflect the retrieval scenario at inference time. In this paper, we explore how the gap between training and inference in dense retrieval can be reduced, focusing on dense phrase retrieval (Lee et al., 2021) where billions of representations are indexed at inference. Since validating every dense retriever with a large-scale index is practically infeasible, we propose an efficient way of validating dense retrievers using a small subset of the entire corpus. This allows us to validate various training strategies including unifying contrastive loss terms and using hard negatives for phrase retrieval, which largely reduces the training-inference discrepancy. As a result, we improve top-1 phrase retrieval accuracy by 2 3 points and top-20 passage retrieval accuracy by 2 4 points for open-domain question answering. Our work urges modeling dense retrievers with careful consideration of training and inference via efficient validation while advancing phrase retrieval as a general solution for dense retrieval.

## introduction dense retrieval aims to learn effective representations of queries and documents by making representations of relevant query-document pairs to be similar @xcite . with the success of dense passage retrieval for open-domain question answering (qa) @xcite , recent studies build an index for a finer granularity such as dense phrase retrieval @xcite , which largely improves the computational efficiency of open-domain qa by replacing the retriever-reader model @xcite with a retriever-only model @xcite . also, phrase retrieval provides a unifying solution for multi-granularity retrieval ranging from open-domain qa (formulated as retrieving phrases) to document retrieval @xcite , which makes it particularly attractive. building a dense retrieval system involves multiple steps (figure 1 ) including training a dual encoder ( §4), selecting the best model with validation ( §3), and constructing an index (often with filtering) for an efficient search ( §5). however, these components are somewhat loosely connected to each other. for example, model training is not directly optimizing the retrieval performance using the full corpus on which models should be evaluated. in this paper, we aim to minimize the gap between training and inference of dense retrievers to achieve better retrieval performance. however, developing a better dense retriever requires validation, which requires building large indexes from a full corpus (e.g., the entire wikipedia for open-domain qa) for inference with a huge amount of computational resources and time. to tackle this problem, we first propose an efficient way of validating dense retrievers without building large-scale indexes. analysis of using a smaller random corpus with different sizes for the validation reveals that the accuracy from small indexes does not necessarily correlate well with the retrieval accuracy on the full index. as an alternative, we construct a compact corpus using a pre-trained dense retriever so that validation on this corpus better correlates well with the retrieval on the full scale while keeping the size of the corpus as small as possible to perform efficient validation. with our efficient validation, we revisit the training method of dense phrase retrieval @xcite , a general framework for retrieving different granularities of texts such as phrases, passages, and documents. we reduce the training-inference @xcite and (b) proposed procedure for densephrases training (top) and validation (bottom). we unify training loss terms l inp and l inb that enforce the representation of a question (q) similar to the representation of a positive phrase (p + ) while contrasting from representations of in-passage negative phrases (p - inp ) and in-batch negative phrases (p - inb ) respectively into a single term l train and expand negatives in number and difficulty with hard negatives (p - hard ). also, we use a retrieval accuracy on the development set q dev using a smaller corpus instead of the full corpus as an efficient validation metric for selecting the best checkpoint. query-side fine-tuning and token filtering are not described in this overview figure. lastly, we study the effect of a representation filter @xcite , an essential component for efficient search. we separate the training and validation of a phrase filtering module to disentangle the effect of contrastive learning and representation filtering. this allows us to do careful validation of the representation filter and achieve a better precision/recall trade-off. interestingly, we find that a representation filter has a dual role in reducing the index size and also improving retrieval accuracy, meaning smaller indexes are often better than larger ones in terms of accuracy. this gives a different view of other filtering methods that have been applied in previous studies for efficient open-domain qa @xcite @xcite . we reemphasize that phrase retrieval is an attractive solution for open-domain question answering compared to other retriever-reader models, considering both accuracy and efficiency. our contributions are summarized as follows: ## related work dense retrieval retrieving relevant documents for a query (mitra and craswell, 2018) is crucial in many nlp applications like open-domain question answering and knowledge-intensive tasks @xcite . dense retrievers typically build a search index for all documents by pre-computing the dense representations of documents using an encoder. off-the-shelf libraries for a maximum inner product search (mips) @xcite enable model training and indexing to be developed independently @xcite . however, both training dense retrievers and building indexes should take into account the final retrieval accuracy. in this respect, we aim to close the gap between training and inference of dense retrievers. phrase retrieval phrase retrieval @xcite directly finds an answer with mips from an index of contextualized phrase vectors. this removes the need to run an expensive reader for open-domain qa. as a result, phrase retrieval allows real-time search tens of times faster than retriever-reader approaches as an alternative for open-domain qa. densephrases @xcite removes the requirement of sparse features and significantly improves the accuracy from previous phrase retrieval methods @xcite . @xcite show how retrieving phrases could be translated into retrieving larger units of text like a sentence, passage, or document, making phrase retrieval a general framework for retrieval. despite these advantages, phrase retrieval requires building a large index from billions of representations. in this work, we focus on improving phrase retrieval with more efficient validation. ## efficient validation of phrase retrieval our goal is to train a dense retriever m that can accurately find a correct answer in the entire corpus c (in our case, wikipedia). careful validation is necessary to confirm whether new training methods are truly effective. it also helps find optimal configurations induced by those techniques. however, building a large-scale index for every model makes the model development process slow and also requires huge memory. thus, an efficient validation method could expedite modeling innovations in the correct direction. it could also allow frequent comparison of different checkpoints when updating a full index simultaneously during the training is computationally infeasible. 2 measuring the retrieval accuracy on an index from a smaller subset of the full corpus (denoted as c ⋆ ) for model validation would be a practical choice, hoping argmax @xmath0, c ⋆ ) ≈ argmax @xmath1, c) where ω is a set of model candidates and acc means the retrieval accuracy on a qa dataset d. we first examine how a relative order of accuracy between modeling ap-proaches may change with varying sizes of the random subcorpus ( §3.1) and then develop a clever way to construct a compact subcorpus that maintains reasonable correlation with the retrieval accuracy in the full scale ( §3.2). ## optimized training of densephrases in this section, we briefly review the original training method of densephrases ( §4.1) and improve it further to reduce the gap between training and retrieval in inference by modifying the training objective ( §4.2) and introducing additional training with hard negatives ( §4.3). ## token filtering representation filtering is often applied in practice to reduce the index size for efficient search @xcite . for phrase retrieval, tokens that are not likely to be a start/end position of an answer are filtered out using a trained filter classifier based on a logit score for each token to reduce an index size without losing much accuracy. only tokens with a score larger than a specific threshold are kept. after the filtering, the index is compressed using optimized product quantization @xcite . ## experiments to show the effectiveness of our proposed method, we evaluate densephrases models on open-domain qa benchmarks following the experimental setup of lee et al. (2021a,b). 7 datasets we measure phrase retrieval accuracy and passage retrieval accuracy on two open-domain qa datasets following the standard train/dev/test splits: natural questions @xcite and triviaqa @xcite . we first train our phrase retrieval models on natural questions (densephrases ♡ ) or on multiple reading comprehension datasets (densephrases ♠ ), namely natural questions, webquestions @xcite , curatedtrec @xcite , trivi-aqa, and squad @xcite . then each model is further query-side fine-tuned on natural questions and triviaqa. we build the phrase index with smaller subsets of corpora (r r or h k ) for validation and use the 2018-12-20 wikipedia snapshot (c) for the final inference. training details we use the same training hyperparameters of the original densephrases except for the batch size @xmath2. we set the number of training epochs to 2 with the generated questionanswer pairs and the number of training epochs to 10 with the standard reading comprehension dataset for more careful validation. we set λ @xmath3. we set @xmath4. token filtering our token filter achieves an improved auc-pr value over the filter from the original densephrases model (e.g., 0.348 vs. 0.307). we use a filter threshold of -3.0 for the index with the full corpus. this threshold reduces the index size by more than 70% of the original size. ## conclusion in this study, we aim to bridge the gap between training and inference of phrase retrieval. we first develop an efficient validation metric that measures retrieval accuracy on the index from a small corpus with hard passages using a pre-trained retriever. based on this validation, we show that the improvements in dense phrase retrieval training with unified loss and hard negatives are effective. as a result, we achieve state-of-the-art phrase retrieval and pas-sage retrieval accuracy in open-domain question answering among retrieval-only approaches. our work demonstrates that thorough validation is crucial for the accurate and efficient development of phrase retrieval with a large corpus. also, we prove that modeling and training methods should be designed closely to retrieval in inference time. despite its remarkable efficiency and flexibility, phrase retrieval has been relatively less studied than passage retrieval. we believe that our work can encourage more study on phrase retrieval with an efficient development cycle. furthermore, we hope our findings could be extended to dense retrieval in general to help a wide variety of applications. moreover, it could be especially beneficial in real applications where the corpus size is much larger than benchmark datasets.