Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? (2022)

Abstract: Despite their recent popularity and well-known advantages, dense retrievers still lag behind sparse methods such as BM25 in their ability to reliably match salient phrases and rare entities in the query and to generalize to out-of-domain data. It has been argued that this is an inherent limitation of dense models. We rebut this claim by introducing the Salient Phrase Aware Retriever (SPAR), a dense retriever with the lexical matching capacity of a sparse model. We show that a dense Lexical Model Λ can be trained to imitate a sparse one, and SPAR is built by augmenting a standard dense retriever with Λ. Empirically, SPAR shows superior performance on a range of tasks including five question answering datasets, MS MARCO passage retrieval, as well as the EntityQuestions and BEIR benchmarks for out-of-domain evaluation, exceeding the performance of state-of-the-art dense and sparse retrievers. The code and models of SPAR are available at: https://github.com/facebookresearch/dpr-scale/tree/main/spar

## introduction text retrieval is a crucial component for a wide range of knowledge-intensive nlp systems, such as open-domain question answering (odqa) models and search engines. recently, dense retrievers @xcite have gained popularity and demonstrated strong performance on a number of retrieval tasks. dense retrievers employ deep neural networks to learn continuous representations for the queries and documents, and perform retrieval in this dense embedding space using nearest neighbor search @xcite . compared to traditional sparse retrievers that rely on discrete bag-of-words representations, dense retrievers can derive more semantically expressive embeddings, thanks to its end-to-end learnability and powerful pre-trained encoders. this helps dense retrievers to overcome several inherent limitations of sparse systems such as vocabulary mismatch (where different words are used for the same meaning) and semantic mismatch (where the same word has multiple meanings). on the other hand, while existing dense retrievers excel at capturing semantics, they sometimes fail to match the salient phrases in the query. for example, @xcite show that dpr, unlike a sparse bm25 retriever @xcite , is unable to catch the salient phrase "thoros of myr" in the query "who plays thoros of myr in game of thrones?". in addition, dense retrievers struggle to generalize to out-of-domain test data compared to training-free sparse retrievers such as bm25. for instance, @xcite find that dpr performs poorly compared to bm25 on simple entity-centric questions, and @xcite introduce a new beir benchmark to evaluate the zero-shot generalization of retrieval models showing that bm25 outperforms dense retrievers on most tasks. with dense and sparse retrievers each having their own distinctive pros and cons, researchers have long aspired to develop retriever models that combine the strengths of both. this, however, has proven challenging as dense and sparse retrievers are supported by drastically different algorithms and data structures (inverted index @xcite for sparse and approximate nearest neighbor search @xcite for dense). most existing research towards this goal extends sparse retrievers with improved representations from neural models @xcite . these methods, nonetheless, still rely on exact matching on a bag of tokens, which arguably cannot fully leverage the representational power of the pre-trained encoders. the opposite route of building better dense retrievers with the strengths of sparse models is much less explored. in fact, there have been theoretical and empirical studies suggesting that such drawbacks of dense retrievers may be a result of in- random sentences teacher: bm25 + -top 10 46 -50 query encoder q passage encoder p λ training dense retriever (e.g. dpr) lexical model λ query q passage p q(q) p(p) μ ⋅ q λ (q) p λ (p) query emb. passage emb. the spar model: figure 1 : spar augments a dense retriever with a dense lexical model λ trained to imitate a sparse teacher retriever. λ is trained using random sentences as queries with positive and negative passages produced by the teacher. λ is then combined with a dense retriever via vector concatenation to form a salient-phrase aware retriever. we evaluate spar on five odqa datasets ( §5.1) as well as on the ms marco @xcite passage retrieval benchmark ( §5.2), and show that it outperforms existing dense and sparse retrievers. we also examine the out-of-domain generalization of spar showing strong zero-shot performance across datasets ( §5.3), including on the beir bechmark @xcite and a recently released dataset of entity-centric questions @xcite . in addition, we conduct analyses of λ showcasing its lexical matching capability ( §6). a few recent studies focus on the limitations of current dense retrievers. @xcite and @xcite study the generalization issue of dense retrievers in various aspects, such as the overlap between training and test data, compositional generalization and the performance on matching novel entities. @xcite introduce a new beir benchmark to evaluate the zeroshot generalization of retrieval models showing that bm25 outperforms dense retrievers on most tasks. a different line of research explores using multiple dense vectors as representations which achieves higher accuracy but is much slower @xcite . @xcite further propose a knowledge distillation method to train a standard dense retriever with similar performance of the multi-vector colbert model. more recently, @xcite create entityquestions, a synthetic dataset of entity-centric questions to highlight the failure of dense retrievers in matching key entities in the query. we evaluate the generalization of spar on beir and entityquestions in §5.3. hybrid retrievers directly combine sparse and dense retrievers, and have been the most commonly used approach to overcome the limitations of a dense or sparse retriever @xcite before this work. a hybrid system retrieves two separate sets of candidates using a dense and a sparse retriever and rerank them using the hybrid retrieval score. compared to a hybrid retriever, spar offers the same performance with a much simpler architecture. we compare spar with the hybrid models in more details in §4.3. ## preliminaries: dense retrieval in this work, we adopt dpr @xcite as our dense retriever architecture for learning the lexical model λ. we give a brief overview of dpr in this section and refer the readers to the original paper for more details. dpr is a bi-encoder model with a query encoder and a passage encoder, each a bert transformer @xcite , which encodes the queries and passages into d-dimensional vectors, respectively. passage vectors are generated offline and stored in an index built for vector similarity search using libraries such as faiss @xcite . the query embedding is computed at run time, which is used to look up the index for k passages whose vectors are the closest to the query representation using dot-product similarity. dpr is trained using a contrastive objective: given a query and a positive (relevant) passage, the model is trained to increase the similarity between the query and the positive passage while decreas-ing the similarity between the query and negative ones. it is hence important to have hard negatives (irrelevant passages that are likely confused with positive ones) for more effective training foot_1 . we employ the dpr implementation from @xcite , which supports efficient multi-node training as well as memory-mapped data loader, both important for the large-scale training of λ. for model training, we also adopt their validation metrics of mean reciprocal rank (mrr) on a surrogate corpus using one positive and one hard negative from each query in the development set. assuming a set of n dev queries, this creates a mini-index of 2n passages, where the mrr correlates well with full evaluation while being much faster. ## the spar model in this section, we present spar, our salient phrase aware dense retriever. as illustrated in figure 1 , the basic idea of spar is to first train a dense lexical model λ such that it produces similar predictions to a sparse teacher retriever. λ is then combined with a regular dense retriever via vector concatenation. although the high-level idea of spar is fairly straightforward, details of the model training process dictate its success in practice, and thus require a careful experimental study. to find the best configuration of spar, we conduct pilot experiments using the validation set of naturalquestions (nq, @xcite following the odqa setting @xcite . bm25 is adopted as the teacher model for training λ. below we describe the key results on how to successfully train λ ( §4.1) and on how to best leverage λ in spar ( §4.2). ## conclusion in this paper, we propose spar, a salient-phrase aware dense retriever, which can augment any dense retriever with the lexical matching capacity and out-of-domain generalization from a sparse retriever. this is achieved by training a dense lexical model λ to imitate the behavior of the teacher sparse retriever, the feasibility of which remained unknown until this work. we show that spar outperforms previous state-of-the-art dense and sparse retrievers, matching or even exceeding more complex hybrid systems, on various in-domain and outof-domain evaluation datasets. for future work we plan to explore if a dense retriever can be trained to learn lexical matching directly without relying on a teacher model. this way, we can avoid imitating the errors of the sparse retriever, and devise new ways of training dense retrievers that can potentially surpass hybrid models. moreover, there are several intriguing findings in this work that may warrant further study, such as why spar's acc@k improves relatively to the hybrid model as k increases, and why joint training is less effective than post-hoc vector concatenation.