Boosted Dense Retriever (2022)

Abstract: We propose DrBoost, a dense retrieval ensemble inspired by boosting. DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on retrieval mistakes made by the current ensemble. The final representation is the concatenation of the output vectors of all the component models, making it a drop-in replacement for standard dense retrievers at test time. DrBoost enjoys several advantages compared to standard dense retrieval models. It produces representations which are 4x more compact, while delivering comparable retrieval results. It also performs surprisingly well under approximate search with coarse quantization, reducing latency and bandwidth needs by another 4x. In practice, this can make the difference between serving indices from disk versus from memory, paving the way for much cheaper deployments.

## introduction identifying a small number of relevant documents from a large corpus to a given query, information retrieval is not only an important task in-and-of itself, but also plays a vital role in supporting a variety of knowledge-intensive nlp tasks @xcite , such as opendomain question answering (odqa, @xcite and fact checking @xcite . while traditional retrieval methods, such as tf-idf and bm25 @xcite , are built on sparse representations of queries and documents, dense retrieval approaches have shown superior performance recently on a range of retrieval and ranking tasks @xcite @xcite . dense retrieval involves embedding queries and documents as lowdimensional, continuous vectors, such that query and document embeddings are similar when the document is relevant to the query. the embedding function leverages the representational power of pretrained language models and is further finetuned using any available training query-document pairs. document representations are computed offline in an index allowing dense retrieval to scale to millions of documents, with query embeddings being computed on the fly. when deploying dense retrievers in real-world settings, however, there are two practical concerns: the size of the index and the retrieval time latency. the index size is largely determined by the number of documents in the collection, as well as the embedding dimension. whilst we cannot generally control the former, reducing the embedding size is an attractive way to reduce index size. on lowering latency, approximate nearest-neighbor (ann) or maximum inner product search (mips) techniques are required in practice. this implies that it is far more important for retrieval models to perform well under approximate search rather than in the exact search setting. developing a dense retrieval model that produces more compact embeddings and are more amenable to approximate search is thus the focus of this research. in this paper, we propose drboost, an ensemble method for learning a dense retriever, inspired by boosting @xcite . drboost attempts to incrementally build compact representations at training time. it consists of multiple component dense retrieval models ("weak learners" in boosting terminology), where each component is a bert-based bi-encoder, producing vector embeddings of the query and document. these component embeddings are in lower dimensions (e.g., 32 vs. 768) compared to those of regular bert encoders. the final relevance function is implemented as a linear combination of inner products of embeddings produced by each weak learner. this can be efficiently calculated by concatenating vectors from each component and then performing a single mips search, which makes drboost a drop-in replacement for standard dense retrievers at test time. component models are trained and added to the ensemble sequentially. each model is trained as a reranker over negative examples sampled by the current ensemble and can be seen as specializing on retrieval mistakes made previously. for example, early components focus on high-level topical information, whereas later components can capture finer-grained tail phenomena. through this mechanism, individual components are disentangled and redundancy minimized, leading to more compact representations. there are a couple of noticeable differences in training drboost when compared to existing dense retrieval models. although iterative training using negatives sampled by models learned in the previous rounds has been proposed @xcite @xcite , existing methods keep only the final model. in contrast, each weak learner in drboost is preserved and added to the ensemble. the construction of the embedding also differs. dr-boost can be viewed as a method of slowly "growing" overall dense vector representations, lending some structure to otherwise de-localized representations, while existing retrieval models encode queries and documents in one step. more importantly, drboost enjoys several advantages in real-world settings. because each weak learner in drboost produces very low-dimensional embeddings to avoid overfitting (32-dim in our experiments), many components can be added whilst the index stays small. our experiments demonstrate that drboost produces very compact embeddings overall, achieving accuracy on par with a comparable non-boosting baseline with 4-5x smaller vectors, and strongly outperforming a dimensionally-matched variant. probing dr-boost's embeddings using a novel technique, we also show that the embeddings can be used to recover more topical information from wikipedia than a dimensionally-matched baseline. empirically, drboost performs superbly when using approximate fast mips. with a k-mean inverted file index (ivf), the simple and widely used approach, especially in hierarchical indices and web-scale settings @xcite @xcite , drboost greatly outperforms the baseline dpr model @xcite ) by 3-10 points. alternatively, it can reduce bandwidth and latency requirements by 4-64x while retaining accuracy. in principle, this allows for the approximate index to be served ondisk rather than in expensive and limited ram (which is typically 25x faster), making it feasible to deploy dense retrieval systems more cheaply and at much larger scale. we also show that dr-boost's index is amenable to compression, and can be compressed to 800mb, 2.5x smaller than a recent state-of-the-art efficient retriever, whilst being more accurate @xcite . ## dense retrieval we give here the background of dense retrieval and boosting, as well as our proposed method. more extensive related work can be found in §5.5. dense retrieval involves learning a scalable relevance function h(q, c) which takes high values for passages c that are relevant for question q, and low otherwise. in the popular dense bi-encoder framework, h(q, c) is implemented as the dot product between q and c, dense vector representations of passages and questions respectively, produced by a pair of neural network encoders, e q and e c , ## discussion in this work we have explored boosting in the context of dense retrieval, inspired by the similarity of iteratively-sampling negatives to boosting. we find that our simple boosting approach, drboost, performs largely on par with a 768-dimensional dpr baseline, but produces more compact vectors, and is more amenable to approximate search. we note that drboost requires maintaining more neural models at test time, which may put a greater demand on gpu resources. however, the models can be run in parallel if latency is a concern, and if needed, these models can be distilled into a single model with little drop in accuracy. we hope that future work will build on boosting approaches for dense retrieval, including adding adaptive weights, and investigating alternative losses and sampling techniques. we also suggest that emphasis in dense retrieval should be placed on more holistic evaluation than just exact retrieval accuracy, demonstrating that models with quite similar exact retrieval can perform very differently under practically-important approximate search settings. and run our experiments on 16 v100 gpus. for all training rounds, we used the same set of training hyperparameters -we set learning rate as 3e-5, dropout as 0.1, weight decay as 0.01, batch size as 2 (per gpu) and max training steps as 30k. the maximum question and passage lengths are set as 40 and 200 respectively and we accompany each question with 50 passages during training. using our training infrastructure, the first round of the training takes about 8 hours and each additional training round takes about 1.5 hours until convergence. we always use the dev loss for model selection.