Boosted Dense Retriever
AI { plewis,barlaso,xwhan,fabiopetroni,scottyih,sriedel } @fb.com
Abstract
We propose DrBoost, a dense retrieval ensemble inspired by boosting.
DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on retrieval mistakes made by the current ensemble.
The final representation is the concatenation of the output vectors of all the component models, making it a drop-in replacement for standard dense retrievers at test time.
DrBoost enjoys several advantages compared to standard dense retrieval models.
It produces representations which are 4x more compact, while delivering comparable retrieval results.
It also performs surprisingly well under approximate search with coarse quantization, reducing latency and bandwidth needs by another 4x.
In practice, this can make the difference between serving indices from disk versus from memory, paving the way for much cheaper deployments.
Introduction
Identifying a small number of relevant documents from a large corpus to a given query, information retrieval is not only an important task in-and-of itself, but also plays a vital role in supporting a variety of knowledge-intensive NLP tasks (Lewis et al., 2020; Petroni et al., 2021), such as opendomain Question Answering (ODQA, Voorhees and Tice, 2000; Chen et al., 2017) and Fact Checking (Thorne et al., 2018).
While traditional retrieval methods, such as TF-IDF and BM25 (Robertson, 2008), are built on sparse representations of queries and documents, dense retrieval approaches have shown superior performance recently on a range of retrieval and ranking tasks (Guu et al., 2020;
Karpukhin et al., 2020; Reimers and Gurevych, 2019; Hofst¨atter et al., 2021b).
Dense retrieval involves embedding queries and documents as lowdimensional, continuous vectors, such that query and document embeddings are similar when the document is relevant to the query.
The embedding function leverages the representational power of pretrained language models and is further finetuned using any available training query-document pairs.
Document representations are computed offline in an index allowing dense retrieval to scale to millions of documents, with query embeddings being computed on the fly.
When deploying dense retrievers in real-world settings, however, there are two practical concerns: the size of the index and the retrieval time latency .
The index size is largely determined by the number of documents in the collection, as well as the embedding dimension.
Whilst we cannot generally control the former, reducing the embedding size is an attractive way to reduce index size.
On lowering latency, Approximate Nearest-Neighbor (ANN) or Maximum Inner Product Search (MIPS) techniques are required in practice.
This implies that it is far more important for retrieval models to perform well under approximate search rather than in the exact search setting.
Developing a dense retrieval model that produces more compact embeddings and are more amenable to approximate search is thus the focus of this research.
In this paper, we propose DrBoost, an ensemble method for learning a dense retriever, inspired by boosting (Schapire, 1990; Freund and Schapire, 1997).
DrBoost attempts to incrementally build compact representations at training time .
It consists of multiple component dense retrieval models (“weak learners” in boosting terminology), where each component is a BERT-based bi-encoder, producing vector embeddings of the query and document.
These component embeddings are in lower dimensions (e.g., 32 vs. 768) compared to those of regular BERT encoders.
The final relevance function is implemented as a linear combination of inner products of embeddings produced by each weak learner.
This can be efficiently calculated by concatenating vectors from each component and then performing a single MIPS search, which makes DrBoost a drop-in replacement for standard dense retrievers at test time.
Component models are trained and added to the ensemble sequentially .
Each model is trained as a reranker over negative examples sampled by the current ensemble and can be seen as specializing on retrieval mistakes made previously.
For example, early components focus on high-level topical information, whereas later components can capture finer-grained tail phenomena.
Through this mechanism, individual components are disentangled and redundancy minimized, leading to more compact representations.
There are a couple of noticeable differences in training DrBoost when compared to existing dense retrieval models.
Although iterative training using negatives sampled by models learned in the previous rounds has been proposed (Xiong et al., 2020; Qu et al., 2021; O˘guz et al., 2021; Sachan et al., 2021, inter alia. ), existing methods keep only the final model.
In contrast, each weak learner in
DrBoost is preserved and added to the ensemble.
The construction of the embedding also differs.
DrBoost can be viewed as a method of slowly “growing” overall dense vector representations, lending some structure to otherwise de-localized representations, while existing retrieval models encode queries and documents in one step.
More importantly, DrBoost enjoys several advantages in real-world settings.
Because each weak learner in DrBoost produces very low-dimensional embeddings to avoid overfitting (32-dim in our experiments), many components can be added whilst the index stays small.
Our experiments demonstrate that DrBoost produces very compact embeddings overall, achieving accuracy on par with a comparable non-boosting baseline with 4– 5x smaller vectors, and strongly outperforming a dimensionally-matched variant.
Probing DrBoost’s embeddings using a novel technique, we also show that the embeddings can be used to recover more topical information from Wikipedia than a dimensionally-matched baseline.
Empirically, DrBoost performs superbly when using approximate fast MIPS.
With a K -mean inverted file index (IVF), the simple and widely used approach, especially in hierarchical indices and
Web-scale settings (J´egou et al., 2011; Johnson et al., 2019; Matsui et al., 2018), DrBoost greatly outperforms the baseline DPR model (Karpukhin et al., 2020) by 3–10 points.
Alternatively, it can reduce bandwidth and latency requirements by 4– 64x while retaining accuracy.
In principle, this allows for the approximate index to be served ondisk rather than in expensive and limited RAM (which is typically 25x faster), making it feasible to deploy dense retrieval systems more cheaply and at much larger scale.
We also show that DrBoost’s index is amenable to compression, and can be compressed to 800MB, 2.5x smaller than a recent state-of-the-art efficient retriever, whilst being more accurate (Yamada et al., 2021).
Dense Retrieval
We give here the background of dense retrieval and boosting, as well as our proposed method.
More extensive related work can be found in §5.5.
Dense Retrieval involves learning a scalable relevance function h ( q, c ) which takes high values for passages c that are relevant for question q , and low otherwise.
In the popular dense bi-encoder framework, h ( q, c ) is implemented as the dot product between q and c , dense vector representations of passages and questions respectively, produced by a pair of neural network encoders, E Q and E C ,
[Equation]
where q = E Q ( q ) and c = E C ( c ) .
At inference time, retrieval from a large corpus C = { c 1 , . .
. , c |C| } is accomplished by solving the following MIPS problem: c ∗ = arg max c ∈C q ⊤ c .
In standard settings, we assume access to a set of m gold question-passage pairs D
= { ( q i , c + i ) } mi =1 .
It is most common to learn models by training to score gold pairs higher than sampled negatives .
Negatives can be obtained in a variety of ways, such as by sampling at random from corpus C , or by using some kind of importance sampling function on retrieval results (see §2.1).
When augmented by n negatives per gold passagedocument pair, we have training data of the form (cid:101)
D
= { ( q i , c
+ i , c − i, 1 , . . .
c − i,n ) } mi =1 , which we use to train a model, e.g., using a ranking or margin objective, or in our case, by optimizing negative loglikelihood (NLL) of positive pairs
[Equation]
Require: D train , D dev , C ▷ Training Data and Corpus Require:
h▷ Initial Retrieval Model Require: τ ▷ Min.
Error Reduction Tolerance 1: r ←2: h ← h▷ Initialize Current Model with Initial Model 3: ϵ old ← ∞ 4: ϵ ← GetModelError ( D dev , C , h ) 5: while ( ϵ old − ϵ ) >
τ do 6: r ← r +7: (cid:101) D r train =
AddNegatives ( D train , C , h ) 8: (cid:101) D r dev =
AddNegatives ( D dev , C , h ) 9: h r = TrainDenseRetriever ( (cid:101)
D r train , (cid:101) D r dev ) 10: if Dense Retrieval w/ Iteratively-sampled Negs.
then 11: h ← h r 12: else if Boosted Dense Retrieval then 13: h ← CombineModels ( h, h r ) 14: end if 15: ϵ old ← ϵ 16: ϵ ← GetModelError ( (cid:101) D dev , C , h ) 17: end while 18: return h 2. Iterated Negatives for Dense Retrieval The choice of negatives is an important factor for what behaviour dense retrievers will learn. Simply using randomly-sampled negatives has been shown to perform poorly, because they are too easy for the model to discriminate.
Thus it is common to mix in some hard negatives along with random negatives, which are designed to be more challenging to distinguish from gold passages (Karpukhin et al., 2020).
Hard negatives are typically collected by retrieving passages using an untrained retriever, such as BM25, and filtering out any unintentional golds.
This ensures the hard negatives are at least topically-relevant.
Recently, it has become common practice to run a number of rounds of dense retrieval training to bootstrap hard negatives (Xiong et al., 2020; Qu et al., 2021; O˘guz et al., 2021; Sachan et al., 2021, inter alia.).
Here, we first train a dense retriever following the method we describe above, and then use this retriever to produce a new set of hard negatives.
This retriever is discarded, and a new one is trained from scratch, using the new, “harder” negatives.
This process can then be repeated until performance ceases to improve.
This approach, which we refer to dense retrieval with iteratively-sampled negatives is described in Algorithm 1.
2. Boosting
Boosting is a loose family of training algorithms for machine learning problems, based on the principle of gradually ensembling “weak learners” into a strong learner.
Boosting can be described by the following high-level formalism (Schapire, 2007).
For a task with a training set { ( x 1 , y 1 ) , · · · , ( x m , y m ) } , where ( x i , y i ) ∈
X × Y we want to learn a function h : X → Y , such that h ( x i ) =
ˆ y i
≈
y
i .
This is achieved using an iterative procedure over R steps:
• For round r , we construct an importance distribution D r over the training data, based on where error ϵ of our current model h is high • Learn a “weak learner” h r to minimize error ϵ r = (cid:80) i D r ( i ) L ( h r ( x i ) , y i ) for some loss function L measuring the discrepancy between predictions and real values.
• Combine h and h r to form a new, stronger overall model, e
g., by a linear combination
h new =
αh r + βh .
The iteration can now be repeated.
The initial importance distribution D 0 is usually assumed to be a uniform distribution, and h 0 models a constant function.
Note that how each additional model added to h is specifically designed to solve instances that h currently struggles with.
2. Boosted Dense Retrieval: DrBoost
We note similarities between the boosting formulation, and the dense retrieval with iterativelysampled negatives.
We can adapt a boostinginspired approach to dense retrieval with minimal changes, as shown in Algorithm 1.
Algorithmically, the only difference (lines 10–13) is that in the case of iterative negatives, the model h after r rounds is replaced by the new model h r , whereas in the boosting case, we combine h r and h .
In this paper, we view the boosted “weak learner” models h r as rerankers over the retrieval distribution from the current model h .
That is, when training dense boosted retrievers, we only train using hard negatives, and do not use any random or in-batch negatives.
Using the construction of negatives as a mechanism to define the importance distribution, each new model is directly trained to solve the retrieval mistakes that the current ensemble makes.
Each model h r is implemented as a bi-encoder, as in Eq.
We combine models as linear combinations:
[Equation]
The coefficients could be learnt from development data, or, simply by setting all coefficients to 1, which we find to be empirically effective.
The overall model after R rounds can be written as:
[Equation]
where [ . . .
] indicates vector concatenation.
Thus h can be computed as a single inner product, making it a drop-in replacement for standard MIPS dense retrievers at test time.
One downside of the boosting approach is that we must maintain R encoders for both passages and questions.
Since passages are embedded offline, this does not create additional computational burden on the passage side at test time.
However, on the query side, for a question q , boosted dense retrieval requires R forward passes to compute the full representation, one for each subvector q r .
While this step is fully parallelizable, it is still undesirable.
We can remedy this for low-latency, lowresource settings by distilling the question encoders of h into a single encoder, which can produce the overall question representations ¯ q directly.
Here, given the training dataset D train of gold questionpassage pairs, and a model h we want to distill, we first compute overall representations ¯ q and ¯ c for all pairs using h as distillation targets, then train a new question encoder E dist Q with parameters ϕ , by minimizing the objective:
[Equation]
Experiments 3. Datasets
Natural Questions (NQ)
We evaluate retrieval for downstream ODQA using the widely-used NQopen retrieval task (Kwiatkowski et al., 2019).
This requires retrieving Wikipedia passages which contain answers to questions mined from Google search logs.
We use the preprocessed and gold pairs prepared by Karpukhin et al.
(2020), and report recall-at-K (R@K) for K ∈ { 20 , 100 } .
MSMARCO
We evaluate in a Web-text setting using the widely-used passage retrieval task from
MSMARCO (Bajaj et al., 2016).
Queries consist of user search queries from Bing, with humanannotated gold relevant documents.
We use the preprocessed corpus, training and dev data (gold pairs and data splits) from O˘guz et al.
We follow the common practice of reporting the MeanReciprocal-Rank-at-10 (MRR@10) metric for the public development set.
3. Tasks
In this section, we will describe the experiments we perform, and the motivations behind them.
Exact Retrieval
We are interested in understanding whether the boosting approach results in superior performance for exhaustive (exact) retrieval.
Here, no quantization or approximations are made to MIPS, which results in large indices, and slow retrieval, but represents the upper bound of accuracy.
This setting is the most commonly-reported in the literature.
Approximate MIPS: IVF
Exact Retrieval does not evaluate how a model performs in practicallyrelevant settings.
As a result, we also evaluate in two approximate MIPS settings.
First, we consider approximate MIPS with an Inverted File Index (IVF, Sivic and Zisserman, 2003).
IVF works by first clustering the document embeddings offline using K -means (Lloyd, 1982) resulting K cluster centroids.
At test time, for a given query vector, rather than compute an inner product for each document in the index, we instead compute inner products to the K centroids.
We then visit the n probes highest scoring clusters, and compute inner products for only the documents in these clusters.
This technique increases the speed of search significantly, at the expense of some accuracy.
Increasing K , the number of centroids, increases speed, at the expense of accuracy, as does decreasing the value of n probes .
A model is preferable if retrieval accuracy remains high with very fast search, i.e., low n probes and high K 2 .
In our experiments we fit K = 65536 clusters and sweep over a range of values of n probes from 2to 2 15 .
Other methods such as HNSW (Malkov and
Yashunin, 2020) are also available for fast search, but are generally more complex and can increase index sizes significantly.
IVF is a particularly popular approach due it its simplicity, and as a first coarse quantizer in hierarchical indexing (Johnson et al., 2019), since it is straightforward to apply sharding to the clusters, and further search indices can be built for each cluster.
Approximate MIPS: PQ
Whilst IVF will increase search speeds, it does not reduce the size of the index, which may be important for scalability, latency and memory bandwidth considerations.
To investigate whether embeddings are amenable to compression, we experiment with applying Product
Quantization (PQ, J´egou et al., 2011).
PQ is a lossy quantization method that works by 1) splitting vectors into subvectors 2) clustering each subvector space and 3) representing vectors as a collection cluster assignment codes.
We apply PQ using 4dimensional sub-vectors and 256 clusters per subspace, leading to a compression factor of 16x over uncompressed float32 .
All MIPS retrieval is implemented using
FAISS (Johnson et al., 2019).
Generalization Tests
In addition to in-domain evaluation, we also perform two generalization tests.
These will determine whether the boosting approach is superior to iteratively-sampling negatives in out-of-distribution settings.
We evaluate
MSMARCO-trained models for zero-shot generalization using selected BEIR (Thakur et al., 2021) datasets that have binary relevance labels.
Namely, we test on the SciFact, FiQA, Quora and ArguAna subsets.
This will test how well models generalize to new textual domains and different query surface forms.
We also evaluate NQ-trained models on
EntityQuestions (Sciavolino et al., 2021), a dataset of simple entity-centric questions which has been recently shown to challenge dense retrievers.
This dataset uses the same Wikipedia index as NQ, and tests primarily for robustness and generalization to new entities at test time.
3. Models
We compare a model trained with iterativelysampled negatives to an analogous model trained with boosting, which we call DrBoost.
There are many dense retrieval training algorithms available which would be suitable for training with iteratively-sampled negatives and boosting with
DrBoost.
Broadly-speaking, any dense retriever could be used if utilizes negative sampling, and could be trained in Step 9 of Algorithm 1.
We choose Dense Passage Retriever (DPR, Karpukhin et al., 2020) with iteratively-sampled negatives due to its comparative simplicity and popularity.
Iteratively-sampled negatives baseline:
DPR
DPR follows the dense retrieval paradigm outlined in §2 It is trained with a combination of in-batch negatives, where gold passages for one question are treated as negatives for other questions in the batch (which efficiently simulates random negatives), and with hard negatives, sampled initially from BM25, and then from the previous round, as in Algorithm 1.
We broadly follow the DPR training set-up of O˘guz et al.
We train
BERT-base DPR models using the standarddimensions, as well as models which match the final dimension size of DrBoost.
We use parametersharing for the bi-encoders, and layer-norm after linear projection.
Models are trained to minimize the negative log-likelihood of positives, and the number of training rounds is decided using development data, as in Algorithm 1, using an initial hretriever BM25.
3.3. DrBoost Implementation
For our DrBoost version of DPR, we keep as many experimental settings the same as possible.
There are two exceptions, which are required for adapting dense retrieval to boosting.
The first is that each component “weak learner” model has a low embedding dimension.
This is to avoid overfitting, and to make sure the final index size is manageable.
We report using models of 32 dims (cf.
the standard 768 dims), but note that training with dimension as low as 8 is stable.
The second is that, as motivated in §2.3, we train each weak learner using only hard negatives, and no in-batch negatives.
In effect, this choice of negatives means that each model is essentially trained as a reranker.
3 DrBoost models are fit following Algorithm 1, and we stop adding models when the development set performance stops improving.
The initial retriever h 0 for DrBoost is a constant function, and thus the initial negatives for DrBoost are sampled at random from the corpus, unlike DPR, which uses initial hard negatives collected from BM25.
DrBoost α Coefficients
DrBoost combines weak learners as a linear combination.
We experiment with learning the α coefficients using development data, however this does not significantly improve results over simply setting them all to 1.
There- fore, for the sake of simplicity and efficiency, we report DrBoost numbers with all α = 1 . 0 .
Empirically, we find the magnitudes of embeddings for DrBoost’s component models to be similar.
In other words, one component does not dominate over others.
DrBoost Distillation
We experiment with distilling DrBoost ensembles into a single model for latency-sensitive applications using the L2 loss at the end of §2.3.
We distill a single BERT-base query encoder, and perform early stopping and model selection using development L2 loss.
Results 4. Exact Retrieval
Exact Retrieval results for MSMARCO and Natural Questions are shown in Table 1 in the “Exact Search” Column.
We find that our DrBoost version of DPR reaches peak accuracy after 5 or 6 rounds when using 32-dim weak learners (see §A.1), leading to overall test-time index of 160/192dim.
In terms of Exact Search, DrBoost outperforms the iteratively-sampled negatives DPR baseline on MSMARCO by 2.2%, and trails it by only 0.3% on NQ R@100, despite having a total dimension 4–5x smaller.
It also strongly outperforms a dimensionally-matched DPR, by 3% on MSMARCO, and 1% on NQ in R@100, demonstrating
DrBoost’s ability to learn high-quality, compact embeddings.
We also quote recent state-of-the-art results, which generally achieve stronger exact search results (AR2, Zhang et al., 2021).
Our emphasis, however, is on comparing iteratively-sampled negatives to boosting, and we note that state-ofthe-art approaches generally use larger models and more complex training strategies than the “inner loop” BERT-base DPR we report here.
Such strategies could also be incorporated into DrBoost if higher accuracy was desired, as DrBoost is largelyagnostic to the training algorithm used.
4. Approximate MIPS
Table 1 also shows how DPR and DrBoost behave under IVF MIPS search, which is shown graphically in Figure 1 as well.
We find that DrBoost dramatically outperforms DPR in IVF search, indicating that much faster search is possible with
DrBoost.
High-dimensional embeddings suffer under IVF due to the the curse of dimensionality, and thus compact embeddings are important.
Using search probes, DrBoost outperforms DPR by 10.5% on MSMARCO and 6.3% on NQ in R@100.
The dimensionally-matched DPR is stronger, but still trails DrBoost by about 4% using 8 probes.
The strongest exact search model is thus not necessarily the best in practical approximate MIPS settings.
For example, if we can tolerate a 10% relative drop in accuracy from the best performing system’s exact search, DrBoost requires 16 (4) probes for MSMARCO (NQ) to reach the required accuracy, whereas DPR will require 1024 (16), meaning
DrBoost can be operated approximately 64x (4x) faster.
The distilled DrBoost is also shown for NQ in
Table 1.
The precision (low R@K values) is essentially unaffected, (exact search drops by 0.1% for
R@20), but recall drops slightly (-0.7% R@100).
Interestingly, the distilled DrBoost performs even better under IVF search, improving over DrBoost by ∼ 1% at low numbers of probes.
Crucially, whilst the distilled DrBoost is only slightly better than the 192-dim DPR under exact search, it is 4– 5% stronger under IVF with 8 probes (alternatively, 8x faster for equivalent accuracy).
Aside from fast retrieval, small indices are also important for edge devices, or for scalability reasons.
While DrBoost can already produce high quality compact embeddings, Product Quantization
[Table] [Table]
can reduce this even further.
Table 2 shows that
DrBoost’s NQ index can be compressed from [-13.
-]13.
GB to 840MB with less than 1% drop in performance.
We compare to BPR (Yamada et al., 2021), a method specifically designed to learn small indices through binarization.
DrBoost’s PQ index is 2.4x smaller than the BPR index reported by
Yamada et al.
(2021), whilst being 2.4% more accurate (R@20).
A more aggressive quantization leads to a 420MB index — 4.8x smaller than BPR — whilst only being 1.2% less accurate.
Analysis 5. Qualitative Analysis
Since each round’s model is learned on the errors of the previous round, we expect each learner to “specialize” and learn complementary representations.
To see if this is qualitatively true, we look at the retrieved passages from each round’s retriever in isolation (Table 10 in §A.3).
Indeed, we find that each 32-dim sub-vector tackles the query from different angles.
For instance, for the query “who got the first nobel prize in physics?” , the first subvector captures general topical similarity based on keywords, retrieving passages related to the “Nobel
Prize” .
The second focuses mostly on the first paragraphs of articles of prominent historical personalities, presumably because these are highly likely to contain answers in general; and the third one retrieves from the pages of famous scientists and inventors.
The combined DrBoost model would favor passages in the intersection of these sets.
5. In-distribution generalization
Boosting algorithms are remarkably resistant to over-fitting, even when the classifier has sufficient capacity to achieve zero training error.
In their landmark paper, Bartlett et al.
(1998) show that this generalization property is a result of the following: the training margins increase with each iteration of boosting.
We empirically show the same to be true for DrBoost.
For a fixed query embedding, dense retrieval acts as a linear binary classifier, where the gold passage is positive and all other passages are negatives (Eq.
We adopt the classical definition of margin for linear classifiers to dense retrieval by defining a top- k margin:
[Equation]
Method
SciFact
FiQA
Quora
ArguAna NDCG@10 NDCG@10 NDCG@10 NDCG@
SotA Dense 64.
DPR (160 dim) 50.
42.5 DrBoost (160 dim) 49.
[Table]
where µ c is the average norm of passage embeddings and the operator max { k } returns the k -th maximum element in the set.
For a fixed q i and k = 1 , this definition is identical to the classical margin definition.
Figure 2 plots the 50 th , 75 th [-and
-]and 90 th percentiles of the top-20 margin for DrBoost on the NQ training set.
We clearly see that margins indeed increase at each step, especially for cases that the model is confident in (high margin).
We hypothesize this property to be the main reason for the strong in-distribution generalization of DrBoost that we observed, and potentially also for the surprisingly strong IVF results, since wide margins should intuitively make clustering easier as well.
5. Cross-domain generalization
It has been observed in previous work (Thakur et al., 2021) that dense retrievers still largely lag behind sparse retrievers in terms of generalization capabilities.
We are interested to test whether our method could be beneficial for out-of-domain transfer as well.
We show the results for zero-shot transfer on a subset of the BEIR benchmark in Tableand the EntityQuestions dataset in Table 4.
While
DrBoost improves slightly over the dimensionmatched baseline on EntityQuestions, where the passage corpora stays the same, it produces worse results on the BEIR datasets.
We conclude that boosting is not especially useful for cross-domain transfer, and should be combined with other methods if this is a concern.
5. Representation Probing
One of the hypothesis we formulate for the stronger performance of DrBoost over DPR is that the for-
[Figure]
mer might better capture topical information of passages and questions.
To test this, we collected topics for all Wikipedia articles in Natural Questions using the strategy of Johnson et al.
and associate them with both passages and questions.
We then probed both DPR and DrBoost representations with an SVM (Steinwart and Christmann, 2008) classifier considering a 5-fold cross-validation over 500 instances and 8 different seeds.
Results (in Figure 3) confirms our hypothesis: the topic classifier accuracy is higher with DrBoost representations with respect to DPR ones of the same dimension (i.e., 192), for both questions and passages.
5. Related Work Boosting for retrieval
Boosting has been studied in machine learning for over three decades (Kearns and Valiant, 1989; Schapire, 1990).
Models such as AdaBoost (Freund and Schapire, 1997) and GBMs (Friedman, 2001) became popular approaches to classification problems, with implementations such as XGBoost still popular today (Chen and Guestrin, 2016).
Many boosting approaches have been proposed for retrieval and learning-to-rank (LTR) problems, typically employing decision trees, such as AdaRank (Xu and Li, 2007), RankBoost (Freund et al., 2003) and lamdaMART (Wu et al., 2009).
Apart from speed and accuracy, boosting is attractive due to promising theoretical properties such as convergence and generalization.
(Bartlett et al., 1998; Freund et al., 2003; Mohri et al., 2012).
Boosted decision trees have recently been demonstrated to be competitive on LTR tasks (Qin et al., 2021), but, in recent years, boosting approaches have generally received less attention, as (pretrained) neural models began to dominate much of the literature.
However, modern neural models and boosting techniques need not be exclusive, and a small amount of work exploring boosting in the context of modern pre-trained neural models has been carried out (Huang et al., 2020;
Qin et al., 2021).
Our work follows this line of thinking, identifying dimensionally-constrained biencoders as good candidates as neural weak learners, adopting a simple boosting approach which allows for simple and efficient MIPS at test time.
Dense Retrieval
Sparse, term-based Retrievers such as BM25 (Robertson and Zaragoza, 2009) have dominated retrieval until recently.
Dense,
MIPS-based Retrieval using bi-encoder architectures leveraging contrastive training with gold pairs (Yih et al., 2011) has recently shown to be effective in several settings (Lee et al., 2019;
Karpukhin et al., 2020; Reimers and Gurevych, 2019; Hofst¨atter et al., 2021b).
See Yates et al.
(2021) for a survey.
The success of Dense Retrieval has led to many recent papers proposing schemes to improve dense retriever training by innovating on how negatives are sampled (Xiong et al., 2020;
Qu et al., 2021; Zhan et al., 2021c;
Lin et al., 2021, inter alia.), and/or proposing pretraining objectives (O˘guz et al., 2021; Guu et al., 2020; Chang et al., 2020; Sachan et al., 2021; Gao and Callan, 2021).
Our work also innovates on how dense retrievers are trained, but is arguably orthogonal to most of these training innovations, since these could still be employed when training each component weak learner.
Distillation
We leverage a simple distillation technique to make DrBoost more efficient at test time.
Distillation for dense retrievers is an active area, and more complex schemes exist which could improve results further (Izacard and Grave, 2021;
Qu et al., 2021; Yang and Seo, 2020; Lin et al., 2021; Hofst¨atter et al., 2021a; Barkan et al., 2020;
Gao et al., 2020).
Multi-vector Retrievers
Several approaches represent passages with multiple vectors.
Humeau et al.
(2020) represent queries with multiple vectors, but retrieval is comparatively slow as relevance cannot be calculated with a single MIPS call.
ME-BERT (Luan et al., 2021) index a fixed number of vectors for each passage and ColBERT (Khattab and Zaharia, 2020) index a vector for every word.
Both can perform retrieval with a single MIPS call (although ColBERT requires reranking) but produce very large indices, which, in turn, slows down search.
DrBoost can also be seen as a multivector approach, with each weak learner producing a vector.
However, each vector is small, and we index concatenated vectors, rather than indexing each vector independently, leading to small indices and fast search.
This said, adapting DrBooststyle training to these settings would be feasible.
SPAR (Chen et al., 2021) is a two-vector method: one from a standard dense retriever, and the other from a more lexically-oriented model.
SPAR uses a similar test-time MIPS retrieval strategy to ours, and SPAR’s lexical embeddings could be trivially added to DrBoost as an additional subvector.
Efficient retrievers
There have been several recent efforts to build more efficient retrieval and question answering systems (Min et al., 2021).
Izacard et al.
and Yang and Seo (2021) experiment with post-hoc compression and lowerdimensional embeddings, Lewis et al.
(2021) index and retrieve question-answer pairs and Yamada et al.
(2021) propose BPR, which approximates
MIPS using binary vectors.
There is also a line of work learning embeddings specifically suited for approximate search (Yu et al., 2018; Zhan et al., 2021a,b) Generative retrievers (De Cao et al., 2021) can also be very efficient.
DrBoost also employs lower-dimensional embeddings and off-the-shelf post-hoc compression for its smallest index.  Discussion
In this work we have explored boosting in the context of dense retrieval, inspired by the similarity of iteratively-sampling negatives to boosting.
We find that our simple boosting approach, DrBoost, performs largely on par with a 768-dimensional
DPR baseline, but produces more compact vectors, and is more amenable to approximate search.
We note that DrBoost requires maintaining more neural models at test time, which may put a greater demand on GPU resources. However, the models can be run in parallel if latency is a concern, and if needed, these models can be distilled into a single model with little drop in accuracy. We hope that future work will build on boosting approaches for dense retrieval, including adding adaptive weights, and investigating alternative losses and sampling techniques. We also suggest that emphasis in dense retrieval should be placed on more holistic evaluation than just exact retrieval accuracy, demonstrating that models with quite similar exact retrieval can perform very differently under practically-important approximate search settings.
Nearest
Neighbor
Negative Contrastive Learning for Dense Text Retrieval.
arXiv:2007.00808 [cs] .
ArXiv: 2007.00808.
Jun Xu and Hang Li.
AdaRank: a boosting
Sohee Yang and Minjoon Seo.
Is Retriever Merely an Approximator of Reader?
arXiv:2010.10999 [cs] .
ArXiv: 2010.10999.
Sohee Yang and Minjoon Seo.
arXiv
[cs] .
ArXiv: 2104.
. 2021a
Jointly Optimiz ing Query Encoder
Product Quantization
Improve Retrieval Performance.
arXiv:
00644 [cs] .
ArXiv: 2108.00644.
Jingtao Zhan
Jiaxin
Yiqun
Jiafeng Guo
Min
Zhang,
Shaoping Ma arXiv
[cs]
ArXiv:
05789 version: 1.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo,
Min Zhang, and Shaoping Ma.
2021c
Model
Hard Negatives.
arXiv:2104.08051 [cs] .
ArXiv: 2104.08051.
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng
Lv, Nan Duan, and Weizhu Chen. 2021.
Adversarial retriever-ranker for dense text retrieval.
arXiv preprint arXiv:2110.03611 .
[Table]
A Appendix A. Number of Rounds
The performance of DPR and DrBoost on MSMARCO for different numbers of rounds are shown in Table 5.
We find that all models saturate at about 4 or 5 rounds.
Note DrBoost does not need more iterations to train, even though it does not use BM negatives for the first round.
On NQ, adding a 6 th model slightly improves DrBoost’s precision, at the expense of recall (see Table 1).
While iterative training is expensive, we find that subsequent rounds are much cheaper than the first round, with the first round taking ∼ 20K steps in our experiments to converge, with additional DrBoost rounds converging after about 3K steps.
Bagging Dense Retrieval
We also trained a simple ensemble of six 32-dim DPR models for NQ, which we compare to our 6 × 32-dim component
DrBoost.
This experiment investigates whether the improvement over DPR is just a simple ensembling effect, or whether it is due to boosting effects and specialization of concerns.
This DPR ensemble performs poorly, scoring 74.5 R@20 (not shown in tables), 6.8% below the equivalent DrBoost, confirming that the boosting formulation is important, not simply having several ensembled dense retrievers.
A. Implementation Details
We implement our models architectures based on
HuggingFace’s
Transformers (Wolf et al., 2020) and run our experiments on 16 V100 GPUs.
For all training rounds, we used the same set of training hyperparameters — we set learning rate as 3e-5, dropout as 0.1, weight decay as 0.01, batch size as(per GPU) and max training steps as 30k.
The maximum question and passage lengths are set asand 200 respectively and we accompany each question with 50 passages during training.
Using our training infrastructure, the first round of the training takes about 8 hours and each additional training round takes about 1.5 hours until convergence.
We always use the dev loss for model selection.
A. Detailed results
[Table]
[Table]
[Table] [Table]
[Table]
[Table]