A Study on the Efficiency and Generalization of Light Hybrid Retrievers (2023)

Abstract: Hybrid retrievers can take advantage of both sparse and dense retrievers. Previous hybrid retrievers leverage indexing-heavy dense retrievers. In this work, we study “Is it possible to reduce the indexing memory of hybrid retrievers without sacrificing performance”? Driven by this question, we leverage an indexing-efficient dense retriever (i.e. DrBoost) and introduce a LITE retriever that further reduces the memory of DrBoost. LITE is jointly trained on contrastive learning and knowledge distillation from DrBoost. Then, we integrate BM25, a sparse retriever, with either LITE or DrBoost to form light hybrid retrievers. Our Hybrid-LITE retriever saves 13\times memory while maintaining 98.0% performance of the hybrid retriever of BM25 and DPR. In addition, we study the generalization capacity of our light hybrid retrievers on out-of-domain dataset and a set of adversarial attacks datasets. Experiments showcase that light hybrid retrievers achieve better generalization performance than individual sparse and dense retrievers. Nevertheless, our analysis shows that there is a large room to improve the robustness of retrievers, suggesting a new research direction.

## introduction the classical ir methods, such as bm25 @xcite , produce sparse vectors for question and documents based on bag-of-words approaches. recent research pays attention toward building neural retrievers which learn dense embeddings of the query and document into a semantic space @xcite . sparse and dense retrievers have their pros and cons, and the hybrid of sparse and dense retrievers can take advantage of both worlds and achieve better performance than individual sparse and dense retrievers. therefore, hybrid retrievers are widely used in practice @xcite . previous hybrid retrievers are composed of indexing-heavy dense retrievers (dr), in this work, we study the question "is it possible to reduce the indexing memory of hybrid retrievers without sacrificing performance?" to answer this question, we reduce the memory by using the state-ofthe-art indexing-efficient retriever, drboost @xcite , a boosting retriever with multiple "weak" learners. compared to dpr @xcite , a representative dr, drboost reduces the indexing memory by 6 times while maintaining the performance. we introduce a lite model that further reduces the memory of drboost, which is jointly trained on retrieval task via contrastive learning and knowledge distillation from drboost (see figure 1 ). we then integrate bm25 with either lite and drboost to form light hybrid retrievers @xcite to assess whether light hybrid retrievers can achieve memory-efficiency and sufficient performance. we conduct experiments on the naturalquestion dataset @xcite and draw interesting results. first of all, lite retriever maintains 98.7% of the teacher model performance and reduces its memory by 2 times. second, our hybrid-lite saves more than 13× memory compared to hybrid-dpr, while maintaining more than 98.0% performance; and hybrid-drboost reduces the indexing memory (8×) compared to hybrid-dpr and maintains at least 98.5% of the performance. this shows that the light hybrid model can achieve sufficient performance while reducing the indexing memory significantly, which suggests the practical usage of light retrievers for memory-limited applications, such as on-devices. one important reason for using hybrid retrievers in real-world applications is the generalization. thus, we further study if reducing the indexing memory will hamper the generalization of light hybrid retrievers. two prominent ideas have emerged to test generalization: out-of-domain (ood) generalization and adversarial robustness @xcite . we study ood generalization of retrievers on entityquestion @xcite . to study the robustness, we leverage six techniques @xcite to create adversarial attack testing sets based on nq dataset. our experiments demonstrate that hybrid-lite and hybrid-drboost achieve better generalization performance than individual components. the study of robustness shows that hybrid retrievers are always better than sparse and dense retrievers. nevertheless all retrievers are vulnerable, suggesting room for improving the robustness of retrievers, and our datasets can aid the future research. ## related work hybrid retriever integrates the sparse and dense retriever and ranks the documents by interpolating the relevance score from each retriever. the most popular way to obtain the hybrid ranking is applying linear combination of the sparse/dense retriever scores @xcite @xcite @xcite . instead of using the scores, @xcite adopts reciprocal rank fusion @xcite to obtain the final ranking by the ranking positions of each candidate retrieved by individual retriever. @xcite trains a classification model to select one of the retrieval strategies: sparse, dense or hybrid model. most of the hybrid models rely on heavy dense retrievers, and one exception is @xcite , where they use linear projection, pca, and product quantization @xcite to compress the dense retriever component. our hybrid retrievers use either drboost or our proposed lite as the dense retrievers, which are more memory-efficient and achieve better performance than the methods used in @xcite . indexing-efficient dense retriever. efficiency includes two dimensions: latency @xcite @xcite and memory. in this work, our primary focus is on memory, specifically the memory used for indexing. most of the existing drs are indexing heavy @xcite @xcite . to improve the indexing efficiency, there are mainly three types of techniques. one is to use vector product quantization @xcite . second is to compress a high dimension dense vector to a low dimension dense vector, for e.g. from 768 to 32 dimension @xcite . the third way is to use a binary vector @xcite . our proposed method lite ( §3.2) reduces the indexing memory by joint training of retrieval task and knowledge distillation from a teacher model. ## model in this section, we first review drboost @xcite , and our model lite which further reduces the memory of drboost, and lastly, we describe the hybrid retrievers that integrate light dense retrievers (i.e. lite and drboost) and bm25. ## adversarial attack robustness dataset adversarial attacks are used to asses model's robustness, where testing samples are obtained by small perturbations of the original samples, and such perturbations keep the label unchanged. to test the robustness of ir systems, we create 6 different adversarial attacks 1 for nq @xcite . each method is chosen because they do not change the original meaning of the queries and the relevant documents should be the same as the original relevant documents (see figure 2 ). the six methods include: char-swap (cs): augments words by swapping characters out for other characters; word deletion (wd): delete a word randomly from the original query; synonym replacement (sr): replaces a word in the query with a synonym from the wordnet @xcite ; word-order-swap (wos): swaps the order of the words in the original query; synonym insertion (si): insert a synonym of a word from the wordnet to the original query; back-translation (bt) translates the original query into a target language and translates it back to the source language. figure 2 shows an example of each attacked instance 2 . ## experiments and results existing methods. we include four existing methods in this work, drboost @xcite , dpr @xcite , spar @xcite and a heavy hybrid model bm25 + dpr @xcite . in our baselines. three baselines are presented, bm25, dpr 32 , and drboost-2. dpr 32 refers to dpr with a linear projection layer to representation to 32 dimension. drboost-2 takes dpr 32 as the first weak learner, and uses it to mine negative passages to train the next weak learner and then combine these two models. we do not go beyond 2 weak learners because our goal is to achieve memory-efficiency while increasing the number of encoders in the drboost will yield larger indexing. our models. lite and the three light hybrid models are presented. lite is trained by the method we introduce in §3.2 with the distilled knowledge from drboost-2 teacher model. we present three hybrid models bm25 + lite, bm25 + dpr 32 , and bm25 + drboost-2, which are memory-efficient compared to existing methods. next we present the experiments and the findings. ## conclusion to achieve indexing efficiency, in this work, we study light hybrid retrievers. we introduce lite, which is jointly trained on retrieval task via contrastive learning and knowledge distillation from a more capable teacher models which requires heavier indexing-memory. while in this work, we mainly take drboost as the teacher model, lite is a flexible training framework that can be incorporated with most of the neural retriever. then, we integrate bm25 with lite or drboost to form light hybrid retrievers. our light hybrid models achieve sufficient performance and largely reduce the memory. we also study the generalization of retrievers and suggest that all sparse, dense, and hybrid retrievers are not robust enough, which opens up a new avenue for research.