Towards Robust Ranker for Text Retrieval (2023)

Abstract: A neural ranker plays an indispensable role in the de facto ‘retrieval & rerank’ pipeline, but its training still lags behind due to the weak negative mining during contrastive learning. Compared to retrievers boosted by self-adversarial (i.e., in-distribution) negative mining, the ranker’s heavy structure suffers from query-document combinatorial explosions, so it can only resort to the negative sampled by the fast yet out-of-distribution retriever. Thereby, the moderate negatives compose ineffective contrastive learning samples, becoming the main barrier to learning a robust ranker. To alleviate this, we propose a multi-adversarial training strategy that leverages multiple retrievers as generators to challenge a ranker, where i) diverse hard negatives from a joint distribution are prone to fool the ranker for more effective adversarial learning and ii) involving extensive out-of-distribution label noises renders the ranker against each noise distribution, leading to more challenging and robust contrastive learning. To evaluate our robust ranker (dubbed R2anker), we conduct experiments in various settings on the passage retrieval benchmarks, including BM25-reranking, full-ranking, retriever distillation, etc. The empirical results verify the new state-of-the-art effectiveness of our model.

## introduction text retrieval plays a crucial role in many applications, such as web search @xcite and recommendation @xcite . given a text query, it aims to retrieve all relevant documents from a large-scale collection 1 @xcite . for a better efficiencyeffectiveness trade-off, the text retrieval de facto paradigm relies on a 'retrieval & rerank' pipeline ( @xcite . that is, 'retrieval' is to use a fast retriever to fetch a set of top document candidates given a query, while 'rerank' is to re-calculate the relevance of the query to each candidate by a heavy yet effective ranker for better results. differing from most natural language understanding (nlu) tasks defined as categorical classification @xcite , training retrieval models, including the retriever and the ranker, are usually formulated as a contrastive learning problem. however, there are merely positive querydocument pairs provided in most applications, regardless of negative samples. hence, a critical prerequisite of the training is to sample negative documents from the collection for training queries. as random sampling is prone to mine trivial negatives and proven less effective in training (a.k.a. in-batch negatives), a primary sampling method was proposed to leverage bm25 @xcite to fetch relatively challenging negatives for more effective training. in contrast to such an out-of-distribution negative sampling technique where the negatives mined by one retriever are used to train another, recent advanced negative mining methods resort to in-distribution sampling technique that leverages the retriever being trained to obtain the challenging-so-far negative documents from the collection. it has been proven in-distribution sampling is superior to outof-distribution one as the former offers more modelspecific contrastive learning samples towards the nuance among the positive and negatives for a given query @xcite . nonetheless, a necessary condition of such an in-distribution sampling technique is the efficiency of the target model in large-scale retrieval. unfortunately, compared to the bi-encoder based retriever that satisfies the efficiency requirement of largescale retrieval, a cross-encoder based ranker suffers from combinatorial explosion brought by applying the heavy cross-encoder (e.g., the transformer encoder) to every query-document concatenation. thereby, the training of ranker can depend merely on out-of-distribution negative sampling technique -either by the bm25 retriever (nogueira and cho, 2019) or a trainable semantic retriever @xcite ) -leading to sub-optimal ranker due to a lack of adversarial training samples towards the expressively powerful cross-encoder. in this paper, we aim to train a robust ranker by mining more challenging negatives and thus more effective contrastive samples. to this end, we propose a simple yet effective multi-adversarial training framework towards a robust ranker (r 2 anker), where multiple retrievers as generators are integrated to mine diverse hard negatives and challenge a single ranker as the discriminator. as such, r 2 anker has certain merits regarding the robustness of its model training. first, intuitively, sampling negative over a joint distribution of various retrievers is more likely to offer more challenging hard negatives, which compensates the weakness of the previous single-retriever generator and makes the adversarial learning more robust. second, as the false negatives are closely subject to the relevance distribution over the collection by a specific retriever, various negative generators achieved by different retrievers are prone to sample out-of-distribution or open-set label noise @xcite to each other. in light of 'insufficient capacity' assumption @xcite , such open-set noise has been proven effective in improving robustness @xcite when learning a ranker with open-set noises. in experimentswe adopt several passage benchmark datasets @xcite to evaluate our proposed model in various settings. specifically, our method achieves new state-of-the-art performance on bm25 reranking and full-ranking on passage retrieval. meantime, to verify the expressive power of our ranker model, we conduct an experiment to distill our well-trained model to a retriever, which shows state-of-the-art first-stage in terms of passage retrieval performance. moreover, our extensive analyses unveil the essence regarding negative distributions to reach a robust ranker and also compare with negatives sampled by redistribution. ranker training. using negatives to train a ranker has proven effective in many works @xcite . since the ranker is based on the cross-encoder structure, a query and document need to be concatenated and passed to the ranker for relevance calculation. however, directly using the ranker to sample negatives on the collection suffers from combinatorial explosion. therefore, many methods @xcite adopt the static hard negatives sampled from a retriever, which are fixed during ranker training. in addition, some methods @xcite introduce a joint training approach for dense passage retrieval and passage reranking, which dynamically update both the parameters of the ranker and the retriever. nevertheless, these methods can depend merely on out-of-distribution negative sampling techniques leading to sub-optimal rankers due to a lack of adversarial training samples towards the expressively powerful cross-encoder. therefore, we integrate multiple retrievers regarded as generators to mine diverse hard negatives and challenge a single ranker as the discriminator. hard negative mining. hard negative mining @xcite @xcite has been proven very effective in contrastive learning for text representation of retrievers. in contrast to random or in-batch negative sampling, it can find more challenging negatives for a pair of an anchor (i.e., query) and its positive example. they compose effective contrastive samples to help models learn against contextual nuance between the positive and negatives. at early stage, a large number of works employ the off-theshelf bm25 retriever to fetch negative from a large collection @xcite , which greatly boosts the retrievers. furthermore, recent works @xcite leverage a retriever to sample retriever-specific hard negatives for each query, which are considered the most challenging negatives. in this study, we sample negatives over a joint distribution of various retrievers, which is likely to offer more challenging hard negatives. ## r 2 anker: robust ranker task formulation. given a text query q, a ranker model, k(q, d), is responsible for calculating a relevance score between q and an arbitrary document d from a large-scale collection d (i.e., @xmath0. it usually serves as a downstream module for an efficient retriever, r, to compose a 'retrieval & rerank' pipeline, where a lightweight retriever r (e.g., biencoder) is to retrieve top candidates and then a relatively heavy-structured k (says cross-encoder @xcite ) to make the results better. ## experiments we evaluate our ranker on following 3 retrieval tasks, and refer to §c for setups of ranker training, retriever distillation, and retriever pre-training. bm25 reranking task. bm25 reranking uses a ranker to re-rank the top 1000 passages by bm25 for each query. here, we adopt the popular passage retrieval datasets, ms-marco @xcite and trec deep learning 2019 @xcite , as well as their official bm25 retrieval candidates. following previous work, we report the performance in mrr@10 on ms-marco and ndcg@10 on trec deep learning 2019. full ranking task. full ranking leverages a ranker to rank the top-1000 passages retrieved from the full collection by a specific retriever. we adopt the ms-marco dataset on which we will specify the retriever and report the performance in mrr@10. large-scale retrieval task. large-scale retrieval uses a retriever to fetch top-relevant passages from a collection. here, a ranker only plays a teaching role for knowledge distillation into a biencoder based retriever. the retriever is then used to perform this task on ms-marco dataset, where mrr@10 and r@50 are used as metrics. ## conclusion in this work, we propose a multi-adversarial strategy for robust ranker training where a negative generator built upon a joint of diverse retrievers is proposed to sample challenging hard negatives for effective adversarial learning. empirically, our proposed ranker, r 2 anker, achieves state-of-theart performance in both bm25-reranking and fullranking tasks on benchmark datasets. and empowered by our r 2 anker as a teacher for distillation, previous basic retrievers can now deliver state-ofthe-art results in large-scale retrieval tasks. moreover, our insightful analysis also reveals the minor impact of training-test distribution shift in bm25 reranking due to the generality of bm25 retriever. meantime, we also find that there is a negative correlation between the ranker-generator distance and performance of the ranker, and re-distribution towards a ranker is toxic due to false negative labels.