Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training (2023)

Abstract: Dense retrievers have achieved impressive performance, but their demand for abundant training data limits their application scenarios. Contrastive pre-training, which constructs pseudo-positive examples from unlabeled data, has shown great potential to solve this problem. However, the pseudo-positive examples crafted by data augmentations can be irrelevant. To this end, we propose relevance-aware contrastive learning. It takes the intermediate-trained model itself as an imperfect oracle to estimate the relevance of positive pairs and adaptively weighs the contrastive loss of different pairs according to the estimated relevance. Our method consistently improves the SOTA unsupervised Contriever model on the BEIR and open-domain QA retrieval benchmarks. Further exploration shows that our method can not only beat BM25 after further pre-training on the target corpus but also serves as a good few-shot learner. Our code is publicly available at https://github.com/Yibin-Lei/ReContriever .

## introduction dense retrievers, which estimate the relevance between queries and passages in the dense embedding space, have achieved impressive performance in various applications, including web search @xcite and open-domain question answering @xcite . one key factor for the success of dense retrievers is a large amount of human-annotated training data, e.g., ms-marco @xcite with above 500,000 examples. however, a recent study @xcite shows that even trained with enormous labeled data, dense retrievers still suffer from a generalization issue, where they perform relatively poorly on novel domains in comparison to bm25. meanwhile, collecting human-annotated data for new domains is always hard and expensive. thus improving dense retrievers with limited annotated data becomes essential, considering the significant domain variations of practical retrieval tasks. contrastive pre-training, which first generates pseudo-positive examples from a universal corpus and then utilizes them to contrastively pre-train retrievers, has shown impressive performance without any human annotations @xcite @xcite @xcite . for instance, contriever @xcite crafts relevant querypassage pairs by randomly cropping two random spans within the same document. however, owing to the high information density of texts, even nearby sentences in a document can be very irrelevant, as shown in figure 1 . these false positive samples may mislead the model to pull unrelated texts together in the embedding space and further harm the validity of representations. motivated by recent findings in computer vision that pre-training performance can be greatly boosted by reducing the effect of such false positives @xcite , we propose relevance-aware contrastive retriever (re-contriever). at each training step, we utilize the trained models at the current step itself to estimate the relevance of all the positives. then the losses of different positive pairs are adaptively weighed using the estimated relevance, i.e., the pairs that receive higher relevance scores obtain higher weight. moreover, simply applying lower weights to irrelevant pairs will result in insufficient usage of data, since many documents will contribute less to training. therefore, we also introduce a one-documentmultiple-pair strategy that generates multiple positive pairs from a single document, with a pairweighting process conducted among samples originating from a single document. such an operation makes sure that the model can learn positive knowledge from every document in the corpus. to summarize, our contributions in this paper are three-fold: 1) we propose relevance-aware contrastive learning for dense retrieval pre-training, which aims to reduce the false positive problem. 2) experiments show our method brings consistent improvements to the sota unsupervised contriver model on 10/15 tasks on the beir benchmark and three representative open-domain qa retrieval datasets. 3) further explorations show that our method works well given no or limited labeled data. specifically, on 4 representative domainspecialized datasets it outperforms bm25 when only unsupervised pre-training on the target corpora, and with only a few annotated samples its accuracy can be on par with dpr @xcite which is trained on thousands of annotated examples. ## experiments in this section, we evaluate our model in several settings after describing our experimental setup. we consider unsupervised retrieval performance and two practical use cases: further pre-training on the target domain and few-short retrieval. we then conduct an ablation study to separate the impact of our method's two components. ## conclusion in this work, we propose recontriever to further explore the potential of contrastive pre-training to reduce the demand of human-annotated data for dense retrievers. benefiting from multiple positives from the same document as well as relevance-aware contrastive loss, our model achieves remarkable performance under zero-shot cases. additional results on low data resources further verify its value under various practical scenarios.