S im ANS : Simple Ambiguous Negatives Sampling for Dense Text Retrieval (2022)

Abstract: Sampling proper negatives from a large document pool is vital to effectively train a dense retrieval model. However, existing negative sampling strategies suffer from the uninformative or false negative problem. In this work, we empirically show that according to the measured relevance scores, the negatives ranked around the positives are generally more informative and less likely to be false negatives. Intuitively, these negatives are not too hard ( may be false negatives ) or too easy ( uninformative ). They are the ambiguous negatives and need more attention during training.Thus, we propose a simple ambiguous negatives sampling method, SimANS, which incorporates a new sampling probability distribution to sample more ambiguous negatives.Extensive experiments on four public and one industry datasets show the effectiveness of our approach.We made the code and models publicly available in https://github.com/microsoft/SimXNS .

## introduction dense text retrieval, which uses low-dimensional vectors to represent queries and documents and measure their relevance, has become a popular topic @xcite for both researchers and practitioners. it can improve various downstream applications, e.g., web search @xcite ) and question answer @xcite . a key challenge for training a dense text retrieval model is how to select appropriate negatives from a large document pool (i.e., negative sampling), as most existing methods use a contrastive loss @xcite to encourage the model to rank positive documents higher than negatives. however, the commonly-used negative sampling strategies, namely random negative sampling @xcite † † this work was done during internship at msra. * * corresponding author, email: batmanfly@gmail.com. motivated by these problems, we propose to sample the ambiguous negatives foot_0 that are neither too easy (uninformative) nor too hard (potential false negatives). our approach is inspired by an empirical observation from experiments (in §3) using gradients to assess the impact of data instances on deep models @xcite : according to the measured relevance scores using the dense retrieval model, negatives that rank lower are mostly uninformative, as their gradient means are close to zero; negatives that rank higher are likely to be false negatives, as their gradient variances are significantly higher than expected. both types of negatives are detrimental to the convergence of deep matching models @xcite . interestingly, we find that the negatives ranked around positive examples tend to have relatively larger gradient means and smaller variances, indicating that they are informative and have a lower risk of being false negatives, thus probably being high-quality ambiguous negatives. based on these insights, we propose a simple ambiguous negative sampling method, namely simans, for improving deep text retrieval. our main idea is to design a sampling probability distribution that can assign higher probabilities to the ambiguous negatives while lower probabilities to the possible false and uninformative negatives, based on the differences of the relevance scores between positives and candidate negatives. we also incorporate two hyper-parameters to better adjust the peak and density of the sampling probability distribution. our approach is simple and flexible, which can be easily applied to various dense retrieval models and combined with other effective techniques, e.g., knowledge distillation @xcite and adversarial training @xcite . to validate the effectiveness of simans, we conduct extensive experiments on four public datasets and one industrial dataset collected from bing search logs. experimental results show that simans can improve the performance of competitive baselines, including state-of-the-art methods. ## preliminary dense text retrieval. given a query q, the dense text retrieval task aims to retrieve the most relevant top-k documents {d i } k @xmath0. to achieve it, the dual-encoder architecture is widely used due to its efficiency @xcite . it consists of a query encoder e q and a document encoder e d to map the query q and document d into k-dimensional dense vectors h q and h d , respectively. then, the semantic relevance score of q and d can be computed using dot product as recent works mostly adopt pre-trained language models (plms) @xcite as the two encoders, and utilize the representations of the [cls] token as dense vectors. training with negative sampling. the training objective of dense text retrieval task is to pull the representations of the query q and relevant documents d + together (as positives), while pushing apart irrelevant ones d -= d \ d + (as negatives). however, the irrelevant documents are from a large document pool, which would lead to millions of negatives. to reduce the unreachable training cost, negative sampling has been widely used. previous works either randomly sample negatives @xcite , or select the top-k hard negatives ranked by bm25 or the dense retrieval model itself @xcite , denoted as d -. then, the optimization ob-jective can be formulated as: ## motivation study we first analyze the uninformative and false negative problems from the perspective of gradients. then, we perform an empirical study to test how gradients of negatives change w.r.t. ranks according to measured relevance scores using a dense retrieval model, and find that the gradients of negatives ranked near positives have relatively larger means and smaller variances. ## approach based on the findings in §3, we conjecture that the ambiguous negatives ranked near positives according to relevance scores are high-quality negatives, as they are neither too easy (uninformative) nor too hard (may be false negatives). therefore, we propose a simple ambiguous negative sampling method, namely simans. ## conclusion we investigated how the gradient statistics of negative documents affect their relevance ranking for dense text retrieval. we discovered that negative documents with high gradient means and low gradient variances are more likely to be ambiguous negatives, which are informative and less prone to false negatives. based on this insight, we proposed simans, a novel negative sampling method that balances the difficulty of negative examples by adjusting their sampling probabilities. simans improved the performance of various dense retrieval models on four public and one industrial datasets. we plan to apply our method to other information retrieval tasks, such as personal recommendation, and to develop better pre-training schemes for dense text retrieval in the future. too easy negatives too hard negatives 𝑞 figure 4 : an example of the dense embedding distribution of a query with its positive document, too easy, too hard and ambiguous negatives.