Noisy Pair Corrector for Dense Retrieval (2023)

Abstract: Most dense retrieval models contain an implicit assumption: the training query-document pairs are exactly matched. Since it is expensive to annotate the corpus manually, training pairs in real-world applications are usually collected automatically, which inevitably introduces mismatched-pair noise. In this paper, we explore an interesting and challenging problem in dense retrieval, how to train an effective model with mismatched-pair noise. To solve this problem, we propose a novel approach called Noisy Pair Corrector (NPC), which consists of a detection module and a correction module. The detection module estimates noise pairs by calculating the perplexity between annotated positive and easy negative documents. The correction module utilizes an exponential moving average (EMA) model to provide a soft supervised signal, aiding in mitigating the effects of noise. We conduct experiments on text-retrieval benchmarks Natural Question and TriviaQA, code-search benchmarks StaQC and SO-DS. Experimental results show that NPC achieves excellent performance in handling both synthetic and realistic noise.

## introduction with the advancements in pre-trained language models @xcite , dense retrieval has developed rapidly in recent years. it is essential to many applications including search engine @xcite , open-domain question answering @xcite , and code intelligence @xcite . a typical dense retrieval model maps both queries and documents into a low-dimensional vector space and measures the relevance between them by the similarity between their respective representations @xcite . during training, the model utilizes query-document pairs as labelled training data @xcite and samples negative documents for each pair. then the model learns to minimize the contrastive loss for obtaining a good representation ability @xcite . recent studies on dense retrieval have achieved promising results with hard negative mining @xcite , pretraining @xcite , distillation @xcite , and adversarial training @xcite . all methods contain an implicit assumption: each query is precisely aligned with the positive documents in the training set. in practical applications, this assumption becomes challenging to satisfy, particularly when the corpora is automatically collected from the internet. in such scenarios, it is inevitable that the training data will contain mismatched pairs, incorporating instances such as user mis-click noise in search engines or low-quality reply noise in q&a communities. as shown in fig. 1 , the examples are from staqc benchmark @xcite , which is automatically collected from stackoverflow. the document, i.e., code solution, can not answer the query but is incorrectly annotated as a positive docgreen objects refer to annotated pairs, while pentagram and triangle are actually aligned pairs. in the left case, retrieval models are required to push the query with true-positive document (tp doc) together and pull the query with true-negative documents (tn doc) apart. in the right case, the retrieval models are misled by the mismatched data pair, where the false-positive document (fp doc) and the false-negative document (fn doc) are wrongly pulled and pushed, respectively. to train robust dense retrievers, previous works have explored addressing various types of noise. for example, rocketqa @xcite and ar2 @xcite ) mitigate the falsenegative noise with a cross-encoder filter and distillation, respectively; cocondenser @xcite reduce the noise during fine-tuning with pre-training technique; rodr @xcite deal with query spelling noise with local ranking alignment. however, mismatched-pair noise (false positive problem) in dense retrieval has not been well studied. as shown in fig. 2 , mismatched-pair noise will mislead the retriever to update in the opposite direction. based on these observations, we propose a noisy pair corrector (npc) framework to solve the falsepositive problem. npc consists of noise detection and correction modules. at each epoch, the detection module estimates whether a query-document pair is mismatched by the perplexity between the annotated document and easy negative documents. then the correction module provides a soft supervised signal for both estimated noisy data and clean data via an exponential moving average (ema) model. both modules are plug-and-play, which means npc is a general training paradigm that can be easily applied to almost all retrieval models. the contributions of this paper are as follows: (1) we reveal and extensively explore a long-neglected problem in dense retrieval, i.e., mismatched-pair noise, which is ubiquitous in the real world. (2) we propose a simple yet effective method for training dense retrievers with mismatched-pair noise. code is available at https://github.com/ hangzhang-nlp/npc . ## preliminary before describing our model in detail, we first introduce the basic elements of dense retrieval, including problem definition, model architecture, and model training. given a query q, and a document collection d, dense retrieval aims to find document d + relevant to q from d. the training set consists of a collection of query-document pairs, donated as @xmath0, d + 1 ), ..., (q n , d + n )}, where n is the data size. typical dense retrieval models adopt a dual encoder architecture to map queries and documents into a dense representation space. then the relevance score f (q, d) of query q and document d can be calculated with their dense representations: the above training paradigm assumes that the query-document pair (q i , d + i ) in training set c is correctly aligned. however, this assumption is difficult to satisfy in real-world applications @xcite i,1 is the hard negative of q i and {q 3 , d + 3 } is the estimated noisy pair, the retriever θ and teacher θ * compute similarity matrices s θ and s θ * for all queries and documents, respectively. the retriever learns to minimize (1) l cont : the negative likelihood probability of true positive documents; (2) l cons : the kl divergence between s θ and the rectified soft label s θ * after normalization. 2021; @xcite . in practice, most training data pairs are collected automatically without manual inspection, such as inevitably leading to the inclusion of some mismatched pairs. ## method we propose npc framework to learn retrievers with mismatched-pair noise. as shown in fig. 3 , npc consists of two parts: (a) the noise detection module as described in sec. 3.1, and (b) the noise correction module as described in sec. 3.2. ## conclusion this paper explores a neglected problem in dense retrieval, i.e., mismatched-pair noise. to solve this problem, we propose a generalized noisy pair corrector(npc) framework, which iteratively detects noisy pairs per epoch based on the perplexity and then provides rectified soft labels via an ema model. the experimental results and analysis demonstrate the effectiveness of npc in effectively handling both synthetic and realistic mismatchedpair noise.