Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval (2022)

Abstract: Training dense passage representations via contrastive learning has been shown effective for Open-Domain Passage Retrieval (ODPR). Existing studies focus on further optimizing by improving negative sampling strategy or extra pretraining. However, these studies keep unknown in capturing passage with internal representation conflicts from improper modeling granularity. Specifically, under our observation that a passage can be organized by multiple semantically different sentences, modeling such a passage as a unified dense vector is not optimal. This work thus presents a refined model on the basis of a smaller granularity, contextual sentences, to alleviate the concerned conflicts. In detail, we introduce an in-passage negative sampling strategy to encourage a diverse generation of sentence representations within the same passage. Experiments on three benchmark datasets verify the efficacy of our method, especially on datasets where conflicts are severe. Extensive experiments further present good transferability of our method across datasets.

## introduction open-domain passage retrieval (odpr) has recently attracted the attention of researchers for its wide usage both academically and industrially @xcite . provided with an extremely large text corpus that composed of millions of passages, odpr aims to retrieve a collection of the most relevant passages as the evidences of a given question. with recent success in pretrained language models (prlms) like bert @xcite , roberta @xcite , dense retrieval techniques have achieved significant better results * *corresponding author. this work was supported in part by the key projects of national natural science foundation of china under grants u1836222 and 61733011. previous approaches often pretrain the bi-encoders with a specially designed pretraining objective, inverse cloze task (ict) @xcite . more recently, dpr @xcite adopts a simple but effective contrastive learning framework, achieving impressive performance without any pretraining. concretely, for each question q, several positive passages p + and hard negative passages p -produced by bm25 are pre-extracted. by feeding the bi-encoder with (q, p + , p -) triples, dpr simultaneously maximizes the similarity between the representation of q and corresponding p + , and minimizes the similarity between the representations of q and all p -. following such contrastive learning framework, many researchers are seeking further improvements for dpr from the perspective of sampling strategy @xcite @xcite or extra pretraining @xcite , or even using knowledge distillation @xcite . however, these studies fail to realize that there exist severe drawbacks in the current contrastive learning framework adopted by dpr. essentially, as illustrated in figure 1 , each passage p is composed of multiple sentences, upon which multiple semantically faraway questions can be derived, which forms a question set @xmath0 age of enlightenment 1. … indeed, the majority of the wining entries …, a genre commonly stressed in women's education 2. … the royal society of london also played a significant role in the public sphere and the spread of enlightenment ideas… 3. it was founded by a group of independent scientists and given a royal character in 1662. 4. the society played a large role in spreading robert boyle's experimental philosophy around … what literature genre were the majority of female wining context entries? whose method based knowledge on experiments, which had to be witnessed? {q 1 , q 2 , ..., q k }. under our investigation, such a one-to-many problem is causing severe conflicting problems in the current contrastive learning framework, which we refer to as contrastive conflicts. to the best of our knowledge, this is the first work that formally studies the conflicting problems in the contrastive learning framework of dense passage retrieval. here, we distinguish two kinds of contrastive conflicts. since one-to-many problem is the direct cause of both conflicts, this paper presents a simple but effective strategy that breaks down dense passage representations into contextual sentence level ones, which we refer to as dense contextual sentence representation (dcsr). unlike long passages, it is hard to derive semantically faraway questions from one short sentence. therefore, by modeling odpr in smaller units like contextual sentences, we fundamentally alleviate contrastive conflicts by solving the one-to-many problem. note that we do not simply encode each sentence separately. instead, we encode the passage as a whole and use sentence indicator tokens to acquire the sentence representations within the passage, to preserve the contextual information. we further introduce the in-passage negative sampling strategy, which samples neighboring sentences of the positive one in the same passage to create hard negative samples. finally, concrete experiments have verified the effectiveness of our proposed method from both retrieval accuracy and transferability, especially on datasets where contrastive conflicts are severe foot_0 . contributions (i) we investigate the defects of the current contrastive learning framework in training dense passage representation in open-domain passage retrieval. (ii) to handle contrastive conflicts, we propose to index the wikipedia corpus using contextual sentences instead of passages. we also propose the in-passage negative sampling strategy in training the contextual sentence representations. (iii) experiments show that our proposed method significantly outperforms original baseline, especially on datasets where contrastive conflicts are severe. extensive experiments also present better transferability of our dcsr, indicating that our method captures the universality of the concerned task datasets. traditional approaches like tf-idf @xcite , bm25 @xcite retrieve the evidence passages based on the lexical match between questions and passages. although these lexical approaches meet the requirement of low latency, they fail to capture non-lexical semantic similarity, thus performing unsatisfying on retrieval accuracy. with recent advances of pretrained language models (prlms) like bert @xcite , roberta @xcite , a series of neural approaches based on cross-encoders are proposed @xcite . although enjoying satisfying retrieval accuracy, the retrieval latency is often hard to tolerate in practical use. more recently, the bi-encoder structure has captured the researchers' attention. with bi-encoder, the representations of the corpus at scale can be precomputed, enabling it to meet the requirement of low latency in passage retrieval. @xcite first proposes to pretrain the bi-encoder with inverse cloze task (ict). later, dpr @xcite introduces a contrastive learning framework to train dense passage representation, and has achieved impressive performance on both retrieval accuracy and latency. based on dpr, many works make further improvements either by introducing better sampling strategy @xcite @xcite or extra pretraining @xcite , or even distilling knowledge from cross-encoders @xcite . our method follows the contrastive learning research line of odpr. different from previous works that focus on either improving the quality of negative sampling or using extra pretraining, we make improvements by directly optimizing the modeling granularity with an elaborately designed contrastive learning training strategy. contrastive learning contrastive learning recently is attracting researchers' attention in all area. after witnessing its superiority in computer vision tasks @xcite , researchers in nlp are also applying this technique @xcite @xcite @xcite . for the concern of odpr, the research lines of contrastive learning can be divided into two types: (i) improving the sampling strategies for positive samples and hard negative samples. according to @xcite , the quality of positive samples and negative samples are of vital importance in the contrastive learning framework. therefore, many researchers seek better sampling strategies to improve the retrieval performance @xcite . (ii) improving the contrastive learning framework. densephrase @xcite uses memory bank like moco @xcite to increase the number of inbatch negative samples without increasing the gpu memory usage, and models retrieval process on the phrase level but not passage level, achieving impressive performance. our proposed method follows the second research line. we investigate a special phenomenon, contrastive conflicts in the contrastive learning framework, and experimentally verify the effectiveness of mediating such conflicts by modeling odpr in a smaller granularity. more similar to our work, akkalyoncu @xcite also proposes to improve dense passage retrieval based on sentence-level evidences, but their work is not in the research line of contrastive learning, and focuses more on passage re-ranking after retrieval but not retrieval itself. ## conclusion in this paper, we make a thorough analysis on the contrastive conflicts issue in the current opendomain passage retrieval. to well address the issue, we propose an enhanced sentence-aware conflict learning method by carefully generating sentence-aware positive and negative samples. we show that the dense contextual sentence representation learned from our proposed method achieves significant performance gain compared to the original baseline, especially on datasets with severe conflicts. extensive experiments show that our proposed method also enjoys better transferability, and well captures the universality in different datasets.