Query-as-context Pre-training for Dense Passage Retrieval (2023)

Abstract: Recently, methods have been developed to improve the performance of dense passage retrieval by using context-supervised pre-training. These methods simply consider two passages from the same document to be relevant, without taking into account the potential negative impacts of weakly correlated pairs. Thus, this paper proposes query-as-context pre-training, a simple yet effective pre-training technique to alleviate the issue. Query-as-context pre-training assumes that the query derived from a passage is more likely to be relevant to that passage and forms a passage-query pair. These passage-query pairs are then used in contrastive or generative context-supervised pre-training. The pre-trained models are evaluated on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks. Experimental results show that query-as-context pre-training brings considerable gains for retrieval performances, demonstrating its effectiveness and efficiency.

## introduction passage retrieval is the process of retrieving relevant passages from a large corpus in response to a query, which is useful in a variety of downstream applications such as web search @xcite @xcite , question answering @xcite @xcite and dialogue systems @xcite . the success of pre-trained language models (plms) @xcite has led to the development of more powerful plm-based dense and sparse passage retrieval approaches. plm-based dense retrieval methods @xcite @xcite @xcite @xcite @xcite passage 1. a good day at pipeline means an encounter with fellow surfers who can be as friendly as pitbulls with migraines, and waves that can shatter boards into kindling. and then there's the reef. at pipeline there can be 10 foot waves blasting over just three feet of water, so if you fall on the reef or get caught inside the break, you're lucky if you come out merely sliced up. passage 2. this idyll can still be found on the island of lanai. formerly a pineapple plantation, lanai is almost entirely owned by billionaire entrepreneur david murdock, and its only notable commercial hub is a tiny village built around a square of cook pine trees. figure 1 : an example of low-relevance passages within a document from the ms-marco corpus. the two passages are weakly correlated in content. @xcite use plms to encode queries and passages into a shared semantic space. the semantic relationships between query and passage representations are then measured by dot product or cosine similarities. pre-training and finetuning techniques have been developed to improve the performance of dense retrieval models. pretraining processes for dense retrieval aim to improve the text representation modeling ability of the encoder through auxiliary self-supervised or context-supervised tasks. context-supervised pre-training @xcite assumes that two passages foot_0 within the same document are contextual or related to each other and can therefore be used for contrastive learning or contextual decoding. however, context-supervised pre-training ignores the fact that the passages within a document may be weakly related or even irrelevant in many cases. as shown in figure 1 , two passages within a document from the ms-marco corpus @xcite are not directly related in content. according to our statistic results via human annotation, only 35.5% of passage pairs in the training data of cocondenser @xcite have high correlation. for statistical details, please refer to appendix a. these weakly correlated or irrelevant passages do not align with the assumptions on which context-supervised pre-training is based, and are likely to be detrimental to context-supervised pre-training. in contrast to dense retrieval, sparse retrieval is based on the "bag-of-words" assumption and represents passages and queries as sparse term-based vectors. plm-based sparse retrieval @xcite @xcite @xcite @xcite uses plm to improve sparse vectors. one representative technique is query prediction @xcite , which predicts a set of relevant queries to enrich the passage's content and thus alleviates the mismatch problem. query prediction has been shown to be effective in sparse retrieval, but has not yet been explored in the context of dense retrieval, especially in the pre-training process. this raises the question of whether the query prediction technique can benefit the pre-training process tailored for dense retrieval. the observation that predicted queries align better with the content of a passage in our statistical analyses (see appendix a) suggests that query prediction could be a promising way to alleviate the issue of weakly correlated passages for contextsupervised pre-training. thus, this paper focuses on exploring query prediction techniques to improve context-supervised pre-training methods for dense retrieval. our proposed method, termed query-as-context pre-training, assumes that a query derived from a passage (using a generative model like t5) is more likely to be a relevant context to the passage. in contrast to the previous contextsupervised methods that create a training pair using two randomly selected passages from a document, the query-as-context method generates a training pair by combining a passage with a predicted query, as illustrated in figure 2 . there are several advantages to using the queryas-context setting. firstly, the query is more likely to be related to the passage because it is generated from the passage. additionally, the use of passage-query pairs in supervised downstream retrieval training is consistent with using passagequery pairs in pre-training, which helps to bridge the gap between the two processes. finally, since the passage-query pairs are generally shorter than the previously used passage-passage pairs, it speeds up the pre-training process and reduces the training overhead. to verify the effectiveness of our proposed query-as-context pre-training, we conduct experiments on large-scale web search benchmarks: ms-marco passage ranking @xcite , trec deep learning (dl) track 2019 @xcite and track 2020 @xcite . we also evaluate query-as-context pre-trained models on the beir @xcite benchmark with a large set of out-of-domain datasets. experimental results show that query-ascontext achieves considerable gains over competing baselines. our contributions can be summarized as follows: ## preliminary: context-supervised pre-training in this section, we begin by providing an overview of the pre-training corpus. subsequently, we describe the masked language modeling task, which serves as a foundational task of pre-training. finally, we present two representative contrastive and generative context-supervised pre-training methods, on which our proposed query-as-context will be applied. pre-training corpus given a set of documents, we randomly extract pairs of passages from each document, which forms a training corpus as follows: masked language modeling (mlm) formally, given a passage x with n tokens, a special token [cls] is added to the beginning of the passage, resulting in the output of the last layer is then used to calculate the mlm's target loss an additional auxiliary decoder is also appended to the encoder, which consists of n layers of transformers. the auxiliary decoder takes the concatenation of the [cls] representation from the l-th layer, i.e., h l 0 , and the token representations from the encoder's m -th (e.g. 6-th) layer, i.e., {h m 1 , ..., h m n }, as inputs. similar to mlm, the output of the auxiliary decoder's last layer is then used to perform an auxiliary mlm pre-training. finally, the total loss of cocondenser is: for more details, please refer to @xcite . ## query-as-context pre-training in this section, we first introduce the details of query-as-context pre-training, and then introduce the fine-tuning process of the pre-trained models on the retrieval tasks. ## analyses in this section, we examine the efficiency advantage and analyze the impact of different settings on query-as-context pre-training. ## conclusions in this work, we propose query-as-context pretraining, a simple yet effective technique to alleviate the previously ignored issue of weakly correlated pairs during context-supervised pre-training. extensive experiments well validate its effectiveness and efficiency.