Effective Contrastive Weighting for Dense Query Expansion (2023)

Abstract: Verbatim queries submitted to search engines often do not sufficiently describe the user’s search intent. Pseudo-relevance feedback (PRF) techniques, which modify a query’srepresentation using the top-ranked documents, have been shown to overcome such inadequacies and improve retrieval effectiveness for both lexical methods (e.g., BM25) and dense methods (e.g., ANCE, ColBERT). For instance, the recent ColBERT-PRF approach heuristically chooses new embeddings to add to the query representation using the inverse document frequency (IDF) of the underlying tokens. However, this heuristic potentially ignores the valuable context encoded by the embeddings. In this work, we present a contrastive solution that learns to select the most useful embeddings for expansion. More specifically, a deep language model-based contrastive weighting model, called CWPRF, is trained to learn to discriminate between relevant and non-relevant documents for semantic search. Our experimental results show that our contrastive weighting model can aid to select useful expansion embeddings and outperform various baselines. In particular, CWPRF can improve nDCG@10 by upto to 4.1% compared to an existing PRF approach for ColBERT while maintaining its efficiency.

## introduction when using search engines, users frequently enter queries that insufficiently express their desired intent. for instance, a user who issues the query georgia run off elections may indeed be looking for details about a specific electoral procedure in the us state of georgia. for search algorithms that rely on lexical matching, such as bm25, this can result in a lexical gap, since relevant documents may just as easily use different terms (e.g., ga and 2nd-round). pseudo-relevance feedback (prf) techniques are often employed to overcome such a lexical gap. indeed, classical prf techniques, such as rm3 @xcite , have been widely used to enrich the user's query with terms selected from the initial retrieval top-ranked documents, i.e. the pseudo-relevance feedback set @xcite @xcite . this expanded query is capable of overcoming the lexical gap if the pseudo-relevance feedback documents are relevant and additional related terms can be identified (for instance adding ga to the query). however, there is a risk that the added terms drift the intent of the query (for instance, adding terms such as tbilisi that relate to the country of georgia rather than the us state). an alternative approach for overcoming the lexical gap is to perform semantic search over learned embedded documents (single representation, e.g., ance @xcite ) or tokens (multiple representations, e.g., colbert @xcite ). such dense retrieval approaches enable queries to retrieve documents that do not necessarily contain the query terms. however, the encoded query vectors might still not adequately express the user's desired intent. indeed, several recent works have shown that implementing prf techniques within the dense retrieval paradigm -such as ance-prf @xcite , vector-prf @xcite and colbert-prf @xcite ) -can further improve retrieval effectiveness. colbert-prf has been shown to be more effective than vector-prf and ance-prf variants applied on various dense retrieval models. a key limitation of colbert-prf is that it relies on clustering and inverse document frequency (idf) statistics for identifying the expansion embeddings -both of which are heuristics. this approach ignores valuable context present in the embeddings, e.g., for the georgia run off elections query, effectiveness might be improved by adding an embedding for 'us', however, this would not likely be selected due to its low idf (indeed, 'us' is also a pronoun, and is often included in stopword lists). moreover, there is no direct connection between the expansion embeddings selected by the heuristic and the semantic search algorithm itself. to overcome these problems, we propose a contrastive weighting method, called cwprf, to select and weight the usefulness of the feedback embeddings for dense expansion. more specifically, for each feedback token, we construct a contrastive objective, where, given positive and negative documents, cwprf is trained to assign high weights to the tokens that are semantically closer to tokens occurring the positive document than to those in the negative document. introducing the prf passages into the training procedure of cwprf enables the model to take the surrounding context into account when identifying the useful tokens from the prf passages. meanwhile, training cwprf with the contrastive objective allows it to learn the effective weights for expansion embeddings that are tailored for the semantic ranking task. figure 1 presents the trade-off between the retrieval effectiveness and the mean prf stage execution time for a variety of existing dense prf techniques on the trec 2019 deep learning track queries, including vector-prf @xcite , ance-prf @xcite , colbert-prf variants @xcite and our proposed cwprf method. as the figure shows, the default colbert-prf implementation outperforms ance-prf and vector-prf in terms of retrieval effectiveness but requires a longer execution time. meanwhile, our proposed cwprf achieves the highest ndcg@10 score without requiring high computational cost. overall, our contributions are summarised as follows: (1) we propose cwprf, a contrastive weighting method for dense query expansion; (2) we construct the contrastive targets and train our cwprf model to assign high expansion weights for tokens that can discriminate the relevant documents from the non-relevant documents. based on the predicted weights, cwprf helps to identify useful expansion embeddings for generating refined query representations; (3) we perform an extensive empirical evaluation and demonstrate how to effectively train our cwprf in a supervised way; (4) experiments show that our cwprf can achieve significantly higher retrieval effectiveness but with less execution time than the default colbert-prf. ## preliminaries given a query q and a document 1 d, we employ the pre-trained colbert @xcite query and document encoders to encode the query and document, respectively. the colbert query and document encoders share weights but are distinguished by the different prepended special tokens. the colbert model is defined as a linear layer upon the raw token embeddings obtained from a bert model: based on the obtained query and document embeddings, the final similarity score between a query and a document, s(q, d), is given by the summation of the highest cosine similarity among the document embeddings for each query embedding: this section first provides an implementation overview of cwprf for dense query expansion in section 3.1. it then details the contrastive weighting method and the training procedure of cwprf in sections 3.2 & 3.3, respectively. ## experimental setup datasets: we conduct our experiments using the ms marco @xcite passage rank-ing dataset. the corpus consists of 8.8m passages from web pages, along which are provided 0.5m training queries with sparse document relevance judgements. we employ the trec deep learning track 2019 query set (43 queries with an average of 215 relevant documents per query) as our validation set and use trec 2020 (54 queries with 211 relevance assessments per query) query set as our test set due to their dense judgements, which can provide more reliable evaluations @xcite . as pseudo-relevance feedback approaches are known not to show a benefit on sparsely judged documents @xcite , we omit the ms marco dev queries. in addition, we also report the performance of cw-prf on four beir @xcite datasets in appendix a.2. we evaluate our method using the official metrics of trec, such as ndcg@10, map@1000 and recall@1000. we follow the standard practice of trec (non-@xmath0. to investigate the extent that semantic matching, rather than exact token matches occurs when retrieving documents, we also report the semantic match proportion (smp) @xcite for the colbert-based system. the calculation of smp is detailed in appendix b. for significance testing, we use the paired t-test (@xmath1.05) and apply the holm-bonferroni multiple testing correction. experimental implementation: both the aaat and oaat training modes are trained using the ms marco "small" triples training set, i.e. the triplets of ⟨q, d + d -⟩. following the settings of colbert @xcite , we use a colbert checkpoint trained using the ms marco passage ranking training triplets for 44k batches. we employ the query encoder from the trained colbert model to encode the query (the maximum query length is set to 32) and the document encoder to encode the pseudo-relevance feedback documents (the maximum document length is set to 512 for the aaat training mode and 180 for the oaat training mode). we set the maximum length to 180 when encoding the positive and negative passages. for ease of notation, we use » to denote a retrieval pipeline, for instance bm25 » colbert indicates applying the colbert reranker on the results obtained from bm25. for setting the hyper-parameters of cwprf, we use the trec 2019 queries as our validation set; the resulting settings of f @xmath2, f @xmath3, as reported later in appendix a.1. however, we note that f @xmath4, f @xmath5. the high β value indicates the high contribution of the cwprf identified expansion embeddings for semantic ranking. we further provide the ablations of performing only the expansion embeddings in appendix a.1. for both cwprf and colbert-prf, we perform 5 sets of experiments with varied random seeds for each variant and report the median results. compared systems: to test the effect of cw-prf, we compare the retrieval effectiveness of a cwprf-based retrieval system with the following 4 families of retrieval approaches: (1) sparse retrieval systems (denoted as sparse in ## results this section studies the effectiveness as well as the efficiency performance of cwprf in section 5.1. the effects of the various training strategies are investigated in section 5.2. we also provide qualitative analysis of cwprf in appendix a.3 and a breakdown performance of cwprf according to various query types in appendix a.4. ## related work dense retrieval models: different from the popular "cross-encoder" based bert-rerankers @xcite , dense retrieval models usually build upon a bert-based "bi-encoder" structure. the query and document are encoded separately into dense representations. there are two families of dense retrieval models: single representation dense retrieval and multiple representation dense retrieval models @xcite . in particular, in the single representation dense retrieval paradigm, exemplified by dpr @xcite or ance @xcite , each query or document is represented into a single dense representation. thus, with the pre-computed document representations, retrieval can be conducted using the nearest neighbour search. in contrast, a multiple representation dense retrieval model encodes each token of the query and document into a dense representation, for instance, colbert model introduced by khattab and zaharia (2020). during retrieval, colbert performs an approximate nearest neighbour search (using faiss @xcite ) for each query embedding, followed by an exact scoring. pseudo-relevance feedback: traditional lexical pseudo-relevance feedback (prf) approaches, such as rm3 @xcite and bo1 @xcite , as well as some recent proposed neural prf models @xcite @xcite are applied upon sparse retrieval. some initial efforts of implementing prf mechanism for dense retrieval have been proposed recently: for instance, colbert-prf @xcite , which is the most similar work to ours, selects cluster centroids as expansion embeddings. different from colbert-prf, where the expansion embeddings are prioritised by the closest token's idf, our work focuses on learning the contextualised weights of the prf tokens and identifies the prominent ones as the expansion tokens that can better differentiate between the positive and negative documents. on the other hand, ance-prf @xcite is a supervised prf approach, which trains an additional query encoder. similar to cwprf-aaat, the query and passages are passed to this new encoder. however, unlike cwprf, ance-prf is trained to produce a new single embedding for the query. due to the nature of its single embedding output, it is infeasible to analyse how the query representation has been refined in ance-prf, while cwprf provides explicit weights for each selected expansion embedding. ## conclusions pseudo-relevance feedback has recently been shown to be effective for dense retrieval. in this work, we propose a deep language model-based contrastive weighting approach (cwprf) for selecting useful query expansion embeddings and calibrating their expansion weights for semantic search. in particular, cwprf is trained with a contrastive objective to learn to assign a high weight for feedback embeddings that can distinguish relevant documents from non-relevant documents. during retrieval, the embeddings of tokens appearing in the feedback documents that cwprf predicts to be important are appended to the query embeddings. extensive experiments performed on two query sets demonstrate that our proposed cwprf approach can significantly outperform the colbert dense retrieval model. in particular, cwprf significantly improves over colbert-prf by 4.1% in terms of ndcg@10 on the trec 2020 query set without requiring high computational cost.