GPL : Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval (2022)

Abstract: Dense retrieval approaches can overcome the lexical gap and lead to significantly improved search results. However, they require large amounts of training data which is not available for most domains. As shown in previous work (Thakur et al., 2021b), the performance of dense retrievers severely degrades under a domain shift. This limits the usage of dense retrieval approaches to only a few domains with large training datasets. In this paper, we propose the novel unsupervised domain adaptation method Generative Pseudo Labeling (GPL), which combines a query generator with pseudo labeling from a cross-encoder. On six representative domain-specialized datasets, we find the proposed GPL can outperform an out-of-the-box state-of-the-art dense retrieval approach by up to 9.3 points nDCG@10. GPL requires less (unlabeled) data from the target domain and is more robust in its training than previous methods. We further investigate the role of six recent pre-training methods in the scenario of domain adaptation for retrieval tasks, where only three could yield improved results. The best approach, TSDAE (Wang et al., 2021) can be combined with GPL, yielding another average improvement of 1.4 points nDCG@10 across the six tasks. The code and the models are available at https://github.com/UKPLab/gpl .

## introduction information retrieval (ir) is a central component of many natural language applications. traditionally, lexical methods @xcite have been used to search through text content. however, these methods suffer from the lexical gap @xcite and are not able to recognize synonyms and distinguish between ambiguous words. recently, information retrieval methods based on dense vector spaces have become popular to address these challenges. these dense retrieval methods map queries and passages foot_0 to a shared, dense vector space and retrieve relevant hits by nearest-neighbor search. significant improvement over traditional approaches has been shown for various tasks @xcite . this method is also adapted increasingly by industry to enhance the search functionalities of various applications @xcite . however, as shown in @xcite , dense retrieval methods require a large amount of training data to work well. 3 most importantly, dense retrieval methods are extremely sensitive to domain shifts: models trained on ms marco perform rather poorly for questions for covid-19 scientific literature @xcite . the ms marco dataset was created before covid-19, hence, it does not include any covid-19 related topics and models did not learn how to represent this topic well in a vector space. in this work, we present generative pseudo labeling (gpl), an unsupervised domain adaptation technique for dense retrieval models (see figure 1 ). for a collection of paragraphs from the desired domain, we use an existing pre-trained t5 encoderdecoder to generate synthetic queries. these input passages are viewed as the positive passages for the generated queries. for each generated query, we retrieve the most similar paragraphs as the negative passages using an existing dense retrieval model. we term this step negative mining and term these negative passages hard negatives. finally, we use an existing cross-encoder to score each (query, passage)-pair and train a dense retrieval model on these generated, pseudo-labeled queries using marginmse-loss @xcite . we use publicly available models for query generation, negative mining, and the cross-encoder, which have been trained on the ms marco dataset @xcite , a large-scale dataset from bing search logs combined with relevant passages from diverse web sources. we evaluate gpl on six representative domain-specific datasets from the beir benchmark @xcite . gpl improves the performance by up to 9.3 points ndcg@10 compared to state-of-the-art model trained solely on ms marco. compared to the previous state-of-the-art domain-adaptation method qgen @xcite , gpl improves the performance by up to 4.5 ndcg@10 points. training with gpl is easy, fast, and data efficient. we further analyze the role of six recent pretraining methods in the scenario of domain adaptation for retrieval tasks. the best approach is tsdae @xcite , that outperforms the second best approach, masked language modeling @xcite on average by 2.5 points ndcg@10. tsdae can be combined with gpl, yielding another average improvement of 1.4 point ndcg@10. ## related work pre-training based domain adaptation. the most common domain adaptation technique for transformer models is domain-adaptive pretraining @xcite , which continues pre-training on in-domain data before fine-tuning with labeled data. however, for retrieval it is often difficult to get in-domain labeled data and models are applied in a zero-shot setting on a given corpus. besides masked language modeling (mlm) @xcite , different pre-trained strategies specifically for dense retrieval methods have been proposed. inverse cloze task (ict) @xcite generates query-passage pair by randomly selecting one sentence from the passage as the query and the other part as the paired passage. condensor (cd) @xcite applies mlm on top of the cls token embedding from the final layer and the other context embeddings from a previous layer to force the model to learn meaningful cls representation. simcse @xcite passes the same input twice through the network with different dropout masks and min-imizes the distance of the resulting embeddings, while contrastive tension (ct) @xcite passes the input through two different models. tsdae @xcite uses a denoising auto-encoder architecture for representation learning: words from the input text are removed and passed through an encoder to generate a fixed-sized embedding. a decoder must reconstruct the original text without noise. as we show in appendix e, just using these unsupervised techniques is not sufficient and the resulting models perform poorly. so far, ict and cd have only been studied on in-domain performance, i.e. a large in-domain labeled dataset is available which is used for subsequent supervised fine-tuning. simcse, ct, and tsdae have been only studied for unsupervised sentence embedding learning. as our results show in appendix e, they do not work at all for purely unsupervised dense retrieval. if these pre-training approaches can be used for unsupervised domain adaptation for dense retrieval was so far unclear. in this work, we transfer the setup from @xcite to dense retrieval and first pre-train on the target corpus, followed by supervised training on labeled data from ms marco (nguyen et al., 2016) 4 . performance is then measured on the target corpus. query generation. query generation has been used to improve retrieval performances. doc2query @xcite expands passages with predicted queries, generated by a trained encoderdecoder model, and uses traditional bm25 lexical search. this performed well in the zero-shot retrieval benchmark beir @xcite . @xcite proposes qgen, that uses a query generator trained on general domain data to synthesize domain-targeted queries for the target corpus, on which a dense retriever is trained from scratch. as a concurrent work, @xcite al. (2020) also proposes the similar method. following this idea, @xcite views qgen as a posttraining method to adapt powerful ms marco retrievers to the target domains. despite the success of qgen, previous methods only consider the cross-entropy loss with in-batch negatives, which provides coarse-grained relevance and thus limits the performance. in this work, we show that extending this approach by using pseudolabels from a cross-encoder together with hard neg- query generation via t5 negative mining via dense retrieval pseudo labeling via cross-encoder what is python python is … what is python ❌ ❌ java is … what is python 10.3 0.3 6.2 2.0 python is … ❌ ❌ java is … atives can boost the performance by several points ndcg@10. other methods. recently, @xcite proposes modir to use domain adversarial training (dat) @xcite for unsupervised domain adaptation of dense retrievers. modir trains models by generating domain invariant representations to attack a domain classifier. however, as argued in @xcite , dat trains models by minimizing the distance between representations from different domains and such learning objective can result in bad embedding space and unstable performance. for sentiment classification, @xcite proposes udalm based on multiple stages of training. udalm first applies mlm training on the target domain; and it then applies multi-task learning on the target domain with mlm and on the source domain with a supervised objective. however, as shown in section 5, we find this method cannot yield improvement for retrieval tasks. pseudo labeling and cross-encoders: bi-encoders map queries and passage independently to a shared vector space from which the querypassage similarity is computed. in contrast, crossencoders @xcite ) work on the concatenation of the query and passage and predict a relevance score using cross-attention between query and passage. this can be used in a re-ranking setup @xcite , where the relevancy is predicted for all query-passage-pairs for a small candidate set. previous work has shown that cross-encoders achieve much higher performances @xcite @xcite and are less prone to domain shifts @xcite . but cross-encoders come with an extremely high computational overhead, making them less suited for a production set-ting. transferring knowledge from cross-encoder to bi-encoders have been shown previous for sentence embeddings @xcite and for dense retrieval: hofstätter et al. (2020) predict cross-encoder scores for (query, positive)-pairs and (query, negative)-pairs and learns a bi-encoder to predict the margin between the two scores. this has been shown highly effective for in-domain dense retrieval. ## method this section describes our proposed generative pseudo labeling (gpl) method for the unsupervised domain adaptation of dense retrievers. figure 1 illustrates the idea of gpl. for a given target corpus, we generate for each passage three queries (cf. we use the synthetic dataset d @xmath0, p i , p - i , δ i )} i with the marginmse loss (hofstätter et al., 2020) for training a domain-adapted dense retriever that maps queries and passages into the shared vector space. our method requires from the target domain just an unlabeled collection of passages. further, we use use pre-existing t5-and cross-encoder models that have been trained on the ms marco passages dataset. query generation: to enable supervised train-ing on the target corpus, synthetic queries can be generated for the target passages using a query generator trained on a different, existing dataset like ms marco. previous work qgen @xcite used the simple multiplenegativesranking (mnrl) loss @xcite with in-batch negatives foot_3 to train the model: marginmse loss: multiplenegativesranking loss considers only the coarse relationship between queries and passages, i.e. the matching passage is considered as relevant while all other passages are considered irrelevant. however, the query encoder is not without flaws and might generate queries that are not answerable foot_4 by the passage. further, other passages might actually be relevant as well for a given query, which is especially the case if training is done with hard negatives as we do it for gpl. in contrast, marginmse loss @xcite ) uses a powerful cross-encoder to soft-label (query, passage) pairs. it then teaches the dense retriever to mimic the score margin between the positive and negative query-passage pairs. formally, here the dot-product is usually used due to the infinite range of the cross-encoder scores. this loss is a critical component of gpl, as it solves two major issues from the previous qgen method: a badly generated query for a given passage will get a low score from the cross-encoder, hence, we do not expect the dense retriever to put the query and passage close in the vector space. a false negative will lead to a high score from the cross-encoder, hence, we do not force the dense retriever to assign a large distance between the corresponding embeddings. in section 6.3, we show that gpl is a lot more robust to badly generated queries than the previous qgen method. ## experiments in this section, we describe the experimental setup, the datasets used and the baselines for comparison. ## analysis in this section, we analyze the influence of training steps, corpus size, query generation and choices of starting checkpoints on gpl. ## case study: fine-grained labels gpl uses continuous pseudo labels from a crossencoder, which can provide more fine-grained information and is more informative than the simple 0-1 labels as in qgen. in this section, we give a more detailed insight into it by a case study. one example from fiqa is shown in ## conclusion in this work we propose gpl, a novel unsupervised domain adaptation method for dense retrieval models. it generates queries for a target corpus and pseudo labels these with a cross-encoders. pseudolabeling overcomes two important short-comings of previous methods: not all generated queries are of high quality and pseudo-labels efficiently detects those. further, training with mined hard negatives is possible as the pseudo labels performs efficient denoising. in this work, we also evaluated different pre-training strategies in a domain-adaptive pretraining setup: we first pre-trained on the target domain, then performed supervised training on ms marco. ict and mlm were able to yield a small improvement (@xmath1.5 ndcg@10 points on average), while tsdae was able to yield a significant improvement of 4 ndcg@10 points on average. other approaches degraded the performance.