Towards Robust Neural Retrieval with Source Domain Synthetic Pre-Finetuning (2022)

Abstract: Research on neural IR has so far been focused primarily on standard supervised learning settings, where it outperforms traditional term matching baselines. Many practical use cases of such models, however, may involve previously unseen target domains. In this paper, we propose to improve the out-of-domain generalization of Dense Passage Retrieval (DPR) - a popular choice for neural IR - through synthetic data augmentation only in the source domain. We empirically show that pre-finetuning DPR with additional synthetic data in its source domain (Wikipedia), which we generate using a fine-tuned sequence-to-sequence generator, can be a low-cost yet effective first step towards its generalization. Across five different test sets, our augmented model shows more robust performance than DPR in both in-domain and zero-shot out-of-domain evaluation.

## introduction traditional approaches to information retrieval (ir) such as tf-idf @xcite ) and bm25 @xcite rely on lexical matching for query-passage alignment. in contrast, neural ir encodes passages and questions into continuous vector representations, enabling deeper semantic matching. modern neural ir systems @xcite based on pre-trained masked language models (mlm) @xcite typically employ a dual encoder architecture @xcite , where two separate mlms encode the question and the passage. @xcite show that useful weak supervision for such systems can be derived from the related task of machine reading comprehension (mrc) @xcite . their dense passage retrieval (dpr) model demonstrates state-of-the-art (sota) in-domain * work done while at ibm research ai 1 synthetic question generation code is available at: https://github.com/primeqa/primeqa/tree/main/primeqa/qg performance on multiple wikipedia-based datasets @xcite @xcite , outperforming both term matching baselines like bm25 and prior neural approaches, e.g., the inverse cloze task @xcite and latent learning of the retriever during mlm pre-training @xcite . despite its high in-domain utility, however, @xcite show that dpr performance can drop significantly in novel test domains. they propose target domain synthetic data augmentation as a solution to this problem, which augments dpr with additional synthetic training data generated from target domain text. while this approach does indeed improve dpr scores in the new test domain, it has a key practical limitation: for every new target domain, it requires generating a new synthetic training corpus and re-training the model. here we ask if an augmentation approach that only operates once in the source domain, and does not require re-training every time a new test domain is encountered, can also help improve domain generalization. to better understand dpr's zero-shot out-ofdomain (ood) utility, we first run an empirical evaluation where both bm25 and dpr are applied to several out-of-domain test datasets. we observe that (i) dpr still holds an advantage over bm25 in near domain evaluation on wikipediabased datasets, but the difference is considerably lower than in the in-domain case, and (ii) in the far domain of biomedical text, dpr actually underperforms bm25. our ood evaluation is more comprehensive than @xcite , demonstrating the zero-shot utility of dpr in a more detailed and fine-grained manner. next we investigate if a one-off pre-finetuning of dpr with large amounts of source domain synthetic ir data can help improve its robustness to domain shift. utilization of synthetic training data is common in related tasks such as machine reading comprehension (mrc) @xcite ; zhang bart encoder bart decoder nq passage answer sentence answer question (a) bart wikipedia passage synthetic training finetuning nq ... question ... (b) history of tanzania the african great lakes nation of tanzania dates formally from 1964, when it was formed out of the union of the much larger mainland territory of tanganyika and the coastal archipelago of zanzibar. the former was a colony and part of german east africa from the 1880s to 1919, when, under the league of nations, it became a british mandate. it served as a military outpost ... when did tanzania became a country in africa? who owned zanzibar and tanganyika before they were independent? et al. @xcite . nevertheless, a close examination of synthetic pre-finetuning as an augmentation technique is key for zero-shot neural ir due to the presence of highly effective and domain-agnostic term matching baselines like bm25. we fine-tune a sequence-to-sequence generator on labeled mrc data and use it to generate synthetic ir examples from source domain passages ( §2). our experiments show that pre-finetuning dpr with these generated examples does indeed improve its accuracy on both in-domain and out-ofdomain test sets. crucially, the gap with bm25 in far domain evaluation is significantly reduced. the main contributions of this paper are: ## source domain synthetic pre-finetuning in this section, we describe the procedure for synthetic pre-finetuning of the dpr model. we first detail how we train the sequence-to-sequence generator and generate source domain syntheic data from it. next, we describe how this data is used for training the dpr model. let c be a text corpus and @xmath0. an ir example, more specifically a passage retrieval example, consists of a question q and a passage p in d such that p contains an answer a to q. let s be the sentence in p that contains a. we first train an example generator by finetuning bart @xcite )-a pre-trained encoder-decoder language model-to generate an ordered triple (s, a, q) from an input passage p. this procedure in essence uses generation to first identify a candidate sentence s in p, then extract a candidate answer a from s, and finally generate a corresponding question q. in practice, we approximate the generation of s by generating only its first and last words. finally, (q, p) is retained as a synthetic ir example. labeled (p, s, a, q) tuples needed for the supervision of this model are taken from natural questions (nq) @xcite , an existing mrc dataset over wikipedia articles. with the generator, we produce positive synthetic pre-finetuning examples for dpr from wikipedia passages. following sultan et al. (2020), we use top-p top-k sampling @xcite to promote diversity in the generated examples. training and inference of the synthetic example generator are depicted in figures 1a and 1b , respectively. figure 1c shows two example questions output by the generator from a wikipedia passage. to obtain a negative sample for each generated question q, we retrieve passages from wikipedia using bm25 and randomly sample one that does not contain the generated answer a. following @xcite , we also use in-batch negative samples for training. after pre-finetuning with synthetic examples, we fine-tune the model with ir examples derived from nq. we name this synthetically augmented dpr model augdpr. we refer the reader to @xcite for a more detailed description of the dpr training process. ## results and discussion to assess the cross-domain utility of augdpr, we evaluate it zero shot on both near and far domain test sets. to investigate the relative underperformance of neural ir on bioasq, we take a closer look at the vocabularies of the two domains of wikipedia articles and biomedical literature. following @xcite , we compute the overlap between the 10k most frequent tokens (excluding stop words) in the two domains, represented by 3m randomly sampled passages from each. we observe a vocabulary overlap of only 17%, which shows that the two domains are considerably different in terminology, explaining in part the performance drop in our neural models. based on these results, we also believe that performance of neural ir in distant target domains can be significantly improved via pre-finetuning on synthetic examples that are generated from raw text in the target domain. we plan to explore this idea in future work. we also examine the lexical overlap between the questions and their passages, since a high overlap would favor term matching methods like bm25. we find that the coverage of the question tokens in the respective gold passages is indeed higher in bioasq: 72.1%, compared to 58.6% and 63.0% in nq and triviaqa, respectively. to analyze how much synthetic data is required, we experiment with pre-finetuning using 1m and 4m synthetic examples while keeping the number of training updates fixed. as ## conclusion we have shown that pre-finetuning a sota neural ir model using large amounts of source domain synthetic data improves its robustness in zero-shot application settings. our experiments show consistent performance gains on five in-domain and out-of-domain test sets, including a far target domain that has significant vocabulary mismatch with the training domain. future work will explore incorporating more control into the generation of synthetic data to increase its diversity and also to overcome potential biases in finetuning data.