Domain-matched Pre-training Tasks for Dense Retrieval (2022)

Abstract: Pre-training on larger datasets with ever increasing model size isnow a proven recipe for increased performance across almost all NLP tasks.A notable exception is information retrieval, where additional pre-traininghas so far failed to produce convincing results. We show that, with theright pre-training setup, this barrier can be overcome. We demonstrate thisby pre-training large bi-encoder models on 1) a recently released set of 65 millionsynthetically generated questions, and 2) 200 million post-comment pairs from a preexisting dataset of Reddit conversations made available by pushshift.io. We evaluate on a set of information retrieval and dialogue retrieval benchmarks, showing substantial improvements over supervised baselines.

## introduction as a pre-training task, language modeling and its variants (causal @xcite , bi-directional @xcite , masked @xcite , seq2seq @xcite ) have proven to be extremely versatile and shown to transfer well to most, if not all nlp tasks. nevertheless, in-domain fine tuning remains important, as there is still a gap between the pre-training task and the downstream tasks. numerous approaches have been proposed to fill this gap, with an additional (intermediate) pre-training stage, mostly based on multi-task learning @xcite . it's been generally accepted that the more similar the end task is to the pre-training task, the larger the gains (e.g., nli tasks transfer better to other nli tasks @xcite , qa tasks to qa tasks @xcite , inter alia). from this perspective, information retrieval (ir), which is the task of identifying the most relevant document to a given query from a large corpus of * equal contribution candidates, has a unique position. at the surface, ir looks similar to other nlp tasks in standard benchmarks, such as nli or paraphrase detection. however, the need to accommodate large corpora imposes computational constraints, which leads to important practical differences. most importantly, indexing needs to happen offline, therefore the candidate representations need to be calculated independently of the query representation. as a result, neural retrieval systems typically use a bi-encoder model (figure 1 ), trained to minimize the similarity between the document representation and the query representation. this shallow interaction between document and query encoders makes neural ir models architecturally unique, compared to the block cross-attention transformers which are the universal choice for almost every other nlp task. researchers have therefore recognized the need to construct intermediate pre-training tasks that are better matched to retrieval. @xcite proposed the inverse cloze task (ict), which treats sentences as pseudo-queries, and matches them to the paragraph they originate from. @xcite combined this with body first selection (bfs) (selecting the first paragraph given a sentence from the same document), and wiki link prediction. @xcite pre-trained a retrieval model jointly in an end-to-end system to minimize a language modelling objective. in each of these cases, pre-training approaches were shown to improve over their respective baselines. however, subsequent work showed that a careful fine-tuning over a vanilla bert model can outperform all of these approaches @xcite . the findings for model scaling are also similar to those of data scaling. published results show only modest improvements from larger models for retrieval, and retrieval models which top the most competitive document ranking leaderboards are still based on the relatively small bert-base architecture. 1 this is in sharp contrast to other nlp benchmarks, where data and model scaling has been extremely successful. we hypothesise that previously proposed pretraining tasks might be still too distant from the target task, which limits useful transfer. we therefore investigate pre-training tasks for retrieval which are as closely matched to the the target task and domain as possible. to this end, we propose using two corpora for retrieval pre-training: we conduct extensive evaluations on two popular information retrieval tasks, a benchmark composed of 8 knowledge-intensive retrieval tasks, and 3 dialogue retrieval benchmarks. we find that pretraining leads to strong improvements in all cases, and also demonstrate robust generalization. we compare different pre-training tasks, investigating the effect of domain and task similarity, and find both to be important. we also experiment with models of varying sizes, with and without pre-training, showing in some cases that retrieval can indeed benefit from larger models. ## dense retrieval in this section we give an overview of dense retrieval models and how they are trained. ## main results in this section, we summarize our main results, before we dive into some analysis in the next section. ## conclusion we have investigated domain-matched pre-training tasks for bi-encoder dense retrieval models. we found that the proposed approach is more effective than previously proposed artificial pre-training tasks. we demonstrated the generality of our conclusions, by evaluating on a large and varied set of passage retrieval and dialogue retrieval benchmarks. our work should be considered as a new statement in the ongoing dialogue of how to best train dense retrieval models. we believe we have addressed some important open questions, such as whether and when pre-training can be useful. however we have also raised new questions, in addition to the many which remain open. for instance, many different ways of leveraging reader models for better retrieval have been recently proposed, including end-to-end training, distillation, data filtering and data augmentation. what is the relationship between these approaches? are they complementary? which ones are more efficient, and more performant? we believe these questions deserve a more thorough investigation. we have focused mostly on dense retrieval when full supervision is available, and showed that for @xmath0, the performance is already approaching a ceiling. there is more room for improvement for smaller k. in this regime, however, re-ranking models also become feasible and separable architecture is not a strict requirement. therefore, further improvements to retrieval will likely need to be discussed with more emphasis on the computation-accuracy trade-off. few-shot and zero-shot retrieval will also be of increasing importance, and there are already works investigating this direction @xcite .