Multi-Task Retrieval for Knowledge-Intensive Tasks (2021)

Abstract: Retrieving relevant contexts from a large corpus is a crucial step for tasks such as open-domain question answering and fact checking. Although neural retrieval outperforms traditional methods like tf-idf and BM25, its performance degrades considerably when applied to out-of-domain data. Driven by the question of whether a neural retrieval model can be _universal_ and perform robustly on a wide variety of problems, we propose a multi-task trained model. Our approach not only outperforms previous methods in the few-shot setting, but also rivals specialised neural retrievers, even when in-domain training data is abundant. With the help of our retriever, we improve existing models for downstream tasks and closely match or improve the state of the art on multiple benchmarks.

## introduction knowledge-intensive tasks is the common designation for a class of real-world nlp problems which, because of their nature, require large amounts of knowledge about the world @xcite . for example, open-domain question answering requires producing answers to general factoid questions; fact checking involves determining the veracity of claims based on a database of trusted evidence. practical solutions to these tasks usually involve an efficient retrieval component that, given an input query, selects a limited subset of relevant information from a large knowledge source. sophisticated downstream models then consider the input only in the context of the retrieved information, and perform the final task. 1 the standard retrieval component in many systems (e.g., @xcite @xcite has long relied on term-matching methods, such as tf-idf or bm25 @xcite . these methods rely on efficient algorithms and usually perform reasonably well regardless of the problem. in contrast, recent neural retrieval models, such as ict @xcite , dpr @xcite and rag @xcite achieve better results by learning directly from task-specific training data and going beyond simple keyword matching. while task specialisation results in improved task performance, researchers have observed that a retriever trained for one specific domain will typically achieve low out-of-domain performance, and even lower performance on entirely different tasks @xcite . this has two implications. first, unlike tfidf or bm25, neural retrieval models are unsuitable for low data regimes such as few-and zero-shot settings. second, task-specific retrievers complicate practical applications where multiple knowledgeintensive tasks may need to be performed using the same supporting database or over the same input text. it may not be practical to deploy multiple separate specialised models due to computational performance or memory concerns. in this work, we ask the following question: can we develop a universal neural retriever? namely, we target a retriever which can perform well on a wide variety of problems without domain-specific training, but which -if additional in-domain labelled data is available -can be further fine-tuned to improve its performance. we perform a large experimental study to attempt to build such a universal retrieval model. we find that, by jointly training on an extensive selection of retrieval tasks, we obtain a model which is not only more robust than previous approaches, but also can lead to better performance on the downstream knowledge-intensive tasks when plugged into an existing system. our approach combines the benefits from ir-based models with those of task-specific neural retrievers -namely, good performance when no (or not enough) training data is available and high task performance due to its ability to learn highly specialised representations. our contributions can be summarised as follows. ## background in this section, we first give an overview of retrieval methods based on sparse and dense representations. we then discuss a wide range of knowledgeintensive nlp tasks, where retrieval plays a crucial role in solving the problems. ## related work the approach most closely related to ours is dpr @xcite , upon which we built all our retrievers. it is covered in detail, along with historical context, in § 2.1. another closely related approach is the retrieval-augmented generation (rag) model of @xcite . in its base configuration it augments dpr with a generative reader, and trains the query encoder end-to-end (differing from traditional retriever-reader architectures, which treat the two steps as disjoint). a natural extension of our work would be to combine rag with the multi-task learning approach, to study whether it can lead to further gains in performance or robustness. a number of promising techniques to boost retrieval performance have been proposed recently. these are orthogonal to our work, and as such they could be combined with it. amongst these, pretraining methods form one class. inverse cloze task @xcite and its extensions @xcite are self-supervised pre-training methods designed for retrieval in open-domain question answering. whether such specific pre-training is beneficial to tasks other than question answering remains an open question. cert @xcite is an alternative pre-training approach, inspired by some recent advances in computer vision. while to our knowledge this has not been applied to retrieval problems, we believe it might be promising due to its focus on sentence-level semantics (as opposed to the more standard masked language modelling pre-training, which focuses on the token level). another class of orthogonal improvements to dense retrieval involves models which embed passages into multiple fixed-size vectors. of these, colbert @xcite and me-bert @xcite are two representative examples. one further approach is colbert-qa @xcite , which additionally uses a data augmentation strategy closely related to our own approach described in appendix d. retrieval does not strictly have to be performed with a model which contains an explicit memory. large-scale pre-trained models have been shown to store knowledge directly into their parameters. a model which demonstrates this ability is t5 @xcite ) -which we used as a baseline in § 4. regarding the multi-task aspect of our approach, a related strategy has been demonstrated by @xcite . in this recent work, the authors multi-task train a pre-trained model on around 50 datasets, before performing the final fine-tuning. while they do not focus on retrieval, their results are consistent with ours and show that multi-task training leads to improved performance and increased sample efficiency. on the topic of question answering, @xcite show in a recent notable paper that, for several popular qa datasets, a portion of questions in the test set has near-duplicates in the training sets, and the same holds true for an even larger set of answers. to our knowledge, similar analyses have yet to be performed on the other kilt tasks. finally two entity linkers, genre (de cao et al., 2020) and blink @xcite , are worth mentioning. being trained specifically for entity linking, these models will generally outperform retrieval-based approaches on that task. while they are not comparable to retrieval models and will not generally be applicable to information retrieval tasks, we cite them here to provide readers with a fuller context of the existing literature on related tasks. the study started with the question of whether the way in which information is embedded for retrieval purposes is universal. §4.2 provided evidence that to a large extent it is, with a single "universal" retriever, trained jointly on 8 datasets, often performing comparably to task-specific models. armed with this knowledge, in §4.3 we plugged our single model in a larger pipeline, in order to see its contribution to the downstream performance on a wide range of knowledge-intensive tasks. this led to an overall improvement in downstream performance, setting new top results for a number of tasks in the kilt benchmark. next, in §4.4, we evaluated the model's performance in the zero-shot and few-shot settings. by evaluating on a wide range of tasks, we were able to show that our proposed approach performs comparably to bm25 in the zero shot setting, and quickly overtakes it even with minimal in-domain training. in the appendices, readers interested in getting a fuller picture will find further experiments. namely, in appendix b we test two more complex variants of the model involving task specialisation, but fail to see clear performance improvements. in appendix d we show how a simple iterative approach to data augmentation, easily applied to our base approach, can lead to better performance throughout. we provide a pre-trained snapshot of our bestperforming model, in the form of a bert checkpoint. 3 as shown, this model will be useful in zero-shot and few-shot settings as a better performing alternative to both ir-based approaches such as bm25, as well as task-specific models. the multitask training approach demonstrated here can also be useful in industry settings where several retrieval operations may need to be performed on the same piece of content, 4 and the deployment of multiple task-specific models might not be possible due to space or computational performance concerns.