Questions Are All You Need to Train a Dense Passage Retriever (2023)

Abstract: We introduce ART, a new corpus-level autoencoding approach for training dense retrieval models that does not require any labeled training data. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples. ART, in contrast, only requires access to unpaired inputs and outputs (e.g., questions and potential answer passages). It uses a new passage-retrieval autoencoding scheme, where (1) an input question is used to retrieve a set of evidence passages, and (2) the passages are then used to compute the probability of reconstructing the original question. Training for retrieval based on question reconstruction enables effective unsupervised learning of both passage and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning. Extensive experiments demonstrate that ART obtains state-of-the-art results on multiple QA retrieval benchmarks with only generic initialization from a pre-trained language model, removing the need for labeled data and task-specific losses.1 Our code and model checkpoints are available at: https://github.com/DevSinghSachan/art .

## introduction dense passage retrieval methods @xcite , initialized with encoders such as bert @xcite and trained using supervised contrastive losses @xcite , have surpassed the performance achieved by previously popular keyword-based approaches like bm25 @xcite . such retrievers are core components in models for open-domain tasks, such as open qa, where state-of-the-art methods typically require large supervised datasets with custom hardnegative mining and denoising of positive examples. in this paper, we introduce the first unsupervised method, based on a new corpus-level autoencoding approach, that can match or surpass strong supervised performance levels with no labeled training data or task-specific losses. we propose art, autoencoding-based retriever training, which only assumes access to sets of unpaired questions and passages. given an input question, art first retrieves a small set of possible evidence passages. it then reconstructs the original question by attending to these passages (see figure 1 for an overview). the key idea in art is to consider the retrieved passages as a noisy representation of the original question and question reconstruction probability as a way of denoising that provides soft-labels for how likely each passage is to have been the correct result. to bootstrap the training of a strong model, it is important to both have a strong initial retrieval model and to be able to compute reliable initial estimates of question reconstruction probability when conditioned on a (retrieved) passage. although passage representations from bertstyle models are known to be reasonable retrieval baselines, it is less clear how to do zero-shot question generation. we use a generative pre-trained language model (plm) and prompt it with the passage as input to generate the question tokens using teacher-forcing. as finetuning of the questiongeneration plm is not needed, only the retrieval model, art can use large plms and obtain accurate soft-label estimates of which passages are likely to be the highest quality. the retriever is trained to penalize the divergence of a passage likelihood from its soft-label score. for example, if the question is ''where is the bowling hall of fame located?'' as shown in figure 1 , then the training process will boost the retrieval likelihood of the passage ''bowling hall of fame is located in arlington,'' as it is relevant and would lead to a higher question reconstruction likelihood, while the likelihood of the passage ''hall of fame is a song by ...'' would be penalized as it is irrelevant. in this manner, the training process encourages correct retrieval results and vice-versa, leading to an iterative improvement in passage retrieval. comprehensive experiments on five benchmark qa datasets demonstrate the usefulness of our proposed training approach. by simply using questions from the training set, art outperforms models like dpr by an average of 5 points absolute in top-20 and 4 points absolute in top-100 accuracy. we also train using all the questions contained in the natural questions (nq) dataset @xcite and find that even with a mix of answerable and unanswerable questions, art achieves strong generalization on outof-distribution datasets due to relying on plm. our analysis further reveals that art is highly sample-efficient, outperforming bm25 and dpr with just 100 and 1000 questions, respectively, on the nq-open dataset, and that scaling up to larger retriever models consistently improves performance. ## experimental setup in this section, we describe the datasets, evaluation protocol, implementation details, and baseline methods for our passage retrieval experiments. ## conclusions and future work we introduced art, a novel approach to train a dense passage retriever using only questions. art does not require question-passage pairs or hardnegative examples for training and yet achieves state-of-the-art results. the key to making art work is to optimize the retriever to select relevant passages such that conditioning on them, the question generation likelihood computed using a large pre-trained language model iteratively improves. despite requiring much less supervision, art substantially outperforms dpr when evaluated on multiple qa datasets and also generalizes better on out-of-distribution questions. art presents several directions for future work. it would be interesting to apply this approach in low-resource retrieval including multilingual @xcite and cross-lingual question answering @xcite . our training framework can also be extended to train cross-modality retrievers such as for image or code search @xcite using textual queries. finally, other directions worth exploring would be to make use of labeled data when available such as by finetuning plm on passage-question aligned data and to train multi-vector retrievers @xcite with art.