Auto-Encoding Questions with Retrieval Augmented Decoding for Unsupervised Passage Retrieval and Zero-Shot Question Generation (2023)

Abstract: Dense passage retrieval models have become state-of-the-art for information retrieval on many Open-domain Question Answering (ODQA) datasets. However, most of these models rely on supervision obtained from the ODQA datasets, which hinders their performance in a low-resource setting. Recently, retrieval-augmented language models have been proposed to improve both zero-shot and supervised information retrieval. However, these models have pre-training tasks that are agnostic to the target task of passage retrieval. In this work, we propose Retrieval Augmented Auto-encoding of Questions for zero-shot dense information retrieval. Unlike other pre-training methods, our pre-training method is built for target information retrieval, thereby making the pre-training more efficient. Our method consists of a dense IR model for encoding questions and retrieving documents during training and a conditional language model that maximizes the question’s likelihood by marginalizing over retrieved documents. As a by-product, we can use this conditional language model for zero-shot question generation from documents. We show that the IR model obtained through our method improves the current state-of-the-art of zero-shot dense information retrieval, and we improve the results even further by training on a synthetic corpus created by zero-shot question generation.

## introduction open domain question answering (odqa) with dense passage retrieval has been quite successful in recent years. this is primarily because of the availability of large question-answering corpora. however, annotations for the creation of open-domain question answering (odqa) datasets consume significant time and effort, although, the indispensable need for labeled data is evident in the decline of cross-domain performance across various odqa datasets @xcite for both information retrieval and question-answering tasks. to this end, in this work, we address the task of unsupervised dense passage retrieval (udpr). that is, to be able to retrieve relevant documents without the labels on ground truth question-passage pairs, which reflects a real-world scenario. in this work, we propose retrieval augmented auto-encoding of questions (named as autoqir foot_0 ) as a means to obtain similarity between documents and questions to perform zero-shot dense passage retrieval. our method not only complements the supervised methods but unlike other zero-shot pretrained models, it also considers a pre-training task that is directly relevant to questions. the following are the contributions of this work: traditionally, lexical models with sparse vector spaces, such as bm25 @xcite , have been used for unsupervised retrieval of the neighboring documents of a query. these models consider documents and queries as bags of words and rely upon possible word overlap between the query and the relevant document to assign a high cosine similarity between them. consequently, they suffer from the problem of the lexical gap between query and document @xcite and are unable to capture the meaning that comes through word order. alternatively, dense passage 2 retrieval models capture the meaning by encoding the sequence of words. hence, unsupervised methods using dense passage retrieval can potentially yield better recall than the sparse retrieval models. recently, several pre-training methods have been proposed to improve the joint dense embedding space of queries and documents. retrieval augmented pre-training and fine-tuning methods @xcite have been shown to improve dense passage retrieval. these methods train an information retrieval model to improve the context required for adjoining pre-training tasks. amongst these, @xcite showed the ability for unsupervised dense passage retrieval while pre-training on masked language modeling. @xcite used a contrastive loss to discriminate between positive and negative documents while considering pseudo questions. while these methods are effective, the pre-training task chosen in their approach lacks explicit adaptation for the target task of passage retrieval for queries. estimating the likelihood of the question given a context is useful in various steps of question answering and information retrieval tasks. for example, @xcite maximized question likelihood (by decomposing the posterior probability) instead of answer likelihood and showed that qa models relying on question likelihood are robust to perturbations in the input. another approach @xcite used unsupervised question generation methods to augment data for extractive question answering. @xcite used auto-encoding of questions for unsupervised answer span selection. it is shown by @xcite that pre-trained language models can be used to re-rank the retrieved documents via 'prompt-based' question likelihood. furthermore, parallel to our work, @xcite have proposed that the retrieved documents @xcite can be used to finetune a retriever by a 2 please note that we use the terms document and passage interchangeably throughout this paper teacher-student network. in their approach, the ground-truth distribution of the documents given a question is derived from the output of a frozen large pre-trained language model (> 3b parameters). the dense retriever is trained by minimizing the kl divergence between its estimated distribution with the aforementioned ground truth distribution of the teacher network. the main difference between our work and theirs is that we utilize an auto-encoding loss while fine-tuning a bart decoder (406m parameters), thereby avoiding sole reliance on pre-existing (large) language models. consequentially, our model can perform zero-shot question generation in addition. ## approach maximizing the likelihood of question given a context has been proven useful for information retrieval and question answering tasks @xcite @xcite . however, in an unsupervised setup, we don't have access to ground truth questions associated with passages. to mitigate this, we propose auto-encoding of questions by assuming an underlying conditional distribution over documents. in other words, our approach seeks to maximize the likelihood of a question by first obtaining the relevant passages. for this, we take the approach proposed by @xcite . our training setup requires a set of questions q and a set of documents s and no further labels for answers or relevant documents. note that both sets of q and s can be obtained without human annotations, for example, via web crawling. our only assumption is that the set s contains relevant documents to most of the questions in set q. without this assumption, we model a uniform conditional distribution over documents. this expectation of relevant documents in a document corpus is fairly common in situations where an information retrieval task ought to be performed. formally, we aim to reconstruct the input question by assuming document z, as a latent variable. the loss l is obtained as the negative loglikelihood of the reconstructed question q given the input question q, as shown in eq. 1. the probability p(q|q) can be further decomposed by marginalizing over all known documents in the corpus s as shown in eq. 2. the input q for the conditional language model may provide an unwanted strong signal during reconstruction. this will lead to over-fitting of the decoder and a weak encoder. hence, we relax this term to p(q|z i ) by removing the dependency on input question q. furthermore, the sum in eq. 2 is intractable to compute especially when the set s is very large. also, note that when s is very large, most of the documents will have probabilities close to zero. to mitigate this, we approximate the sum by taking top-k documents. similar to @xcite , our method mainly consists of two components: a passage retriever and a sequence-to-sequence generator. the equations below describe our loss function: ## conclusion in this work, we propose a novel pre-training task to perform unsupervised information retrieval. our method, which is based on retrieval augmented generation @xcite , shows significant improvements from the baseline zero-shot retrieval models (ict and realm). our crossdomain evaluation reveals the significance of using target questions for pre-training. we also show that auto-encoding on questions has a much greater impact than auto-encoding of sentences (ict). our model explicitly captures knowledge stored in language models into ir models. additionally, our method can be used for zero-shot question generation which can further provide data augmentation for ir corpora. in the future, it would be interesting to investigate whether unfreezing the context encoder during training would lead to improved retriever performance.