Improving Passage Retrieval with Zero-Shot Question Generation (2022)

Abstract: We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot question generation model, which uses a pre-trained language model to compute the probability of the input question conditioned on a retrieved passage. This approach can be applied on top of any retrieval method (e.g. neural or keyword-based), does not require any domain- or task-specific training (and therefore is expected to generalize better to data distribution shifts), and provides rich cross-attention between query and passage (i.e. it must explain every token in the question). When evaluated on a number of open-domain retrieval datasets, our re-ranker improves strong unsupervised retrieval models by 6%-18% absolute and strong supervised models by up to 12% in terms of top-20 passage retrieval accuracy. We also obtain new state-of-the-art results on full open-domain question answering by simply adding the new re-ranker to existing models with no further changes.

## introduction text retrieval is a core sub-task in many nlp problems, for example, open-domain question answering where a document must be retrieved and then read to answer an input query. queries and documents are typically embedded in a shared representation space to enable efficient search, before using a task-specific model to perform a deeper, tokenlevel document analysis (e.g. a document reader that selects an answer span). we show that adding a zero-shot re-ranker to the retrieval stage of such models leads to large gains in performance, by doing deep token-level analysis with no task-specific data or tuning. @xcite top-1000 passages, we outperform strong supervised models like dpr @xcite on natural questions and triviaqa datasets. we focus on open-domain question answering and introduce a re-ranker based on zero-shot question generation with a pre-trained language model. our re-ranker, which we call unsupervised passage re-ranker (upr), re-scores the retrieved passages by computing the likelihood of the input question conditioned on a retrieved passage. 2 this simple method enables task-independent cross-attention between query and passage that can be applied on top of any retrieval method (e.g. neural or keywordbased) and is highly effective in practice (figure 1 ). in part, upr is inspired by the traditional models of query scoring with count-based language models @xcite . however, instead of estimating a language model from each passage, upr uses pre-trained language models (plms). more recent work on re-rankers have finetuned plms on question-passage pairs to generate relevance labels @xcite , sometimes to jointly generate question and relevance labels @xcite . in contrast, upr uses off-the-shelf plms, does not require any training data or finetuning, and still leads to strong performance gains (figure 1 ). comprehensive experiments across a wide range of datasets, retrievers, and plms highlight the strengths of upr: to the best of our knowledge, this is the first work to show that a fully unsupervised pipeline (consisting of a retriever and re-ranker) can greatly outperform supervised dense retrieval models like dpr @xcite . as language models continue to improve rapidly @xcite , the performance of upr may see corresponding gains over time. upr requires no annotated data and uses only generic pre-trained models, which means it may be easy to apply to a wide range of retrieval problems. ## method figure 2 presents an overview of our approach for open-domain retrieval, which introduces a new unsupervised re-ranker (sec 2.2) that can be applied to any existing text retriever (sec 2.1). ## experimental setup in this section, we describe the datasets, unsupervised and supervised retrievers, and language models used for our passage re-ranking experiments. ## experiments: passage retrieval we evaluate the performance of our proposed unsupervised passage re-ranker (upr), conduct ablations to better understand the approach, evaluate robustness on challenging test collections, and discuss run-time efficiency. our goal is to improve the rankings of top-{20, 100} passages. hence, in the first stage, a larger candidate list is fetched by retrieving the top-1000 passages. then, in the second stage, these passages are re-ranked with the t0-3b plm unless specified otherwise. to evaluate upr performance, we compute the conventional top-k retrieval accuracy metric. it is defined as the fraction of questions for which at least one passage within the top-k passages contains a span that matches the humanannotated answer to the question. ## experiments: question answering finally, we show that upr improves the performance of full open-domain qa systems. ## related work our work is based on re-ranking passages for opendomain retrieval using pre-trained language models (plms) which we have covered in earlier sections. here, we instead focus on covering previous work related to generative pre-training, query likelihood for document ranking, and open-domain qa. ## conclusions and future work in this work, we propose upr, an approach to perform unsupervised passage re-ranking for opendomain retrieval. to re-rank, upr computes a relevance score for question generation conditioned on each retrieved passage using pre-trained language models. extensive experiments across a wide range of qa datasets show that an unsupervised pipeline consisting of retriever and upr greatly outperforms strong supervised retriever models. in addition, upr further improves the performance of supervised retrievers. on the open-domain qa task, by just performing inference using re-ranked passages and a pre-trained reader model, we achieve new state-of-the-art results. upr presents several interesting directions for future work. first, its applications to other retrieval tasks such as improving source-code retrieval based on textual queries can be explored. second, another promising direction would be to tune instructions according to the nature of the retrieval tasks. for instance, when retrieving similar sentences in the beir benchmark, variations of the instruction prompt used in upr can be explored. finally, it would also be interesting to investigate the extent to which specialized language models such as the ones finetuned to generate questions using passagequestions data would further help in improving retrieval.