Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation (2021)

Abstract: A major obstacle to the wide-spread adoption of neural retrieval models is that they require large supervised training sets to surpass traditional term-based techniques, which are constructed from raw corpora. In this paper, we propose an approach to zero-shot learning for passage retrieval that uses synthetic question generation to close this gap. The question generation system is trained on general domain data, but is applied to documents in the targeted domain. This allows us to create arbitrarily large, yet noisy, question-passage relevance pairs that are domain specific. Furthermore, when this is coupled with a simple hybrid term-neural model, first-stage retrieval performance can be improved further. Empirically, we show that this is an effective strategy for building neural passage retrieval models in the absence of large training corpora. Depending on the domain, this technique can even approach the accuracy of supervised models.

## introduction recent advances in neural retrieval have led to advancements on several document, passage and knowledge-base benchmarks @xcite @xcite @xcite @xcite . most neural passage retrieval systems are, in fact, two stages @xcite , illustrated in figure 1 . the first is a true retrieval model (aka first-stage retrieval 1 ) that takes a question and retrieves a set of candidate passages from a large collection of documents. this stage itself is rarely a neural model and most commonly is an term-based retrieval model such as bm25 @xcite , though there is recent work on neural models (zamani et al., 2018; dai and callan, 2019; chang et al., 1 also called open domain retrieval. ## synthetic question generation in this work, we are specifically investigating the zero-shot scenario where there exists neither user issued questions nor domain specific data except the passage collection itself. we propose to address the ubuntu forums passage: every time i get a notification about and begin updating when they become available, the process is interrupted by an error message: error in foomatic-filters. then i get "error in linux generic package" and a bunch of numbers. this is replaced before i can write it all down with "error in linux package" everything seems to go ok except i don't know if the updates are really being installed. i tried un-installing and re-installing foomatic-filters . . . generated question: how do i get rid of error in foomatic-filters? biomedical literature passage: electroencephalographic tracings of 50 patients who presented the classical features of friedreich's ataxia were reviewed . . . friedreich's ataxia is mainly a spinal disorder. involvement of supraspinal and in particular brain stem or diencephalic structures may be more extensive in those patients who show electrographic abnormalities. this would require confirmation with comparative data based on pathological observations. impaired function of brain stem inhibitory mechanism may be responsible for the slightly raised incidence of seizures in patients with friedreich's ataxia and other cerebellar degenerations. generated question: what is the significance of friedreich's ataxia? to ensure data quality, we further filter the data by only keeping question-answer pairs that were positively rated by at least one user on these sites. in total, the final dataset contains 2 millions pairs, and the average length of questions and answers are 12 tokens and 155 tokens respectively. this dataset is general domain in that it contains questionanswer pairs from a wide variety of topics. our question generator is an encoder-decoder with transformer @xcite layers, which is a common for generation tasks such as translation and summarization @xcite . the encoder is trained to build a representation for a text and the decoder generates a question for which that text is a plausible answer. appendix b has model specifics. our approach is robust to domain shift as the generator is trained to create questions based on a given text. as a result, generated questions stay close to the source passage material. real examples are shown in ## neural first-stage retrieval in this section we describe our architecture for training a first-stage neural passage retriever. our retrieval model belongs to the family of relevancebased dense retrieval 6 that encodes pairs of items in dense subspaces @xcite . let @xmath0, . . . q n ) and @xmath1, . . . , p m ) be a question and passage of n and m tokens respectively. our model consists of two encoders, {f q (), f p ()} and a similarity function, sim(). an encoder is a function f that takes an item x as input and outputs a real valued vector as the encoding, the similarity function, sim(), takes two encodings, q, @xmath2, @xmath3, p). for passage retrieval, the two encoders are responsible for computing dense vector representation of questions and passages. ## experimental setup we outline data and experimental details. the appendix has further information to aid replicability. ## results and discussion our main results are shown in accuracy of pure neural models are shown in the upper group of performance of term-based models and hybrid models are shown in for naturalquestions since there is a single relevant passage annotation, we report precision@1 and mean reciprocal rank (mrr) foot_6 . results are show in ## conclusion we study methods for neural zero-shot passage retrieval and find that domain targeted synthetic question generation coupled with hybrid termneural first-stage retrieval models consistently outperforms alternatives. furthermore, for at least one domain, approaches supervised quality. while out of the scope of this study, future work includes further testing the efficacy of these first-stage models in a full end-to-end system (evaluated briefly in section 6.1), as well as for pre-training supervised models @xcite .