PAQ : 65 Million Probably-Asked Questions and What You Can Do With Them (2021)

Abstract: Open-domain Question Answering models that directly leverage question-answer (QA) pairs, such as closed-book QA (CBQA) models and QA-pair retrievers, show promise in terms of speed and memory compared with conventional models which retrieve and read from text corpora. QA-pair retrievers also offer interpretable answers, a high degree of control, and are trivial to update at test time with new knowledge. However, these models fall short of the accuracy of retrieve-and-read systems, as substantially less knowledge is covered by the available QA-pairs relative to text corpora like Wikipedia. To facilitate improved QA-pair models, we introduce Probably Asked Questions (PAQ), a very large resource of 65M automatically generated QA-pairs. We introduce a new QA-pair retriever, RePAQ, to complement PAQ. We find that PAQ preempts and caches test questions, enabling RePAQ to match the accuracy of recent retrieve-and-read models, whilst being significantly faster. Using PAQ, we train CBQA models which outperform comparable baselines by 5%, but trail RePAQ by over 15%, indicating the effectiveness of explicit retrieval. RePAQ can be configured for size (under 500MB) or speed (over 1K questions per second) while retaining high accuracy. Lastly, we demonstrate RePAQ’s strength at selective QA, abstaining from answering when it is likely to be incorrect. This enables RePAQ to “back-off” to a more expensive state-of-the-art model, leading to a combined system which is both more accurate and 2x faster than the state-of-the-art model alone.

## introduction open-domain qa (odqa) systems usually have access to a background corpus that can be used to answer questions. models that explicitly exploit this corpus are commonly referred to as openbook models @xcite . they typically index the whole corpus, and then retrieve-andread documents to answer questions @xcite . a second class of models, closed-book question answering (cbqa) models, has recently been proposed. they learn to directly map questions to answers from training question-answer (qa) pairs without access to a background corpus @xcite . these models usually take the form of pretrained seq2seq models such as t5 @xcite or bart @xcite , fine-tuned on qa-pairs. it has recently been shown that current closed-book models mostly memorize training qa-pairs, and can struggle to answer questions that do not overlap with training data @xcite . models that explicitly retrieve (training) qapairs, rather than memorizing them in parameters, have been shown to perform competitively with cbqa models @xcite . these models have a number of useful properties, such as fast inference, interpretable outputs (by inspecting retrieved qa-pairs), and the ability to update the model's knowledge at test time by adding or removing qa-pairs. however, cbqa and qa-pair retriever models are currently not competitive with retrieve-andread systems in terms of accuracy, largely because the training qa-pairs they operate on cover substantially less knowledge than background corpora like wikipedia. in this paper, we explore whether greatly expanding the coverage of qa-pairs enables cbqa and qa-pair retriever models which are competitive with retrieve-and-read models. we present probably asked questions (paq), a semi-structured knowledge base (kb) of 65m natural language qa-pairs, which models can memorise and/or learn to retrieve from. paq differs from traditional kbs in that questions and answers are stored in natural language, and that questions are generated such that they are likely to appear in odqa datasets. paq is automatically constructed using a question generation model and wikipedia. to ensure generated questions are not only answerable given the passage they are generated from, we use a global filtering post-processing step using a state-of-the-art odqa system. this greatly reduces the amount of wrong/ambiguous questions compared to other approaches @xcite , and is critical for high-accuracy, downstream qa. to complement paq we develop repaq, an odqa model based on question retrieval/ matching models, using dense maximum inner product search-based retrieval, and optionally, re-ranking. we show that paq and repaq provide accurate odqa predictions, at the level of relatively recent large-scale retrieve-and-read systems such as rag @xcite on naturalquestions @xcite and triviaqa @xcite . paq instances are annotated with scores that reflect how likely we expect questions to appear, which can be used to control the memory footprint of repaq by pruning the kb accordingly. as a result, repaq becomes flexible, allowing us to configure qa systems with near state-of-the-art results, very small memory size, or inference speeds of over 1,000 questions per second. we also show that paq is a useful source of training data for cbqa models. bart models trained on paq outperform standard data baselines by 5%. however, these models struggle to effectively memorize all the knowledge in paq, lagging behind repaq by 15%. this demonstrates the effectiveness of repaq at leveraging paq. finally, we show that as repaq's question matching score correlates well with qa accuracy, it effectively ''knows when it doesn't know'', allowing for selective question answering @xcite where systems may abstain from answering. although answer abstaining is important in its own right, it also enables an elegant ''back-off'' approach where we can defer to a more accurate but expensive qa system when the answer confidence is low. this allows us to make use of the best of both speed and accuracy. in summary, we make the following contributions: i) introduce paq, 65m qa-pairs automatically generated from wikipedia, and demonstrate the importance of global filtering for high quality; ii) introduce repaq, a qa system designed to utilize paq and demonstrate how it can be optimised for memory, speed, or accuracy; ii) investigate the utility of paq for cbqa models, improving by 5% but note significant headroom to repaq iv) demonstrate repaq's strength on selective qa, enabling us to combine repaq; and with a state-of-the-art qa model, making it both more accurate and 2x faster. foot_0 2 open-domain question answering odqa is the task of answering natural language factoid questions from an open set of domains. a typical question might be ''when was the last year astronauts landed on the moon?'', with a target answer ''1972''. the goal of odqa is to develop an answer function m : @xmath0, where q and a, respectively, are the sets of all possible questions and answers. we assume there is a distribution p (q, a) of qa-pairs, defined over q × a. a good answer function will minimize the expected error over p (q, a) with respect to some loss function, such as answer string match. in practice, we do not have access to p (q, a), and instead rely on an empirical sample of qa-pairs k drawn from p , and measure the empirical loss of answer functions on k. our goal in this work is to implicitly model p (q, a) in order to draw a large sample of qa-pairs, paq, which we can train on and/or retrieve from. a sufficiently-large drawn sample will overlap with k, essentially pre-empting and caching questions that humans may ask at test-time. this allows us to shift computation from test-time to train-time compared to retrieve-and-read methods. ## generating question-answer pairs in this section, we describe the process for generating paq. given a large background corpus c, our qa-pair generation process consists of the following components: as shown in figure 1 , these models are applied sequentially to generate qa-pairs, similarly to contextual qa generation @xcite . first a passage c is selected with a high probability under p s . next, candidate answers a are extracted from c using p a , and questions q are generated for each answer using p q . lastly, p f generates a new answer a for the question. if source answer a matches a , then (q, a) is deemed consistent and added to paq. the pipeline is based on @xcite , updated to take advantage of recent modeling advances. passage selection and our filtering approach are novel contributions to the best of our knowledge, specifically designed for odqa qa-pair generation. each component is described in detail below. ## question answering using paq we consider two uses of paq for building qa models. the first is to use paq as a source of training qa-pairs for cbqa models. the second treats paq as a kb, which models learn to directly retrieve from. these are related, as cbqa models have been shown to memorize the train data in their parameters, latently retrieving from them at test time @xcite . ## results we first examine the paq resource in general, before exploring how both cbqa models and repaq perform using paq, comparing to recently published systems. we measure performance using natural questions (nq, @xcite and triviaqa @xcite , evaluating using standard exact match (em) score. ## discussion and conclusion we have introduced a dataset of 65m qa-pairs, and explored its uses for improving odqa models. we demonstrated the effectiveness of repaq, a system that retrieves from paq, in terms of accuracy, speed, space efficiency and selective qa. we found that repaq's errors are driven by a lack of coverage, thus generating more qa-pairs should improve accuracy further. however, phenomena such as compositionality may impose practical limits on this approach. multi-hop repaq extensions suggest themselves as ways forward here, as well as back-off systems (see section 5.2.4). generating paq is also computationally intensive due to its large scale and global filtering, but it should be a useful, re-usable resource for researchers. nevertheless, future work should be carried out to improve the efficiency of generation to expand paq's coverage. we also demonstrated paq's utility for improved cbqa, but note a large accuracy gap between our cbqa models and repaq. exploring the trade-offs between storing and retrieving knowledge parametrically or non-parametrically is of great current interest @xcite , and paq should be a useful testbed for probing this relationship further. we also note that paq could be used as general dataaugmentation when training any open-domain qa model or retriever. we consider such work out-ofscope here, but leveraging paq to improve other models should be explored in future work.