G oo AQ : Open Question Answering with Diverse Answer Types (2021)

Abstract: While day-to-day questions come with a variety of answer types, the current question-answering (QA) literature has failed to adequately address the answer diversity of questions. To this end, we present GooAQ, a large-scale dataset with a variety of answer types. This dataset contains over 5 million questions and 3 million answers collected from Google. GooAQ questions are collected semi-automatically from the Google search engine using its autocomplete feature. This results in naturalistic questions of practical interest that are nonetheless short and expressed using simple language. GooAQ answers are mined from Google’s responses to our collected questions, specifically from the answer boxes in the search results. This yields a rich space of answer types, containing both textual answers (short and long) as well as more structured ones such as collections. We benchmark T5 models on GooAQ and observe that: (a) in line with recent work, LM’s strong performance on GooAQ’s short-answer questions heavily benefit from annotated data; however, (b) their quality in generating coherent and accurate responses for questions requiring long responses (such as ‘how’ and ‘why’ questions) is less reliant on observing annotated data and mainly supported by their pre-training. We release GooAQ to facilitate further research on improving QA with diverse response types.

## introduction research in "open" question answering (also referred to as open-response, open-domain, or direct answer qa) has resulted in numerous datasets and powerful models for answering questions without a specified context. this task requires the use of background knowledge either stored in the qa model or retrieved from large corpora or knowledge bases @xcite . existing effort, however, involves isolated studies on niche answer types, mainly short responses and, in a few cases, long responses @xcite @xcite . in contrast, many of the everyday questions that humans deal with and pose to search engines have a more diverse set of response types, as illustrated in fig. 1 . their answer can be a multi-sentence description (a snippet) (e.g., 'what is' or 'can you' questions), a collection of items such as ingredients ('what are kinds of', 'things to') or of steps towards a goal such as unlocking a phone ('how to'), etc. even when the answer is short, it can have rich types, e.g., unit conversion, time zone conversion, or a variety of knowledge look-up ('how much', 'when is', etc.). 2 such answer type diversity is not represented in any existing dataset. motivated by this, we introduce gooaq (pronounced guac like guacamole), the first open qa benchmark containing questions with all of the above answer types within a unified dataset, collected using the same, coherent process. gooaq contains 3 million questions with short, snippet, or collection answers, such as the ones shown in fig. 1 . besides supporting research on various types of answers, gooaq enables a quantitative study of the inherent differences in systems across different answer types. gooaq questions are automatically mined from google's search-autocomplete feature and thus, we hypothesize, represent popular queries of realworld interest. such questions also trigger 'answer boxes' in the search results, containing responses deemed best by google, which we extract and refer to as google answers. our human evaluation ( §3.2) found the collected questions and answers to be of question: what is the difference between an assignment and a delegation? answer(snippet): the difference is that an assignment can't increase another party's obligations. delegation, on the other hand, is a method of using a contract to transfer one party's obligations to another party. assigning rights is usually easier than delegating, and fewer restrictions are in place. figure 1 : examples from gooaq showing different types of the questions considered in this study. each input is a natural language question, mapped to textual answer(s). the questions/answers come with answer type which are inferred from meta information of the search results. to understand the contribution of pre-training, (q 1 ), we train the powerful t5 language model @xcite on gooaq with a small amount of labeled data. while lms struggle, as expected, in this setting on short response questions, they perform surprisingly well in generating snippet and collection responses. 3 we hypothesize this is because response fluency and coherence have a much higher weight in such questions, and these factors remarkably benefit from the lm pre-training objective. regarding the value of labelled data, (q 2 ), we observe the opposite trend: short response questions benefit consistently from increasing amounts of supervised (labeled) data, whereas both snippet and collection response questions show minimal gains (e.g., only 5-10% improvement when going from 2k training examples to 200k or even 2 million). lastly, on the benefit of model size, (q 3 ), we find larger models to be more effective in all cases as expected, but the gains are much more pronounced for snippet and collection 3 over 30-40% of our best model's snippet and collection answers were preferred by crowdworkers over google's answers; achieving 50% here would mean parity with google. additionally, we expect gooaq to facilitate further research on models for answering snippet and collection response questions. while the largest models we consider score surprisingly high on these questions, they are still far from reaching google's quality under either automated or human evaluations. importantly, due to little benefit observed from more labeled data on such questions, further progress requires rethinking the approach and devising new solutions. lastly, we find gooaq to be a valuable resource for training models. on the long-answer dataset eli5 @xcite , t5 trained only on our snippet questions performs on par with state-of-theart models trained on eli5 data. our closing remarks describe why we aren't simply replicating an existing qa system at google, place our findings in context, and discuss future uses of gooaq, such as creating a neural knowledge-base or a question generation system. contributions. our contributions are threefold: ## related work a closely related work is the natural-questions (nq) dataset @xcite which contains questions written by google users, and answers that were manually extracted from wikipedia articles. while our questions (extracted via autocomplete) were also likely frequently asked by google users, our dataset represents a different and wider distribution of questions ( §3.3), likely because it encompasses different classes of answers, particularly snippet and collection responses. specifically, while nq is dominated by 'who', 'when', and 'how many' questions (cf. fig. 3(d)), gooaq has notably few 'who' questions and a substantial portion of questions starting with 'how to', 'what is', 'what does', 'can you'. one notable qa dataset with long-form responses is eli5 @xcite , containing questions/answers mined from reddit forums. in contrast, gooaq is collected differently and is several orders of magnitude larger than eli5. empirically, we show that models trained on gooaq transfer surprisingly well to eli5 ( §5.3), indicating gooaq's broad coverage. it is worth highlighting that there is precedent for using search engines to create resources for the analysis of ai systems. search engines harness colossal amounts of click information to help them effectively map input queries to a massive collection of information available in their index @xcite @xcite . although academic researchers do not have direct access to information collected from the users of search engines, search results can act as a proxy for them and all the complex engineering behind them. in particular, the gooaq dataset used in this study probably is not representative of a single qa system in google; on the contrary, we hypothesize, this data is produced by a complex combination of many systems, various forms of user feedback, as well as expert annotation/verification of highly popular responses. ## gooaq dataset we describe how gooaq was collected, followed by dataset statistics and quality assessment. ## empirical results and analyses in this section, we evaluate the behavior of models for various answer types ( §5.1). we further show how gooaq can support research in answering questions with long answers ( §5.2; §5.3). ## closing remarks we studied open qa under diverse response types. to this end, we collected gooaq, a very large set of qa pairs mined from google, with a variety of short and long answer types, all of which are collected using a unified, coherent process, enabling a cross-type comparison. the auto-complete system used for our question collection likely reflects a natural distribution of questions asked by users. we benchmarked two variants of a state-ofthe-art self-contained text generation model (t5, without retrieval) on three different sub-tasks of gooaq: short, snippet, and collection response questions. our analysis, using both automatic and human evaluations, brings out the distinct behavior of lms on long and short response questions. for example, while short response models benefit heavily from more labeled data, the surprisingly strong performance of long response models is driven mostly by their pre-training. we also demonstrate that gooaq is a valuable resource for training models by showing high performance on an extrinsic task, eli5, while using only gooaq data for training. scope of our conclusions. one must be careful in taking our specific conclusions out of the context of this study (i.e., the dataset at hand, the models, the evaluation metrics used, etc.). while we expect our findings to be fairly general, it may be possible to come up with a different long-form qa dataset where the trends across answer types differ. knowledge leakage across train and evaluation sets has been shown to significantly inflate performance numbers on recent open qa datasets @xcite . similar concerns have motivated our careful training/evaluation splits of the data ( §4) and experiments with varying training set sizes. nevertheless, we found it challenging to define (and assess) the amount of such leakage, and welcome such studies on gooaq. are we mimicking google's qa? a reader might question the value of this work by noting that the website from which gooaq was mined had likely also used a qa system to begin with. in other words, are we basically reverse-engineer google's internal qa system @xcite )? while we (the authors) are not aware of how google answer box system works, we suspect that it is much more complex than a single qa system built using a single lm like t5 @xcite . the system, besides incorporating one or more qa models, likely makes heavy use of implicit user feedback (e.g., information contained in billions of clicks, the structure of web links, etc.), in addition to explicit feedback from users and possibly some expert curation of answers to common questions. moreover, google's system can decide which questions to display answers for, probably limiting itself to the answers that it is most confident in. thus, the data in google's answer boxes likely captures a variety of signals that contribute towards its high-quality. we believe aiming for a 'standard' nlp qa system that's on par with google qa is therefore a challenging and worthwhile goal. future uses of gooaq. one challenge in the progress on long-form qa is response evaluation. to facilitate future work on gooaq and replicability of our human evaluations, we have released the templates used for crowdsourcing human judgements. efforts on text generation tasks such as ours will benefit from-and should in turn benefit advances in-proposals for streamlining human evaluation of models @xcite . we hope our analysis and data will benefit the understanding of and further development of qa systems for dealing with diverse response types. while we used gooaq for the purposes of qa, we expect this data to have a variety of use-cases, such as building a knowledge-base accessible via question queries @xcite , creating a better question generation system, etc. we leave such investigation to future work.