QAMPARI : A Benchmark for Open-domain Questions with Many Answers (2023)

Abstract: Existing benchmarks for open-domain question answering (ODQA) typically focus on questions whose answers are all in a single paragraph. By contrast, many natural questions, such as “What players were drafted by the Brooklyn Nets?” have a long list of answers extracted from multiple paragraphs. Answering such questions requires retrieving and reading many passages from a large corpus. We introduce QAMPARI, an ODQA benchmark, where answers are lists of entities, spread across many paragraphs. We created QAMPARI by (a) generating questions with multiple answers from Wikipedia’s knowledge graph and tables, (b) automatically pairing answers with supporting evidence in Wikipedia paragraphs, and (c) manually paraphrasing questions and validating each answer. Across a wide range of ODQA models, we find that QAMPARI is challenging in terms of both passage retrieval and answer generation, with models reaching an F1 score of 32.8 at best. We view QAMPARI as a valuable resource for ODQA research, which will aid to develop models that handle a broad range of question types, including single and multi-answer questions.

## introduction open-domain question answering (odqa) is a core language understanding task concerned with answering factoid questions over large document collections @xcite . due to its wide applicability, odqa has received substantial attention in recent years @xcite @xcite . typically, systems tackling odqa tasks follow the "retrieve-and-read" paradigm, where a retriever first retrieves a set of candidate passages, followed by a reader which receives the retrieved passages and produces the final answer. the retrieve-and-read paradigm has been effective for benchmarks such as natural questions q: who are the directors of movies produced by eric newman? producers eric newman and marc abraham developed the film @xcite . ## dataset construction each example in qampari is a triple (q, a, p), where q is a question, a is a set of answers and p is a set of passages from our target corpus. an answer @xmath0. 1 ). we define passages as consecutive sentences from our corpus (wikipedia), that span on average 100 words. as our focus is multi-answer questions, examples in qampari have |a| ≥ 5. ## dataset analysis qampari contains 61,911 training examples, 1,000 development examples and 1,000 test examples. tab. 1 provides example questions of each question type and data sources. we describe key statistics in tab. 2. test examples in qampari have 13.23 answers on average and a median of 7 answers. for comparison, the number of answers per question is substantially higher than in am-bigqa @xcite , where the median is 2. on average, simple questions have more answers than complex ones while being shorter in length. we note that since test and development questions were manually re-phrased by annotators they are generally shorter than the training questions. figure 3a presents a binned distribution of the number of answers per question in the development and test sets. roughly half of the questions have 8 or more answers, with 20% having more than 15 answers and 3.5% with over 50 answers. extended set as discussed in §2.5, we manually validate each answer in qampari is supported by sentences from wikipedia. however, wikipedia might contain additional correct answers. to alleviate this issue, we manually annotate additional gold answers for a subset of test questions, and name it the extendedset. we randomly sampled 200 questions from the test set and had an author manually annotate as many additional answers as possible in 12 minutes, per question. this process is not guaranteed to be complete, as it would require manually reviewing all of wikipedia. moreover, questions with hundreds of gold answers ("who worked for burton f. c?") would incur hours of annotation, which is too expensive. this is similar to work in open information extraction @xcite , where creating the full gold set of triples is not feasible. fig. 3 plots the number of added answers per question on the extended set. in 30% of the questions, we did not add any answer, and the median/average/maximum number of added answers are 2/3.13/16 respectively. evaluation on the test set and the extended set in §4.3 shows that model precision on the extended set is somewhat higher, but does not alter model ranking, illustrating the reliability our test set. ## conclusions we release qampari, a dataset targeting odqa models ability to answer multi-answer questions, and show that it is challenging for current stateof-the-art models. qampari will aid develop models that answer a wide range of question types, including single-and multi-answer questions.