M o QA : Benchmarking Multi-Type Open-Domain Question Answering (2023)

Abstract: Previous research on open-domain question answering (QA) mainly focuses on questions with short answers. However, information-seeking QA often requires various formats of answers depending on the nature of the questions, e.g., why/how questions typically require a long answer. In this paper, we present MoQA, a benchmark for open-domain QA that requires building one system that can provide short, medium, long, and yes/no answers to different questions accordingly. MoQA builds upon Natural Questions with multiple types of questions and additional crowdsourcing efforts to ensure high query quality. We adapt state-of-the-art models, and reveal unique findings in multi-type open-domain QA: (1) For retriever-reader models, training one retriever on all types achieves the overall best performance, but it is challenging to train one reader model to output answers of different formats, or to train a question classifier to distinguish between types; (2) An end-to-end closed-book QA model trained on multiple types struggles with the task across the board; (3) State-of-the-art large language models such as the largest GPT-3 models (Brown et al., 2020; Ouyang et al., 2022) also lag behind open-book QA models. Our benchmark and analysis call for more effort into building versatile open-domain QA models in the future.

## introduction open-domain question answering (qa) leverages a large knowledge source such as wikipedia to answer open-domain questions @xcite . such a task mimics humans' information-seeking process-finding relevant documents and composing answers based on them-and has potential to become a fundamental 1 @xmath0. it is pronounced as mocha. there has been recent efforts to study other forms of answers in open-domain qa @xcite . in particular, gooaq @xcite is a benchmark consisting of different types of questions mined from google autocomplete system and google's answers boxes without human annotation. nonetheless, all of these works consider each type of questions separately. the closest work to ours is kilt @xcite , which also considers different types of open-domain questions and builds one system to answer them. however, kilt is a collection of different tasks such as qa, fact checking, entity linking, etc., each with its own unique collection pipeline, which can introduce superficial cues that help the model determine the format of the answer. building one system for different types of questions has been studied in the reading comprehension setting @xcite , where the supporting evidence is given as input. on the other hand, open-domain qa requires retrieving the evidence from a large text corpus or recalling specific facts from a model's parametric knowledge, both of which poses significantly different challenges. different from previous work, we bring different types of questions drawn from the same distribution together, and aim to build a single open-domain system (either open-book or closed-book qa) that can handle various types of questions with finegrained annotations, based on a single knowledge source i.e., english wikipedia. we further discuss the differences with past works in section 6. in this paper, we propose moqa, a benchmark that requires an open-domain qa system to answer multiple types of questions (figure 1 ). we build our benchmark by extending natural questions (nq) @xcite into the open-domain setup, aligning each answer to a fixedlength wikipedia passage, while keeping questions of four types: short, medium, long, and yes/no. we also use additional human annotations on the test set to filter out ill-defined long-answer questions, ensuring the high quality of moqa evaluation. we conduct a comprehensive set of experiments on moqa with state-of-the-art qa models @xcite @xcite @xcite , and reveal multiple interesting findings: our findings suggest building such a versatile opendomain system still has a series of unsolved challenges and call for more effort in the direction. ## the moqa benchmark in open-domain qa, models are given a set of k passages @xmath1, . . . , d k } for a question q. in moqa, the correct answer a to the question q could be a span ranging from a couple words to a paragraph, , yes, or no. unlike existing opendomain benchmarks that assume an answer span to be short (@xmath2, moqa does not assume a length limit and requires models to output different formats of answers considering the nature of each question. ## adapting qa models for moqa we establish multiple baselines for moqa, by adapting state-of-the-art open-domain qa systems, including both supervised open-book qa models @xcite , supervised closed-book qa models @xcite , and llms @xcite . ## results retriever: different types of questions have similar difficulty. we first demonstrate the retrieval results in though the oracle model is much better than the single reader, we see that training question classifier and 4 readers underperforms training jointly. this comes down to the difficulty to train the question classifier -figure 2 shows the confusion matrix for question type prediction. it is noticeable that medium and long-answer questions are the most difficult to predict. for example, it is not trivial to understand that the question "who is covered under payment of gratuity act 1972" requires a long answer. improving the classifier will be a straightforward to advance moqa performance, as the current systems still have a huge gap to the corresponding type oracles. please refer to appendix j for qualitative examples. closed-book qa models significantly lag behind. though competitive on short-answer questions as shown in previous work, closed-book qa models perform poorly on other types, especially on longanswer questions. both the trained t5-large model as well as the llms lag behind open-book qa models. our result suggests that it is still challenging to solve moqa relying purely on parameters. one possible explanation is the fact that more complex questions and extensive explanations require more memorization than short-form answers from the closed-book qa models, so it is harder to rely on knowledge stored in the model's parameters for this task. furthermore, these explanations are less likely to appear in the pre-training corpus than popular entities such as people's names, famous places, and significant dates. even for models such as gpt-3 that has been trained on an extremely large corpus, these answers still appear to be harder to recall than shorter answers. ## related work long-form qa. eli5 @xcite mines questions and answers from the subreddit explain like i'm five foot_3 , the answers are abstractive and are not grounded in a knowledge source. similar to our work, asqa @xcite also considers long-form answers in open-domain qa and its questions originates from nq @xcite . however, it only focuses on the questions that have long answers due to ambiguity. gooaq @xcite propose a dataset consisted of different question types, but these questions are mined from google's answer boxes and therefore noisy. furthermore, eli5, asqa, and gooaq all study each question type independent of each other. extension of natural questions. recent works leverage the original nq beyond those with only short answers. for example, aquamuse @xcite use the long answer only questions from the original nq for the task of query-based multi-document summarization. they consider nq questions that only have long answers as these question "result in open-ended and complex topic answers". boolq @xcite focuses on boolean questions that can be answered by true or false, and expands on the yes/no questions from the original nq by collecting additional questions using the same pipeline. the authors found the task challenging and require robust reasoning ability. the unanswerable questions in nq were explored by asai and choi (2020), but such questions are difficult to study in the open-domain setting and require extensive manually annotations. in contrast to these works, we study draw a diverse set of questions with different answers from the same query distribution and study these types together. multi-type question answering unifiedqa @xcite ) builds a single qa system to answer different types of questions. however, their setting is limited to the closed-domain setup, and only analyzes fine-tuned generative models. llms achieves impressive performance on many knowledge-intensive tasks due to the knowledge packed in its parameters @xcite . they do not require any additional fine-tuning and only rely on in-context examples to adapt to different tasks. a recent line of work proposes to further improve llms capabilities by prompting them to generate a series of intermediate reasoning steps @xcite . @xcite applies a similar idea to open-domain qa, where the llm first generates a supporting passage instead of relying on external retriever models, and then output the answer. ## conclusion we propose moqa, an open-domain qa benchmark with multi-type questions, and evaluate a range of baseline models. our findings suggest that the main difficulty lies in the reader's task, and building a versatile reader or an accurate classifier is challenging. competitive closed-book qa models also degrade on the new benchmark. improving the classifier or mining more data to build a generalizable reader might be promising directions towards truly all-round systems.