R o MQA : A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering (2023)

Abstract: We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA). RoMQA contains clusters of questions that are derived from related constraints mined from the Wikidata knowledge graph. RoMQA evaluates robustness of QA models to varying constraints by measuring worst-case performance within each question cluster. Compared to prior QA datasets, RoMQA has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. In addition, human annotators rate RoMQA questions as more natural or likely to be asked by people. We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging: zeroshot and few-shot models perform similarly to naive baselines, while supervised retrieval methods perform well below gold evidence upper bounds. Moreover, existing models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.

## introduction a high quality compositional question answering (qa) model should exhibit robustness to subtle variations in the underlying meaning of input questions. for exmaple, consider the question "which pianists born in paris play western classical music?" to show robust understanding, a qa model should not only be able to correctly answer this direct question, but also a wide range of related queries that differ in only a few constraints (e.g. who was a pianist born in paris?, who was a western classical pianist, not born in paris?). prior compositional qa datasets do not evaluate the robustness of qa models to variations in question constraints. we introduce romqa, a benchmark for robust, multi-evidence, multi-answer qa, that explicitly evaluates for robustness to small question perturbations. romqa, shown in figure 1 , differs from previous work in a number of ways. evaluates robustness to constraint variations. romqa contains clusters of related questions that are used to measure robustness to varying implicit question constraints. for each cluster, we compute a robustness score that is the minimum score over the questions it contains. in order to perform well on romqa robustness evaluation, a model must be able to understand many different combinations of the implicit constraints that define the cluster, such as what it means to be a pianist, to be born in paris, and to play western classical music. to our knowledge, romqa is the first qa benchmark that evaluates this type of robustness. more complex questions. natural questions often have many answers and cannot be answered from a single text. when compared to existing datasets, romqa questions have more answers (mean 108.6, median 11, as shown in figure 2 ), cover a wider range of diverse topics (as depicted in figure 3 ), and necessitate a higher amount of evidence text (mean 41.6, median 24). romqa also contains entity-linked, relation-extracted text that provide provenance for constraints, showing that the questions are answerable with multi-evidence reasoning from the text corpus. more natural human written questions. contrary to previous multi-answer compositional qa datasets, romqa provides a significantly larger pool of 28k human-written questions. this represents a ten-fold increase when compared to the previous leading dataset qampari @xcite ), which features a mere 2k human-written questions. human evaluations show that questions in romqa are more natural, as gauged by we evaluate state-of-the-art large language models (lms) on romqa in zero-shot prompting, fewshot in-context learning, and supervised learning settings. in the closed setting where the model selects among 100 candidate entities, zero-shot and few-shot lms perform on par (e.g. finally, no test model is robust to variations in question constraints. the best performing retrieval method obtains a worse-case related question test score of 37.9 f1 in the closed setting -a 25.9 f1 absolute drop compared to evaluating questions independently. training on clusters of related questions, such as romqa clusters, improves model robustness over training on unrelated questions. however, the robustness gap remains large -closing this gap will likely require significant advances in natural language understanding. we open-source romqa at anonymous.url. ## romqa we describe romqa construction and how it differs from prior compositional qa datasets. ## experiments how do existing qa systems perform on romqa? are they robust to variations in question constraints? we answer these questions by experimenting with state-of-the-art models in zero-shot, few-shot, and supervised learning settings. ## related work question answering datasets existing qa datasets focus on answering from a single passage @xcite @xcite training closed open questions f1 f1 r ∆f1 p10 p10 r ∆p10 unrelated 56.2 28.2 -28.0 27.0 1.6 -25.4 related 55.7 28.6 -27.1 26.3 11.7 -14.6 robustness evaluation nlp systems have previous been show to lack robustness. they are susceptible to character based attacks that comprise of both nonsensical inputs @xcite , random sentence/word triggers @xcite , and semantically equivalent inputs adversarially selected to disrupt system behaviour @xcite @xcite . in contrast, the questions in romqa are not adversarial -they are written with reasonable information-seeking intent. zero-shot/few-shot learning recent work has also shown that large pretrained lms can perform zero-shot and few-shot inference @xcite @xcite . for the former, the lm performs inference given a prompt or an instruction. for the latter, the lm is also given training examples as demonstration. we use both settings as baselines for romqa, and find that there is significant room for improvement in large-scale pretraining to capture compositional reasoning over multiple pieces of evidence text. ## conclusion we presented romqa, the first benchmark for robust, multi-evidence, multi-answer qa. romqa evaluates robustness of models to varying question constraints by testing for worst-case performance among clusters of related questions. romqa is a challenging dataset for state-of-theart large lms and provides a quantifiable test for developing more robust qa methods. 6 limitations robustness definition a robust system needs to be robust towards all forms of input variations. the type of robustness to varying implicit constraints measured in romqa is but one form. other forms of robustness include adversarial character attacks, sentence injections, and word triggers. romqa is limited to english -future work should investigate robustness in other and multiple languages. the questions in romqa are based on wikidata triples and contains limitations inherent to wikidata and large-scale knowledge-graphs in general (e.g. staleness of facts, over-represented propositions/entities, incompleteness). the models evaluated in this work (e.g. opt-instruct-175b, text-davicinci-002) are but a sample of state-of-the-art language models at the time of writing. as new models emerge, it is possible that they exhibit robustness beyond that of the evaluated models.