MKQA : A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering (2021)

Abstract: Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open- domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). Answers are based on heavily curated, language- independent data representation, making results comparable across languages and independent of language-specific passages. With 26 languages, this dataset supplies the widest range of languages to-date for evaluating question answering. We benchmark a variety of state- of-the-art methods and baselines for generative and extractive question answering, trained on Natural Questions, in zero shot and translation settings. Results indicate this dataset is challenging even in English, but especially in low-resource languages.1

## introduction training and evaluation data for question answering (qa) is severely lacking outside of highresource languages like english. as unsupervised, transfer learning and zero/few-shot methods narrow the multilingual performance gap with english @xcite @xcite , their real progress is hard to measure without challenging, realistic, and linguistically diverse evaluation sets. existing multilingual qa datasets are realistic and challenging, but they lack linguistic diversity, comparable evaluation between languages, and are often limited to passages provided with the dataset (see we introduce multilingual knowledge questions and answers (mkqa) for evaluation of open-domain question answering. mkqa selects 10k realistic english queries from the natural questions dataset (nq, @xcite and human translates them into 25 additional languages and dialects. accompanying these query translations we replace nq's passage embedded answer spans with high-quality, language-and retrieval-independent answer annotations, linked directly against wikidata entities and a limited set of well-defined value types (numbers, dates, strings, etc.). 2 see one full example in ## dataset collection we aim for certain properties of our evaluation set: (i) realistic questions, (ii) reliable annotations (e.g., via inter-annotator agreement), and (iii) a flexible task setup that makes as few assumptions as possible about the underlying modeling techniques, enabling fair comparison between any approach. ## dataset quality and analysis given our dataset collection and methodology, we evaluate the effect of our choices, and the properties of the final set, including the selected languages, annotation quality, geographical invariance, and answer type distribution as compared to nq. ## discussion difficulty of mkqa our baselines represent a strong and diverse set of methods, that score competitively with state-of-the-art on similar open domain question answering datasets. nonetheless, on english alone, the best system recieves an f1 score of only 52.3%, less than the same methods achieve on the open datasets natural questions and triviaqa, or other standard benchmarks for this task. these comparative results demonstrate mkqa is highly challenging and leaves ample room for improvement in both english and the long tail of natural languages. in this section we explain why, with a detailed comparison to its closest set, natural questions. why is mkqa so challenging for state-of-theart approaches even for english open-domain qa? to shed light on this, we compare the difficulty of english-only annotations between natural questions (nq) and mkqa. in figure 4 we use the same bert-large english model (trained on nq, using gold nq passages) and evaluate it on both sets of annotations. the ''f1 by answer type'' diagram shows unanswerable examples in mkqa (red line) are easier than the unanswerable examples in nq (red dashed line), as the model maintains higher performance at all no answer confidence thresholds. the opposite relationship is observed for answerable examples. we hypothesize that this is due to the retrieval-independence property and high coverage of our re-annotation process (described in section 3). due to the annotation procedures nq uses, there are several cases that can lead to a potential answer missing from the dataset: (a) the initial retrieval may have not produced a candidate, (b) the answer may have not been in wikipedia, or (c) nq graders may have missed a valid answer. mkqa annotations are not susceptible to ( a ) and (b) and likely less impacted by (c). consequently, the most challenging questions migrated from unanswerable in nq to answerable in mkqa, shifting the unanswerable distribution from 63% to 32% (as shown in figure 2 ). consider the following examples. given the answer to these queries are not easily found in the corpus, by retrieval, or by human annotators, they are likely more challenging on average. as such, their label shift from no answer in nq to answerable in mkqa likely explains why there is higher mean difficulty of answerable questions in mkqa, as observed in figure 4 . to understand the prevalence of each error type, we compute how often any mkqa answer appears in the retrieved document for which the nq label says no answer exists. we find a valid answer appears in 70.4% of these documents, suggesting category (c), annotator error, is the largest source of such unanswerable queries in nq (and the largest source of improvement in label quality for mkqa). the middle and right diagrams in figure 4 normalize the answer types by their proportion within the dataset, so we can compare their relative contributions to the aggregate f1 (the sum of answerable and unanswerable). nq labels enable a much higher aggregate f1 score (69.38% at the best threshold) than mkqa (52.08% at the best threshold) primarily due to the higher proportion of unanswerable examples-which are easier on average than answerable examples. by comparing the ratio of unanswerable to answerable examples attempted at the best thresholds in each of the middle and right diagrams (the blue regions vs. the red regions) we see that the mkqa task is more oriented to answering questions rather than abstaining. due to the parallel question property of mkqa, the dataset is similarly challenging in all 26 languages. there is also a noticeable gap between the performance on english and on lowerresourced languages (figure 3 ). for korean and arabic the best f1 score is only 6% higher than the lower bound score of 32.42% obtained from predicting exclusively ''unanswerable.'' this demonstrates that existing transfer learning methods have significant deficits to overcome for low-resource multilingual qa to match english performance. mkqa offers a challenging benchmark to measure this cross-language progress specifically. future work the parallel questions property of mkqa offers alternative task setups in addition to typical open domain question answering. @xcite suggests a generalized crosslingual transfer task (g-xlt) where the question and answer languages are intentionally different. alternatively, future work might assume we are given the english question-answer pairs, and attempt to propagate these answers into other languages by localizing the questions and answers. we anticipate that this dataset will enable industry practitioners and researchers to rapidly test and compare novel cutting-edge techniques for qa against existing techniques in a more fair, comparable, and precise manner than previous benchmarks. additionally, we hope that the linguistic diversity and large number of languages will inspire more researchers to treat model performance across many (partially less-resourced) languages as an important and worthy goal in itself. as mkqa offers the only open-qa option for many of these languages, we also hope to spark important research in these monolingual, non-english settings. ## conclusion in this work, we introduce a multilingual open domain question answering evaluation set. its properties, including geographical invariance, language-parallel questions, retrieval-independent annotations, and linguistic diversity, set it apart from existing resources in terms of annotation quality, difficulty, and flexibility to evaluate new approaches. we encourage future multilingual benchmarks to adopt data collection and annotation principles to promote higher-quality, and informative evaluation practices. we evaluate several baselines, based on state-of-the-art methods, and demonstrate ample room for improvement both in english and in the tail of lower-resourced languages. we hope that this evaluation set enables wider exploration of cross-lingual and monolingual methods in non-english qa.