MLQA : Evaluating Cross-lingual Extractive Question Answering (2020)

Abstract: Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. Such annotated datasets are difficult and costly to collect, and rarely exist in languages other than English, making building QA systems that work well in other languages challenging. In order to develop such systems, it is crucial to invest in high quality multilingual evaluation benchmarks to measure progress. We present MLQA, a multi-way aligned extractive QA evaluation benchmark intended to spur research in this area. MLQA contains QA instances in 7 languages, English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese. MLQA has over 12K instances in English and 5K in each other language, with each instance parallel between 4 languages on average. We evaluate state-of-the-art cross-lingual models and machine-translation-based baselines on MLQA. In all cases, transfer results are shown to be significantly behind training-language performance.

## introduction question answering (qa) is a central and highly popular area in nlp, with an abundance of datasets available to tackle the problem from various angles, including extractive qa, cloze-completion, and open-domain qa @xcite @xcite . the field has made rapid advances in recent years, even exceeding human performance in some settings @xcite . despite such popularity, qa datasets in languages other than english remain scarce, even for relatively high-resource languages @xcite , as collecting such datasets at sufficient scale and quality is difficult and costly. there are two reasons why this lack of data prevents internationalization of qa systems. first, we cannot measure progress on multilingual qa without relevant benchmark data. second, we cannot easily train end-to-end qa models on the task, and arguably most recent successes in qa have been in fully supervised settings. given recent progress in cross-lingual tasks such as document classification @xcite @xcite , semantic role labelling @xcite and nli @xcite , we argue that while multilingual qa training data might be useful but not strictly necessary, multilingual evaluation data is a must-have. recognising this need, several cross-lingual datasets have recently been assembled @xcite . however, these generally cover only a small number of languages, combine data from different authors and annotation protocols, lack parallel instances, or explore less practically-useful qa domains or tasks (see section 3). highly parallel data is particularly attractive, as it enables fairer comparison across languages, requires fewer source language annotations, and allows for additional evaluation setups at no extra annotation cost. a purpose-built evaluation benchmark dataset covering a range of diverse languages, and following the popular extractive qa paradigm on a practically-useful domain would be a powerful testbed for cross-lingual qa models. with this work, we present such a benchmark, mlqa, and hope that it serves as an accelerator for multilingual qa in the way datasets such as squad @xcite have done for its monolingual counterpart. mlqa is a multi-way parallel extractive qa evaluation benchmark in seven languages: english, arabic, german, vietnamese, spanish, simplified chinese and hindi. to construct mlqa, we first automatically identify sentences from wikipedia articles which have the same or similar meaning in multiple languages. we extract the paragraphs that contain such sentences, then crowd-source questions on the english paragraphs, making sure the answer is in the aligned sentence. this makes it possible to answer the question in all languages in the vast majority of cases. 2 the generated questions are then translated to all target languages by professional translators, and answer spans are annotated in the aligned contexts for the target languages. the resulting corpus has between 5,000 and 6,000 instances in each language, and more than 12,000 in english. each instance has an aligned equivalent in multiple other languages (always including english), the majority being 4-way aligned. combined, there are over 46,000 qa annotations. we define two tasks to assess performance on mlqa. the first, cross-lingual transfer (xlt), requires models trained in one language (in our case english) to transfer to test data in a different language. the second, generalised cross-lingual transfer (g-xlt) requires models to answer questions where the question and context language is different, e.g. questions in hindi and contexts in arabic, a setting possible because mlqa is highly parallel. we provide baselines using state-of-the-art crosslingual techniques. we develop machine translation baselines which map answer spans based on the attention matrices from a translation model, and use multilingual bert @xcite and xlm @xcite as zero-shot approaches. we use english for our training language and adopt squad as a training dataset. we find that zero-shot xlm transfers best, but all models lag well behind training-language performance. in summary, we make the following contributions: i) we develop a novel annotation pipeline to construct large multilingual, highly-parallel extractive qa datasets ii) we release mlqa, a 7language evaluation dataset for cross-lingual qa iii) we define two cross-lingual qa tasks, including a novel generalised cross-lingual qa task iv) we provide baselines using state-of-the-art techniques, and demonstrate significant room for improvement. ## the mlqa corpus first, we state our desired properties for a crosslingual qa evaluation dataset. we note that whilst some existing datasets exhibit these properties, none exhibit them all in combination (see section 3). we then describe our annotation protocol, which seeks to fulfil these desiderata. ## related work monolingual qa data there is a great variety of english qa data, popularized by mctest @xcite , cnn/daily mail @xcite cbt @xcite , and wik-iqa @xcite amongst others. large span-based datasets such as squad @xcite , triviaqa @xcite , newsqa @xcite , and natural questions @xcite have seen extractive qa become a dominant paradigm. however, large, high-quality datasets in other languages are relatively rare. there are several chinese datasets, such as dureader @xcite , cmrc @xcite and drcd @xcite . more recently, there have been efforts to build corpora in a wider array of languages, such as korean @xcite and arabic @xcite . cross-lingual qa modelling cross-lingual qa as a discipline has been explored in qa for rdf data for a number of years, such as the qald-3 and 5 tracks @xcite , with more recent work from @xcite . @xcite explore an approach to use english qa data from squad to improve qa performance in korean using an in-language seed dataset. @xcite al. (2019 ) study question generation by leveraging english questions to generate better hindi questions, and lee and lee (2019) and cui et al. (2019a) develop modelling approaches to improve performance on chinese qa tasks using english resources. lee et al. (2019) and hsu et al. (2019) explore modelling approaches for zero-shot transfer and singh et al. (2019) explore how training with cross-lingual data regularizes qa models. cross-lingual qa data gupta et al. (2018) release a parallel qa dataset in english and hindi, hardalov et al. (2019) investigate qa transfer from english to bulgarian, liu et al. (2019b) release a cloze qa dataset in chinese and english, and jing et al. (2019) released bipar, built using parallel paragraphs from novels in english and chinese. these datasets have a similar spirit to mlqa, but are limited to two languages. asai et al. (2018) investigate extractive qa on a manuallytranslated set of 327 squad instances in japanese and french, and develop a phrase-alignment modelling technique, showing improvements over backtranslation. like us, they build multi-way parallel extractive qa data, but mlqa has many more instances, covers more languages and does not require manual document translation. liu et al. (2019a) explore cross-lingual open-domain qa with a dataset built from wikipedia "did you know?" questions, covering nine languages. unlike mlqa, it is distantly supervised, the dataset size varies by language, instances are not parallel, and answer distributions vary by language, making quantitative comparisons across languages challenging. finally, in contemporaneous work, artetxe et al. (2019) release xquad, a dataset of ‫اﻟﻘﺪﯾﻤﺔ‬ ‫اﻹﻧﺠﻠﯿﺰﯾﺔ‬ ‫اﻟﻜﻠﻤﺔ‬ ‫ﻣﻦ‬ ‫"إﻧﺠﻠﺘﺮا"‬ ‫اﺳﻢ‬ ‫ﯾﺸﺘﻖ‬ englaland، ‫واﺣﺪة‬ ‫ﻛﺎﻧﺖ‬ ‫واﻷﻧﺠﻞ‬ ‫اﻷﻧﺠﻞ".‬ ‫"أرض‬ ‫ﺗﻌﻨﻲ‬ ‫واﻟﺘﻲ‬ ‫ﻗﺪﯾﻤﺎ‬ ‫اﻟﻌﺮب‬ ‫ﺳﻤﺎھﺎ‬ ‫وﻗﺪ‬ [...] . ‫ا‬ ‫ر‬ ‫ا‬ ‫ﺧﻼلأوا‬ ‫إﻧﺠﻠﺘﺮا‬ ‫ﻓﻲ‬ ‫اﺳﺘﻘﺮت‬ ‫اﻟﺘﻲ‬ ‫اﻟﺠﺮﻣﺎﻧﯿﺔ‬ ‫اﻟﻘﺒﺎﺋﻞ‬ ‫ﻣﻦ‬ ‫اﻹﻧﻜﺘﺎر‬ the name "england" is derived from the old english name englaland [...] the angles were one of the germanic tribes that settled in great britain during the early middle ages. [...] the welsh name for the english language is "saesneg" der name england leitet sich vom altenglischen wort engaland [...] die angeln waren ein germanischer stamm, der das land im frühmittelalter besiedelte. [...] ein verweis auf die weißen klippen von dover. tên gọi của anh trong tiếng việt bắt nguồn từ tiếng trung. [...] người angle là một trong những bộ tộc german định cư tại anh trong thời đầu trung cổ. [...] dường như nó liên quan tới phong tục gọi người german tại anh là angli saxones hay anh -sachsen. during what time period did the angles migrate to great britain? ‫اﻟﻌﻈﻤﻰ؟‬ ‫ﺑﺮﯾﻄﺎﻧﯿﺎ‬ ‫إﻟﻰ‬ ‫اﻷﻧﺠﻞ‬ ‫ھﺎﺟﺮ‬ ‫زﻣﻨﯿﺔ‬ ‫ﺣﻘﺒﺔ‬ ‫أي‬ ‫ﻓﻲ‬ während welcher zeitperiode migrierten die angeln nach großbritannien? trong khoảng thời gian nào người angles di cư đến anh? powell library [...] the campus is in the residential area of westwood [...] the campus is informally divided into north campus and south campus, which are both on the eastern half of the university's land. [...] the campus includes [...] a mix of architectural styles. el campus incluye [...] una mezcla de estilos arquitectónicos. informalmente está dividido en campus norte y campus sur, ambos localizados en la parte este del terreno que posee la universidad. [...] el campus sur está enfocado en la ciencias físicas [...] y el centro médico ronald reagan de ucla. 1919 , [...] , [...] , , , , , what are the names given to the campuses on the east side of the land the university sits on? ¿cuáles son los nombres dados a los campus ubicados en el lado este del recinto donde se encuentra la universidad? , ? en de ar vi hi zh es en (a) (b) ‫اﻟﻘﺪﯾﻤﺔ‬ ‫اﻹﻧﺠﻠﯿﺰﯾﺔ‬ ‫اﻟﻜﻠﻤﺔ‬ ‫ﻣﻦ‬ ‫"إﻧﺠﻠﺘﺮا"‬ ‫اﺳﻢ‬ ‫ﯾﺸﺘﻖ‬ englaland، ‫واﺣﺪة‬ ‫ﻛﺎﻧﺖ‬ ‫واﻷﻧﺠﻞ‬ ‫اﻷﻧﺠﻞ".‬ ‫"أرض‬ ‫ﺗﻌﻨﻲ‬ ‫واﻟﺘﻲ‬ ‫ﻗﺪﯾﻤﺎ‬ ‫اﻟﻌﺮب‬ ‫ﺳﻤﺎھﺎ‬ ‫وﻗﺪ‬ [...] . ‫ا‬ ‫ر‬ ‫ا‬ ‫ﺧﻼلأوا‬ ‫إﻧﺠﻠﺘﺮا‬ ‫ﻓﻲ‬ ‫اﺳﺘﻘﺮت‬ ‫اﻟﺘﻲ‬ ‫اﻟﺠﺮﻣﺎﻧﯿﺔ‬ ‫اﻟﻘﺒﺎﺋﻞ‬ ‫ﻣﻦ‬ ‫اﻹﻧﻜﺘﺎر‬ the name "england" is derived from the old english name englaland [...] the angles were one of the germanic tribes that settled in great britain during the early middle ages. [...] the welsh name for the english language is "saesneg" der name england leitet sich vom altenglischen wort engaland [...] die angeln waren ein germanischer stamm, der das land im frühmittelalter besiedelte. [...] ein verweis auf die weißen klippen von dover. tên gọi của anh trong tiếng việt bắt nguồn từ tiếng trung. [...] người angle là một trong những bộ tộc german định cư tại anh trong thời đầu trung cổ. [...] dường như nó liên quan tới phong tục gọi người german tại anh là angli saxones hay anh -sachsen. during what time period did the angles migrate to great britain? ‫اﻟﻌﻈﻤﻰ؟‬ ‫ﺑﺮﯾﻄﺎﻧﯿﺎ‬ ‫إﻟﻰ‬ ‫اﻷﻧﺠﻞ‬ ‫ھﺎﺟﺮ‬ ‫زﻣﻨﯿﺔ‬ ‫ﺣﻘﺒﺔ‬ ‫أي‬ ‫ﻓﻲ‬ während welcher zeitperiode migrierten die angeln nach großbritannien? trong khoảng thời gian nào người angles di cư đến anh? powell library [...] the campus is in the residential area of westwood [...] the campus is informally divided into north campus and south campus, which are both on the eastern half of the university's land. [...] the campus includes [...] a mix of architectural styles. el campus incluye [...] una mezcla de estilos arquitectónicos. informalmente está dividido en campus norte y campus sur, ambos localizados en la parte este del terreno que posee la universidad. [...] el campus sur está enfocado en la ciencias físicas [...] y el centro médico ronald reagan de ucla. 1919 , [...] , [...] , , , , , what are the names given to the campuses on the east side of the land the university sits on? ¿cuáles son los nombres dados a los campus ubicados en el lado este del recinto donde se encuentra la universidad? , ? en de ar vi hi zh es en (a) (b) figure 2: (a) mlqa example parallel for en-de-ar-vi. (b) mlqa example parallel for en-es-zh-hi. answers shown as highlighted spans in contexts. contexts shortened for clarity with "[...]". aggregated cross-lingual benchmarks recently, following the widespread adoption of projects such as glue @xcite , there have been efforts to compile a suite of high quality multilingual tasks as a unified benchmark system. two such projects, xglue @xcite and xtreme @xcite incorporate mlqa as part of their aggregated benchmark. ## cross-lingual qa experiments we introduce two tasks to assess cross-lingual qa performance with mlqa. the first, cross-lingual transfer (xlt), requires training a model with (c x , q x , a x ) training data in language x, in our case english. development data in language x is used for tuning. at test time, the model must extract answer a y in language y given context c y and question q y . the second task, generalized cross-lingual transfer (g-xlt), is trained in the same way, but at test time the model must extract a z from c z in language z given q y in language y. this evaluation setup is possible because mlqa is highly parallel, allowing us to swap q z for q y for parallel instances without changing the question's meaning. as mlqa only has development and test data, we adopt squad v1.1 as training data. we use mlqa-en as development data, and focus on zeroshot evaluation, where no training or development data is available in target languages. models were trained with the squad-v1 training method from @xcite and implemented in pytext @xcite . we establish a number of baselines to assess current cross-lingual qa capabilities: translate-train we translate instances from the squad training set into the target language using machine-translation. 4 before translating, we enclose answers in quotes, as in @xcite . this makes it easy to extract answers from translated contexts, and encourages the translation model to map answers into single spans. we discard instances where this fails (∼5%). this corpus is then used to train a model in the target language. translate-test the context and question in the target language is translated into english at test time. we use our best english model to produce an answer span in the translated paragraph. for all languages other than hindi, 5 we use attention scores, a ij , from the translation model to map the answer back to the original language. rather than aligning spans by attention argmax, as by @xcite , we identify the span in the original context which maximizes f1 score with the english span: ## discussion it is worth discussing the quality of context paragraphs in mlqa. our parallel sentence mining approach can source independently-written documents in different languages, but, in practice, articles are often translated from english to the target languages by volunteers. thus our method sometimes acts as an efficient mechanism of sourcing existing human translations, rather than sourcing independently-written content on the same topic. the use of machine translation is strongly discouraged by the wikipedia community, foot_6 but from examining edit histories of articles in mlqa, machine translation is occasionally used as an article seed, before being edited and added to by human authors. our annotation method restricts answers to come from specified sentences. despite being provided several sentences of context, some annotators may be tempted to only read the parallel sentence and write questions which only require a single sentence of context to answer. however, single sentence context questions are a known issue in squad annotation in general @xcite suggesting our method would not result in less challenging questions, supported by scores on mlqa-en being similar to squad (section 5.3). mlqa is partitioned into development and test splits. as mlqa is parallel, this means there is development data for every language. since mlqa will be freely available, this was done to reduce the risk of test data over-fitting in future, and to estab-lish standard splits. however, in our experiments, we only make use of the english development data and study strict zero-shot settings. other evaluation setups could be envisioned, e.g. by exploiting the target language development sets for hyperparameter optimisation or fine-tuning, which could be fruitful for higher transfer performance, but we leave such "few-shot" experiments as future work. other potential areas to explore involve training datasets other than english, such as cmrc @xcite , or using unsupervised qa techniques to assist transfer @xcite . finally, a large body of work suggests qa models are over-reliant on word-matching between question and context @xcite . g-xlt represents an interesting testbed, as simple symbolic matching is less straightforward when questions and contexts use different languages. however, the performance drop from xlt is relatively small (8.2 mean f1), suggesting word-matching in cross-lingual models is more nuanced and robust than it may initially appear. ## conclusion we have introduced mlqa, a highly-parallel multilingual qa benchmark in seven languages. we developed several baselines on two cross-lingual understanding tasks on mlqa with state-of-the-art methods, and demonstrate significant room for improvement. we hope that mlqa will help to catalyse work in cross-lingual qa to close the gap between training and testing language performance.