XOR QA : Cross-lingual Open-Retrieval Question Answering (2021)

Abstract: Multilingual question answering tasks typically assume that answers exist in the same language as the question. Yet in practice, many languages face both information scarcity—where languages have few reference articles—and information asymmetry—where questions reference concepts from other cultures. This work extends open-retrieval question answering to a cross-lingual setting enabling questions from one language to be answered via answer content from another language. We construct a large-scale dataset built on 40K information-seeking questions across 7 diverse non-English languages that TyDi QA could not find same-language answers for. Based on this dataset, we introduce a task framework, called Cross-lingual Open-Retrieval Question Answering (XOR QA), that consists of three new tasks involving cross-lingual document retrieval from multilingual and English resources. We establish baselines with state-of-the-art machine translation systems and cross-lingual pretrained models. Experimental results suggest that XOR QA is a challenging task that will facilitate the development of novel techniques for multilingual question answering. Our data and code are available at https://nlp.cs.washington.edu/xorqa/ .

## introduction information-seeking questions-questions from people who are actually looking for an answerhave been increasingly studied in question answering (qa) research. fulfilling these information needs has led the research community to look further for answers: beyond paragraphs and articles toward performing open retrieval 1 on large-scale document collections @xcite ). yet 1 we use open retrieval-instead of open domain-to refer to models that can access answer context from large document collections. we avoid using open domain due to its double meaning as "covering topics from many domains." ロン・ポールの学部時代の専攻は？ @xcite (what did ron paul major in during undergraduate?) 生物学 (biology) paul went to gettysburg college, where he was a member of the lambda chi alpha fraternity. he graduated with a b.s. degree in biology in 1957. the bulk of this work has been exclusively on english. in this paper, we bring together for the first time information-seeking questions, open-retrieval qa, and multilingual qa to create a multilingual open-retrieval qa dataset that enables crosslingual answer retrieval. while multilingual open qa systems would benefit the many speakers of non-english languages, there are several pitfalls in designing such a dataset. first, a multilingual qa dataset should include questions from non-english native speakers to represent real-world applications. questions in most recent multilingual qa datasets @xcite @xcite are translated from english, which leads to english-centric questions such as questions about american sports, cultures and politics. second, it is important to support retrieving answers in languages other than the original language due to information scarcity of low-resource languages @xcite . moreover, questions strongly related to entities from other cultures are less likely to have answer content in the questioner's language due to cultural bias (information asymmetry, callahan and herring, 2011). for example, fig. 1 shows that the japanese wikipedia article of an american politician, ron paul, does not have information about his college degree perhaps because japanese wikipedia editors are less interested in specific educational backgrounds of american politicians. ## the xor-tydi qa dataset our xor-tydi qa dataset comprises questions inherited from tydi qa @xcite and answers augmented with our annotation process across 7 typologically diverse languages. we focus on cross-lingual retrieval from english wikipedia because in our preliminary investigation we were able to find answers to a majority of the questions from resource-rich english wikipedia, and native speakers with much annotation experience were readily available via crowdsourcing in english. ## xor qa tasks and baselines we introduce three new tasks (fig. 3 ): xor-retrieve, xor-englishspan, and xor-full with our newly collected xor-tydi qa dataset and construct strong baselines for each task. xor-full defines our goal of building a multilingual open-retrieval qa system that uses both cross-lingual and in-language questions from xor-tydi qa. to diagnose where models fail and to allow researchers to use the data with less coding effort or computational resource, we also introduce the first two intermediate tasks that only use the crosslingual data ( ## experiments and analysis we present results from the baselines discussed above. we find that the three xor qa tasks present challenges even for the strong models. ## related work multilingual qa much recent effort has been made to create non-english qa datasets to over in addition to the datasets we already discussed in §2.2, several other non-english reading comprehension datasets have been created @xcite @xcite . @xcite developed a template-based cloze task, leading to different data distributions from realistic questions with a great degree of lexical overlap between questions and reference paragraphs @xcite . more recently, @xcite introduced exams, a multilingual multiple-choice reading comprehension dataset from school exams. our xor-tydi qa is also closely related to @xcite @xcite @xcite @xcite ; both qa@clef and xor-tydi qa attempt to develop and evaluate multilingual qa systems. nevertheless, there are three crucial differences. first, our xor-tydi qa has a large number of questions that are required for training current state-of-the-art qa models like dpr, while qa@clef only has 200 evaluation questions for each language without training data @xcite . secondly, the languages tested in qa@clef are all european languages, with the one exception of indonesian; xor-tydi qa includes typologically diverse languages. lastly, the task setup of qa@clef 2003-2008 is either monolingual-questions and documents are written in the same non-english language-or crosslingual-the source and target languages are prespecified @xcite . in xor qa, questions are asked in a target language but a system does not know in which language it can find an answer in a non-parallel wikipedia collection. those differences from qa@clef tasks better simulate real-world scenarios and introduce new challenges that have yet to be extensively studied. ## conclusion we presented the task of xor qa, in which a system retrieves and reads documents across languages to answer non-english information-seeking questions. we introduced a new large-scale xor qa dataset, xor-tydi qa, with 40k newly annotated open-retrieval questions that cover seven typologically diverse languages. our experiments showed that xor-tydi qa is a challenging benchmark that can benefit from further effort in both qa and multilinguality communities.