MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages (2022)

Abstract: We present the results of the Workshop on Multilingual Information Access (MIA) 2022 Shared Task, evaluating cross-lingual open-retrieval question answering (QA) systems in 16 typologically diverse languages. In this task, we adapted two large-scale cross-lingual open-retrieval QA datasets in 14 typologically diverse languages, and newly annotated open-retrieval QA data in 2 underrepresented languages: Tagalog and Tamil. Four teams submitted their systems. The best constrained system uses entity-aware contextualized representations for document retrieval, thereby achieving an average F1 score of 31.6, which is 4.1 F1 absolute higher than the challenging baseline. The best system obtains particularly significant improvements in Tamil (20.8 F1), whereas most of the other systems yield nearly zero scores. The best unconstrained system achieves 32.2 F1, outperforming our baseline by 4.5 points.

## introduction open-retrieval 3 question answering (qa) is a task of answering questions in diverse domains given large-scale document collections such as wikipedia @xcite . despite the rapid progress in this area @xcite @xcite , the systems have primarily been evaluated in english, yet openretrieval qa in non-english languages has been understudied @xcite . moreover, due to the task complexity, cross-lingual open-retrieval qa has unique challenges such as multi-step inference (retrieval and 1 https://eval.ai/web/challenges/ challenge-page/1638/leaderboard 2 https://github.com/mia-workshop/ mia-shared-task-2022 in this work, we introduce the mia 2022 shared task on cross-lingual open-retrieval qa, which tests open-retrieval qa systems across typologically diverse languages. compared to previous efforts on multilingual open-retrieval qa @xcite , this shared task covers a wider set of languages (i.e., 16 topologically diverse languages) and orders of magnitude more passages in retrieval targets (i.e., 40 million passages in total), and constitutes the first shared task for massive-scale crosslingual open-retrieval qa. four teams submitted systems, three of which significantly improve the baseline system based on a state-of-the-art multilingual open-retrieval qa system @xcite . our analysis reveals that the system performance varies across languages even when the questions are parallel (as in one of our two settings), and several findings from the submitted systems shed light on the importance on entity-enhanced representations, leveraging more passages and data augmentation for future research in multilingual knowledge-intensive nlp. our analysis suggests that (i) it is still challenging to retrieve passages cross-lingually, (ii) generating answers in the target language whose script differs from the script of evidence document is nontrivial, (iii) and potential answer overlaps in existing datasets may overestimate models' performance. we formally introduce our task in section 2, followed by data collection process for 16 languages in section 3. we then introduce our baseline systems in section 4 and the submitted systems. section 5 presents our meta analysis of the systems performances, and we conclude by suggesting future improvements in this area. ## task descriptions we first formulate cross-lingual open-retrieval qa and introduce metrics used to evaluate systems' performance. we then present two submission tracks: constrained and unconstrained tracks. ## shared task data the mia shared task data is derived from two large-scale multilingual evaluation sets: xor-tydi qa @xcite and mkqa @xcite . we first discuss the source datasets, and then discuss how the target languages are selected, and how the data is split into training and evaluation sets. ## baseline models we use a state-of-the-art open-retrieval qa model as our baseline. we open source the code, trained checkpoints, training data, and intermediate/final prediction results. ). mdpr includes all of the target languages' wikipedias as its retrieval target, except for the two surprise languages. mgen g takes as input q and p and generates an answer a l in the target language: a @xmath0, p). mdpr is a multilingual extension of dpr @xcite , which employs a dual-encoder architecture based on bert @xcite and retrieves top passages based on the dot-product similarities between encoded representations. during training, mdpr optimizes the loss function as the negative log likelihood of the positive passages. mgen simply concatenates the question and a set of top k passages, and the fine-tuned multilingual encoderdecoder model generates a final answer in the target language. unlike some prior work in english conducting end-to-end training of the retriever and reader @xcite , we train mdpr and mgen independently. note that during mgen training, we use the passages retrieved by the trained mdpr, as in izacard and grave (2021a). ## main results tables 2 and 3 show final results on xor-tydi qa and mkqa subsets, respectively. three systems are submitted in the constrained setting, while texttron is an unconstrained submission. macro performance. texttron, mluke + mfid, and cmumqa significantly improve the baseline performance. among the constraint submissions, mluke + mfid yields the best performance. while several systems achieve higher than 40 average f1 on xor-tydi qa, only two systems achieve higher than 20 average f1 on mkqa, demonstrating how difficult it is to build a system that performs well in many languages without language-specific supervision. texttron significantly outperforms other baselines on xor-tydi qa while cmumqa shows the best mkqa performance among the submitted systems. language-wise performance. the performance varies across different languages. among xor-tydi qa, all of the systems struggle in korean and bengali, while in arabic, japanese and russian, they generally show relatively high f1 scores. on mkqa, where all of the questions are parallel, the performance still significantly differs across languages. almost all of the systems report lower than 10 f1 in khmer and tamil, which are less represented in existing pretraining corpora @xcite and use their own script systems-with the notable exception of mluke + fid, which achieves 20.8 f1 on tamil. mluke+fid achieves substantially better performance than other systems in tamil. this is partially because they also include the tamil wikipedia passages for passage retrieval, while other systems, including the baseline, do not. as discussed in @xcite , all systems show lower scores in the languages that are distant from english and use non-latin scripts (e.g., cyrillic for russian, hangul for korean). ## analysis we provide further analysis on the submitted systems. in section 7.1 we provide a brief summary of the findings from the submitted system descriptions. section 7.2 provides performance comparison over answer-type, and answer overlap with english or training data. we then analyze the degree of answer agreements among the submitted systems to understand which questions remain challenging in section 7.3. we further conduct manual error analysis in five languages in section 7.4. ## conclusion and discussions we have presented the mia 2022 shared task on cross-lingual open-retrieval qa systems in 16 typologically diverse languages, many of which are unseen during training. several submissions improved significantly over our baseline based on a state-of-the-art cross-lingual open-retrieval qa system and investigated a wide range of techniques. those results shed light on the effectiveness of several techniques in this challenging task, such as entity-enhanced representations, sparse-dense retrieval, and better interactions between passages. we further conducted detailed performance analysis on different subsets of the datasets, such as languages, answer types, the necessity of crosslingual retrieval as well as detailed error analysis. we also suggest several bottlenecks in the area.