M 2 QA : Multi-domain Multilingual Question Answering (2024)

Abstract: Generalization and robustness to input variation are core desiderata of machine learning research. Language varies along several axes, most importantly, language instance (e.g. French) and domain (e.g. news). While adapting NLP models to new languages within a single domain, or to new domains within a single language, is widely studied, research in joint adaptation is hampered by the lack of evaluation datasets. This prevents the transfer of NLP systems from well-resourced languages and domains to non-dominant language-domain combinations. To address this gap, we introduce M2QA, a multi-domain multilingual question answering benchmark.M2QA includes 13,500 SQuAD 2.0-style question-answer instances in German, Turkish, and Chinese for the domains of product reviews, news, and creative writing. We use M2QA to explore cross-lingual cross-domain performance of fine-tuned models and state-of-the-art LLMs and investigate modular approaches to domain and language adaptation.We witness **1)** considerable performance _variations_ across domain-language combinations within model classes and **2)** considerable performance _drops_ between source and target language-domain combinations across all model sizes. We demonstrate that M2QA is far from solved, and new methods to effectively transfer both linguistic and domain-specific information are necessary.

## introduction one of the central goals of natural language processing (nlp) is to develop systems that generalize well across different distributions, such as texts in different languages and domains. 2 while transformer models have brought tremendous progress in nlp in recent years, especially evident with the recent emergence of large language models (llms), the problem of generalizing to new domains and languages is still far from solved. incontext learning (icl), which refers to the ability of llms to perform tasks based on examples or instructions in the input prompt @xcite , is likely the reason for their emergent abilities @xcite ) -yet, even with in-context learning, transformers cannot generalize beyond their pretraining data @xcite , and their performance varies considerably across languages and is particularly low in languages underrepresented in the training data @xcite . with over 7,000 documented languages @xcite and countless domains, ensuring sufficient pre-training data coverage for every possible language-domain pair is hardly feasible. this motivates the development of methods that allow nlp systems to adapt to new languages and domains. while isolated language (e.g. @xcite @xcite @xcite and domain adaptation (e.g. @xcite are extensively covered in prior work, the lack of comprehensive multi-domain multilingual benchmarks makes it difficult to objectively evaluate joint language and domain transfer methods. existing multidomain multilingual benchmarks either contain only one language in addition to english @xcite , use machine-generated text @xcite or are task-oriented dialogue systems that use narrow application-specific domains rather than a diverse set of domains useful for a wide range of applications @xcite . results on these benchmarks suggest that language and domain are not independent axes. therefore, we cannot infer the performance of joint transfer from individual axes, making it hard to systematically compare nlp models across languages and domains and to study joint language and domain adaptation approaches. to address this gap, we introduce m2qa, a multi-domain multilingual squad 2.0-style @xcite ) extractive question answering (qa) dataset. we manually annotate naturally occurring texts in the respective languages -as opposed to translating documents from englishin order to increase lexical diversity @xcite , mitigate the introduction of artifacts @xcite such as "translationese" @xcite , and integrate the cultural idiosyncrasies of the target language @xcite . the new benchmark makes it possible to study how well existing models perform at joint language and domain transfer (rq1), whether specific language-domain combinations are especially hard to tackle for current models (rq2), and whether existing methods (e.g. full fine-tuning, modular setups, icl / instructionbased methods for llms) compare on language and domain transfer (rq3). in summary, our paper makes the following contributions: 1) we create a multi-domain multilingual extractive qa benchmark, covering three domains (product reviews, news, creative writing) and three languages (german, turkish, chinese), resulting in 13,500 qa instances (figure 1 ), with 8,100 answerable and 5,400 unanswerable questions. 2) we evaluate baseline and transfer performance using a wide range of models and transfer techniques, including fully-finetuned models, modular transfer learning and llms. 3) we find that transfer performance considerably varies across domain-language combinations. 4) we find that the widely used squad 2.0 evaluation metric implementation is insufficient for evaluating multilingual extractive qa due to its reliance upon whitespace tokenization and propose a version of the metric that mitigates the issue. 5) our results show that modern llms perform considerably worse on their target than on their source domain-language pair, highlighting the need for further research into methods that transfer both linguistic and domainspecific information. ## experiments the curation of the m2qa benchmark allows us -for the first time -to explore the transfer capabilities of state-of-the-art lms along multiple dimensions. we will use m2qa to investigate the following research questions: (rq1) how well do existing models perform at transfer learning across language and domains jointly? (rq2) what language domain combinations are especially hard to tackle for the current models? (rq3) how do modular adapter-based methods compare to fullyfinetuned models in domain and language transfer? ## further analysis in contrast to english, german, and turkish, which use whitespace characters to separate words, in chinese typesetting the use of whitespace is not required. while the texts from our chinese product review and creative writing sources do not contain whitespaces, chinese news do. we hypothesize that this typographical difference between chinese and the other languages can lead to a substantial drop in measured performance (e.g. xlm-r base achieves an f1 score of 0.11 on answerable creative writing questions), and investigate this further. ## conclusion generalization is a central goal of nlp that is yet unsolved. language and domain are two main axes of variation for natural languages -yet the lack of cross-lingual cross-domain datasets has prevented systematic evaluation of nlp models and transfer approaches across languages and domains. to address this, we introduce m2qa, a multi-domain multilingual question answering benchmark with over 13k human-annotated instances across three typologically diverse languages (german, turkish, chinese) and three distinct domains (product reviews, news, creative writing). our evaluation includes xlm-r baselines, llms (gpt-3.5, aya 23, llama 2 and 3), and adapter-based setups (mad-x+domain and mad-x 2 ), revealing a large gap between llms and fine-tuned lms. we expect that m2qa will help close this gap, increase our understanding of generalization, and find more effective domain and language transfer methods.