X - METRA - ADA : Cross-lingual Meta-Transfer learning Adaptation to Natural Language Understanding and Question Answering (2021)

Abstract: Multilingual models, such as M-BERT and XLM-R, have gained increasing popularity, due to their zero-shot cross-lingual transfer learning capabilities. However, their generalization ability is still inconsistent for typologically diverse languages and across different benchmarks. Recently, meta-learning has garnered attention as a promising technique for enhancing transfer learning under low-resource scenarios: particularly for cross-lingual transfer in Natural Language Understanding (NLU). In this work, we propose X-METRA-ADA, a cross-lingual MEta-TRAnsfer learning ADAptation approach for NLU. Our approach adapts MAML, an optimization-based meta-learning approach, to learn to adapt to new languages. We extensively evaluate our framework on two challenging cross-lingual NLU tasks: multilingual task-oriented dialog and typologically diverse question answering. We show that our approach outperforms naive fine-tuning, reaching competitive performance on both tasks for most languages. Our analysis reveals that X-METRA-ADA can leverage limited data for faster adaptation.

## introduction cross-lingual transfer learning is a technique used to adapt a model trained on a downstream task in a source language to directly generalize to the task in new languages. it aims to come up with common cross-lingual representations and leverages them to bridge the divide between resources to make any nlp application scale to multiple languages. this is particularly useful for data-scarce scenarios, as it reduces the need for api calls implied by machine translation or costly task-specific annotation for new languages. figure 1 : an overview of the x-metra-ada framework: we use english as the source and spanish as the target language. the meta-train stage transfers from the source to the target languages, while the meta-adaptation further adapts the model to the target language. the application is few-shot if the test language is seen in any stage of x-metra-ada; or zeroshot if the test language is unseen. transformer-based contextualized embeddings and their multilingual counterparts such as m-bert @xcite have become popular as off-the-shelf representations for cross-lingual transfer learning. while these multilingual representations exhibit some cross-lingual capability even for languages with low lexical overlap with english, the transfer quality is reduced for languages that exhibit different typological characteristics @xcite . the generalization of such representations has been extensively evaluated on traditional tasks such as part-of-speech (pos) tagging, named entity recognition (ner) and cross-lingual document classification (cldc) @xcite @xcite , with ever-growing open community annotation efforts like universal dependencies @xcite and conll shared tasks @xcite . on the other hand, crosslingual natural language understanding (nlu) tasks have gained less attention, with smaller benchmark datasets that cover a handful of languages and don't truly model linguistic variety @xcite . natural language understanding tasks are critical for dialog systems, as they make up an integral part of the dialog pipeline. understanding and improving the mechanism behind cross-lingual transfer for natural language understanding in dialog systems require evaluations on more challenging and typologically diverse benchmarks. numerous approaches have attempted to build stronger cross-lingual representations on top of those multilingual models; however, most require parallel corpora @xcite and are biased towards highresource and balanced setups. this fuels the need for a method that doesn't require explicit crosslingual alignment for faster adaptation to lowresource setups. meta-learning, a method for "learning to learn", has found favor especially among the computer vision and speech recognition communities @xcite @xcite . meta-learning has been used for machine translation @xcite , few-shot relation classification @xcite , and on a variety of glue tasks @xcite . recently, @xcite apply the maml @xcite algorithm to crosslingual transfer learning for xnli @xcite and mlqa @xcite , nlu tasks that are naturally biased towards machine translation-based solutions. nooralahzadeh et al. are able to show improvement over strong multilingual models, including m-bert. however, they mainly show the effects of meta-learning as a first step in a framework that relies on supervised finetuning, making it difficult to properly compare and contrast both approaches. we study cross-lingual meta-transfer learning from a different perspective. we distinguish between meta-learning and fine-tuning and design systematic experiments to analyze the added value of meta-learning compared to naive finetuning. we also build our analysis in terms of more typologically diverse cross-lingual nlu tasks: multilingual task-oriented dialogue system (mtod) @xcite and typologically diverse question answering (ty-diqa) @xcite . while xnli is a clas-sification task, mtod is a joint classification and sequence labelling task and is more typologically diverse. tydiqa is not a classification task, but we show how meta-learning can be applied usefully to it. we also show greater performance improvements from meta-learning than fine-tuning on transfer between typologically diverse languages. to the best of our knowledge, we are the first to conduct an extensive analysis applied to mtod and tydiqa to evaluate the quality of cross-lingual meta-transfer. our contributions are three-fold: • proposing x-metra-ada, foot_0 a languageagnostic meta-learning framework (figure 1 ), and extensively evaluating it. ## methodology we make use of optimization-based meta-learning on top of pre-trained models with two levels of adaptation to reduce the risk of over-fitting to the target language: (i) meta-training from the source language to the target language(s) (ii) metaadaptation on the same target language(s) for more language-specific adaptation (figure 1 ). we apply our approach to two cross-lingual downstream tasks: mtod (section 2.1) and ty-diqa (section 2.2). we start by describing the base architectures for both tasks, before explaining how they are incorporated into our meta-learning pipeline. applying meta-learning to a task requires the construction of multiple 'pseudo-tasks', which are instantiated as pairs of datasets. we describe this construction for our downstream tasks in section 2.3. finally, we present our x-metra-ada algorithm (section 2.4). ## conclusion in this paper, we adapt a meta-learning approach for cross-lingual transfer learning in natural language understanding tasks. our experiments cover two challenging cross-lingual benchmarks: taskoriented dialog and natural questions including an extensive set of low-resource and typologically diverse languages. x-metra-ada reaches better convergence stability on top of fine-tuning, reaching a new state of the art for most languages.