Zero-Shot Cross-Lingual Transfer with Meta Learning (2020)

Abstract: Learning what to share between tasks has become a topic of great importance, as strategic sharing of knowledge has been shown to improve downstream task performance. This is particularly important for multilingual applications, as most languages in the world are under-resourced. Here, we consider the setting of training models on multiple different languages at the same time, when little or no data is available for languages other than English. We show that this challenging setup can be approached using meta-learning: in addition to training a source language model, another model learns to select which training instances are the most beneficial to the first. We experiment using standard supervised, zero-shot cross-lingual, as well as few-shot cross-lingual settings for different natural language understanding tasks (natural language inference, question answering). Our extensive experimental setup demonstrates the consistent effectiveness of meta-learning for a total of 15 languages. We improve upon the state-of-the-art for zero-shot and few-shot NLI (on MultiNLI and XNLI) and QA (on the MLQA dataset). A comprehensive error analysis indicates that the correlation of typological features between languages can partly explain when parameter sharing learned via meta-learning is beneficial.

## introduction there are more than 7,000 languages spoken in the world, over 90 of which have more than 10 million native speakers each @xcite . despite this, very few languages have proper linguistic resources when it comes to natural language understanding tasks @xcite . although there is growing awareness in the field, as evidenced by the release of datasets such as xnli @xcite , most nlp research still only considers english @xcite . while one solution to this issue is to collect annotated data for all languages, this process is both too time-consuming and expensive to be feasible. additionally, it is not trivial to train a model for a task in a particular language (e.g., english) and apply it directly to another language where only limited training data is available (i.e., low-resource languages). therefore, it is essential to investigate strategies that allow one to use the large amount of training data available for english for the benefit of other languages. meta-learning has recently been shown to be beneficial for several machine learning tasks @xcite @xcite @xcite . for nlp, recent work has also shown the benefits of this sharing between tasks and domains @xcite @xcite . although cross-lingual transfer with meta-learning has been investigated for machine translation @xcite , this paper -to best of our knowledge -is the first attempt to study meta-learning for cross-lingual natural language understanding. our contributions are as follows: ## meta-learning meta-learning tries to tackle the problem of fast adaptation to a handful of new training data instances. it discovers the structure among multiple tasks such that learning new tasks can be done quickly. this is done by repeatedly simulating the learning process on low-resource tasks using many high-resource ones @xcite . there are several ways of performing meta-learning: (i) metric-based @xcite ; (ii) model-based @xcite ; and to the best of our knowledge, we are the first to exploit the idea of meta-learning for transferring zeroshot knowledge in a cross-lingual setting for natural language understanding, in particular for the tasks of nli and qa. specifically, we exploit the usage of model agnostic meta-learning (maml) which uses gradient descent and achieves a good generalisation for a variety of tasks @xcite . maml is able to quickly adapt to new target tasks by using only a few instances at test time, assuming that these new target tasks are drawn from the same distribution. formally, maml assumes that there is a distribution p(t ) of tasks {t 1 , t 2 , ..., t k }. the parameters θ of model m for a particular task t i , sampled from the distribution p(t ), are updated to θ i . in particular, the parameters θ are updated using one or a few iterations of gradient descent steps on the training examples (i.e., d train i ) of task t i . for example, for one gradient update, ## cross-lingual meta-learning the underlying idea of using maml in nlp tasks @xcite @xcite is to employ a set of high-resource auxiliary tasks/languages to find an optimal initialisation from which learning a target task/language can be done using only a small number of training instances. in a cross-lingual setting (i.e., xnli, mlqa), where only an english dataset is available as a high-resource language, and a small number of instances are available for other languages, the training procedure for maml requires some non-trivial changes. for this purpose, we introduce a cross-lingual meta-learning framework (x-maml), which uses the following training steps: compute adapted parameters with gradient descent: a more formal description of the proposed model x-maml is given in algorithm 1. natural language inference (nli): nli is the task of predicting whether a hypothesis sentence is true (entailment), false (contradiction), or undetermined (neutral) given a premise sentence. the multi-genre natural language inference (multinli) dataset has 433k sentence pairs annotated with textual entailment information @xcite . it covers a range of different genres of spoken and written text and thus supports cross-genre evaluation. the nli premise sentences are provided in 10 different genres: facetoface, telephone, verbatim, state, government, fiction, letters, nineeleven, travel and oup. all of the genres appear in the test and development sets, but only five are included in the training set. to verify our learning routine more generally, we define t i as an nli task in each genre. we exploit maml, in its original setting, to investigate whether meta-learning encourages the model to learn a good initialisation for all target genres, which can then be fine-tuned with limited supervision for each genre's development instances @xcite to achieve a good performance on its test set. the cross-lingual natural language inference (xnli) dataset @xcite consists of 5000 test and 2500 development hypothesispremise pairs with their textual entailment labels for english. translations of these pairs are provided in 14 languages: french (fr), spanish (es), german (de), greek (el), bulgarian (bg), russian (ru), turkish (tr), arabic (ar), vietnamese (vi), thai (th), chinese (zh), hindi (hi), swahili (sw) and urdu (ur). xnli provides a multilingual benchmark to evaluate how to perform inference in lowresource languages, in which only training data for the high-resource language english is available from multinli. this allows us to study the impact of meta-learning with one low-resource language to serve as an auxiliary language, and evaluate the resulting nli model on the target languages provided in the xnli test set. question answering (qa): given a context and a question, the task in qa is to identify the span in the context which answers the question. @xcite introduce a multilingual question answering dataset (mlqa) that contains qa instances in 7 languages: english (en), arabic (ar), german (de), spanish (es), hindi (hi), vietnamese (vi) and simplified chinese (zh). it includes over 12k qa instances in english and 5k for every other language, with each qa instance being available in 4 languages (on average). this dataset has been used in many recent studies on cross-lingual transfer learning (e.g., hu et al. (2020); liang et al. (2020) ). in our experiments, we investigate meta-learning for qa with one or two auxiliary languages. ## experiments we want to investigate how meta-learning can be used for cross-lingual sharing. we implement x-maml using the higher library foot_1 . we use the adam optimiser (kingma and ba, 2014) with a batch size of 32 for both zero-shot and few-shot learning. we fix the step size α and learning rate β to 1e-4 and 1e-5, respectively. we experimented using @xcite @xcite @xcite @xcite meta-learning iterations in x-maml. however, 100 iterations led to the best results in our experiments. the sample sizes k and q in x-maml are equal to 16 for each dataset. the results are reported for each experiment by averaging the performance over ten different runs. we experiment with different architectures in order to verify that our method generalises across them, further detailed below. we report results for few-shot, zero-shot cross-domain and cross-lingual learning. ## related work the main motivation for this work is the low availability of labelled training datasets for most of the world's languages. to alleviate this issue, a number of methods, including the so-called few-shot learning approaches have been proposed. few-shot learning methods have initially been introduced within the area of image classification @xcite @xcite , but have recently also been applied to nlp tasks such as relation extraction @xcite ), text classification @xcite and machine translation @xcite . specifically, in nlp, these few-shot learning approaches include: (i) the transformation of the problem into a different task (e.g., relation extraction is transformed to question answering @xcite ); or (ii) meta-learning @xcite . meta-learning: meta-learning or learning-tolearn has recently received a lot of attention from the nlp community. first-order maml has been applied to the task of machine translation @xcite , where they propose to use meta-learning for improving the machine translation performance for low-resource languages by learning to adapt to target languages based on multilingual high-resource languages. however, in the proposed framework, they include 18 high-resource languages as auxil-iary languages and five diverse low-resource languages as target languages. in our work, we assume access to only english as a high-resource language. for the task of dialogue generation, @xcite address domain adaptation using meta-learning. @xcite explore maml variants thereof for low-resource nlu tasks in the glue dataset @xcite . they consider different high-resource nlu tasks such as multinli @xcite and qnli @xcite as auxiliary tasks to learn meta-parameters using maml. they then finetune the low-resource tasks using the adapted parameters from the meta-learning phase. all the above-mentioned works on meta-learning in nlp assume that there are multiple high-resource tasks or languages, which are then adapted to new target tasks or languages with a handful of training samples. however, in a cross-lingual nli and qa setting, the available high-resource language is usually only english. our work thus fills an important gap in the literature, as we only require a single source language. cross-lingual nlu: cross-lingual learning has a fairly short history in nlp, and has mainly been restricted to traditional nlp tasks, such as pos tagging, morphological inflection and parsing. in contrast to these tasks, which have seen much cross-lingual attention @xcite @xcite , there has been relatively little work on cross-lingual nlu, partly due to lack of benchmark datasets. existing work has mainly been focused on nli @xcite @xcite , and to a lesser degree on re @xcite and qa @xcite . previous research generally reports that cross-lingual learning is challenging and that it is hard to beat a machine translation baseline (e.g., @xcite ). such a baseline is for instance suggested by @xcite , where the text in the target language is automatically translated to english. we achieve competitive performance compared to a machine translation baseline (for xnli), and propose a method that requires no training instances for the target task in the target language. furthermore, our method is model agnostic, and can be used to extend any pre-existing model. typological correlations: in order to better explain our results for cross-lingual zero-shot and few-shot learning, we investigate typological fea-tures, and their overlap between target and auxiliary languages. we evaluate on the world atlas of language structure (wals, dryer and haspelmath ( 2013 )), which is the largest openly available typological database. it comprises approximately 200 linguistic features with annotations for more than 2500 languages, which have been made by expert typologists through study of grammars and field work. we draw inspiration from previous work @xcite @xcite which attempts to predict typological features based on language representations learned under various nlp tasks. similarly, we experiment with two conditions: (i) we attempt to predict typological features based on the mutual gain/loss in performance using x-maml; (ii) we investigate whether sharing between two typologically similar languages is beneficial for performance using x-maml. we train a simple logistic regression classifier per condition above, for each wals feature. in the first condition (i), the task is to predict the exact wals feature value of a language, given the change in accuracy in combination with other languages. in the second condition (ii), the task is to predict whether a main and auxiliary language have the same wals feature value, given the change in accuracy when the two languages are used in x-maml. we compare with two simple baselines, one based on always predicting the most frequent feature value in the training set, and one based on predicting feature values with respect to the distribution of feature values in the training set. we then investigate whether any features could be consistently predicted above baseline levels, given different test-training splits. we apply a simple paired t-test to compare our models predictions to the baselines. as we are running a large number of tests (one per wals feature), we apply bonferroni correction, changing our cut-off p-value from @xmath0.05 to @xmath1.00025. we first investigate few-shot x-maml, when using multi-bert, as reported in ## conclusion in this work, we show that meta-learning can be used to effectively leverage training data from an auxiliary language for zero-shot and few-shot crosslingual transfer. we evaluated this on two challenging nlu tasks (nli and qa), and on a total of 15 languages. we are able to improve the performance of state-of-the-art baseline models for (i) zero-shot xnli, and (ii) zero-shot qa on the mlqa dataset. furthermore, we show in a typological analysis that languages which share certain morphosyntactic features tend to benefit from this type of transfer. future studies will extend this work to other crosslingual nlp tasks and more languages.