Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer (2021)

Abstract: Multilingual pre-trained contextual embedding models (Devlin et al., 2019) have achieved impressive performance on zero-shot cross-lingual transfer tasks. Finding the most effective fine-tuning strategy to fine-tune these models on high-resource languages so that it transfers well to the zero-shot languages is a non-trivial task. In this paper, we propose a novel meta-optimizer to soft-select which layers of the pre-trained model to freeze during fine-tuning. We train the meta-optimizer by simulating the zero-shot transfer scenario. Results on cross-lingual natural language inference show that our approach improves over the simple fine-tuning baseline and X-MAML (Nooralahzadeh et al., 2020).

## introduction despite the impressive performance of neural models on a wide variety of nlp tasks, these models are extremely data hungry -training them requires a large amount of annotated data. as collecting such amounts of data for every language of interest is extremely expensive, cross-lingual transfer that aims to transfer the task knowledge from high-resource (source) languages for which annotated data are more readily available to lowresource (target) languages becomes a promising direction. cross-lingual transfer approaches using cross-lingual resources such as machine translation (mt) systems @xcite or bilingual dictionaries @xcite have effectively reduced the amount of annotated data required to obtain reasonable performance on the target language. however, such cross-lingual resources are often limited for low-resource languages. recent advances in cross-lingual contextual embedding models have reduced the need for cross- * work done while interning at amazon ai. lingual supervision @xcite . @xcite show that multilingual bert (mbert) @xcite , a contextual embedding model pre-trained on the concatenated wikipedia data from 104 languages without cross-lingual alignment, does surprisingly well on zero-shot cross-lingual transfer tasks, where they fine-tune the model on the annotated data from the source languages and evaluate on the target language. @xcite propose to freeze the bottom layers of mbert during fine-tuning to improve the cross-lingual performance over the simple fine-tune-all-parameters strategy, as different layers of mbert captures different linguistic information @xcite . selecting which layers to freeze for a downstream task is a non-trivial problem. in this paper, we propose a novel meta-learning algorithm for soft layer selection. our meta-learning algorithm learns layer-wise update rate by simulating the zero-shot transfer scenario -at each round, we randomly split the source languages into a heldout language and the rest as training languages, fine-tune the model on the training languages, and update the meta-parameters based on the model performance on the held-out language. we build the meta-optimizer on top of a standard optimizer and learnable update rates, so that it generalizes well to large numbers of updates. our method uses much less meta-parameters than the x-maml approach @xcite adapted from model-agnostic meta-learning (maml) @xcite to zero-shot cross-lingual transfer. experiments on zero-shot cross-lingual natural language inference show that our approach outperforms both the simple fine-tuning baseline and the x-maml algorithm and that our approach brings larger gains when transferring from multiple source languages. ablation study shows that both the layer-wise update rate and cross-lingual metatraining are key to the success of our approach. ## meta-learning for zero-shot cross-lingual transfer the idea of transfer learning is to improve the performance on the target task t 0 by learning from a set of related source tasks {t 1 , t 2 , ..., t k }. in the context of cross-lingual transfer, we treat different languages as separate tasks, and our goal is to transfer the task knowledge from the source languages to the target language. in contrast to the transfer learning case where the inputs of the source and target tasks are from the same language, in cross-lingual transfer learning we need to handle inputs from different languages with different vocabularies and syntactic structures. to handle the issue, we use the pre-trained multilingual bert @xcite , a language model encoder trained on the concatenation of monolingual corpora from 104 languages. the most widely used approach to zero-shot cross-lingual transfer using multilingual bert is to fine-tune the bert model θ on the source language tasks t 1...k with training objective l and then evaluate the fine-tuned model θ * on the target language task t 0 . the gap between training and testing can lead to sub-optimal performance on the target language. to address the issue, we propose to train a metaoptimizer f ϕ for fine-tuning so that the fine-tuned model generalizes better to unseen languages. we train the meta-optimizer by ## experiments we evaluate our meta-learning approach on natural language inference. natural language inference (nli) can be cast into a sequence pair classification problem where, given a premise and a hypothesis sentence, the model needs to predict whether the premise entails the hypothesis, contradicts it, or neither (neutral). we use the multi-genre natural language inference corpus @xcite , which consists of 433k english sentence pairs labeled with textual entailment information, and the xnli dataset (conneau 2019 ), the highest scores (with a selected subset of layers fixed during finetuning) in @xcite , the best zero-shot results using x-maml @xcite with one auxiliary language. we boldface the highest scores within each auxiliary language setting. et al., 2018), which has 2.5k development and 5k test sentence pairs in 15 languages including english (en), french (fr), spanish (es), german (de), greek (el), bulgarian (bg), russian (ru), turkish (tr), arabic (ar), vietnamese (vi), thai (th), chinese (zh), hindi (hi), swahili (sw), and urdu (ur). we use this dataset to evaluate the effectiveness of our meta-learning algorithm when transferring from english and one or more low-resource auxiliary languages to the target language. ## conclusion we propose a novel meta-optimizer that learns to soft-select which layers to freeze when fine-tuning a pretrained language model (mbert) for zeroshot cross-lingual transfer. our meta-optimizer learns the update rate for each layer by simulating the zero-shot transfer scenario where the model fine-tuned on the source languages is tested on an unseen language. experiments show that our approach outperforms the simple fine-tuning baseline and the x-maml algorithm on cross-lingual natural language inference.