Model Selection for Cross-lingual Transfer (2021)

Abstract: Transformers that are pre-trained on multilingual corpora, such as, mBERT and XLM-RoBERTa, have achieved impressive cross-lingual transfer capabilities. In the zero-shot transfer setting, only English training data is used, and the fine-tuned model is evaluated on another target language. While this works surprisingly well, substantial variance has been observed in target language performance between different fine-tuning runs, and in the zero-shot setup, no target-language development data is available to select among multiple fine-tuned models. Prior work has relied on English dev data to select among models that are fine-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices. In this paper, we show that it is possible to select consistently better models when small amounts of annotated data are available in auxiliary pivot languages. We propose a machine learning approach to model selection that uses the fine-tuned model’s own internal representations to predict its cross-lingual capabilities. In extensive experiments we find that this method consistently selects better models than English validation data across twenty five languages (including eight low-resource languages), and often achieves results that are comparable to model selection using target language development data.

## introduction pre-trained transformers @xcite have achieved state-of-the-art results on a range of nlp tasks, often approaching human inter-rater agreement @xcite . these models have also been demonstrated to learn effective cross-lingual representations, even without access to parallel text or bilingual lexicons @xcite . in the zero-shot transfer learning, training and development data are only assumed in a high resource source language (e.g. english), and performance is evaluated on another target language. because no target language annotations are assumed, source language data is typically used to select among models that are fine-tuned with different hyperparameters and random seeds. however, recent work has shown that english dev accuracy does not always correlate well with target language performance @xcite . in this paper, we propose an alternative strategy for model selection in zero-shot transfer. our approach, dubbed learned model selection (lms), learns a function that scores the compatibility between a fine-tuned multilingual transformer, and a target language. the compatibility score is calculated based on features of the multilingual model's learned representations. this is done by aggregating representations over an unlabeled target language text corpus after fine-tuning on source language data. we show that these modelspecific features effectively capture information about how the cross-lingual representations will transfer. we also make use of language embeddings from the lang2vec package (malaviya et al., 2017), foot_1 which have been shown to encode typological information that may help inform how a multilingual model will transfer to a particular target. these model and language features are combined in a bilinear layer to compute a ranking on the fine-tuned models. parameters of the ranking function are optimized to minimize a pairwise loss on a set of held-out models, using one or more auxiliary pivot languages. our method assumes training data in english, in addition to small amounts of auxiliary language data. this corresponds to a scenario where the multilingual model needs to be quickly adapted to a new language. lms does not rely on any annotated data in the target language, yet it is effective in learning to predict how well fine-tuned representations will transfer. in experiments on twenty five languages, lms consistently selects models with better targetlanguage performance than those chosen using english dev data. furthermore, our proposed approach improves performance on low-resource languages such as quechua, maori and turkmen that are not included in the pretraining corpus ( §6.1). ## background: cross-lingual transfer learning the zero-shot setting considered in this paper works as follows. a transformer model is first pretrained using a standard masked language model objective. the only difference from the monolingual approach to contextual word representations @xcite is the pre-training corpus, which contains text written in multiple languages; for example, mbert is trained on wikipedia data from 104 languages. after pre-training, the resulting network encodes language-independent representations that support surprisingly effective cross-lingual transfer, simply by fine-tuning with english data. for example, after fine-tuning mbert using the english portion of the conll named entity recognition dataset, the resulting model can perform inference directly on spanish text, achieving an f 1 score around 75, and outperforming prior work using cross-lingual word embeddings @xcite . a challenge, however, is the relatively high variance across multiple training runs. although mean f 1 on spanish is 75, the performance of 60 fine-tuned models with different learning rates and random seeds ranges from around 70 f 1 to 78. in zero-shot learning, no validation/development data is available in the target language, motivating the need for a machine learning approach to model selection. ## ranking model compatibility with a target language given a set of multilingual bert-based models, @xmath0, m 2 , ..., m n that are fine-tuned on an english training set using different hyperparameters and random seeds, our goal is to select the model that performs best on a specific target language, l target . our approach (lms) learns to rank a set of models based on two sources of information: (1) the models' own internal representations, and (2) lang2vec representations of the target language @xcite . we adopt a pairwise approach to learning to rank @xcite . the learned ranking is computed using a scoring function, s(m, l) = f (g mbert (m), g lang2vec (l)), where g mbert (m) is a feature vector for model m, which is computed from the model's own hidden state representations, and g lang2vec (l) is the lang2vec representation of language l. the model and language features are each passed through a feed-forward neural network and then combined using a bilinear layer to calculate a final score as follows: using the above score, we can represent the probability that model m i performs better than m j on language l: here 1[m j l m i ] is an indicator function that has the value 1 if m j outperforms m i , as evaluated using labeled development data in language l. the first sum in equation 1 ranges over all languages where development data is available (this excludes the target language). after tuning parameters to minimize cross-entropy loss on these languages, models are ranked based on their scores for the target language, and the highest scoring model, @xmath1, l target ), is selected. ## tasks and datasets we perform model selection experiments on five well-studied nlp tasks in the zero-shot transfer setting: part-of-speech (pos) tagging, question answering (qa), relation extraction (re), eventargument role labeling (arl), and named entity recognition (ner). in total, we cover 25 target languages including 8 low-resource languages in our experiments following prior work (shown in ## experimental design for a multilingual nlp task with n languages l : {l 1 , ..., l n }, our goal is to select the model that performs best on a new target language, l @xmath2. we assume the available resources are english training and development data, in addition to a small development set for each of the pivot languages, l. first, a set of mbert models, m , are finetuned on an english training set using different hyperparameters and random seeds and shuffled into meta-train/dev/test sets. we then evaluate each model, m i , on the pivot languages' dev sets to calculate a set of gold rankings, l , that are used in the cross-entropy loss (equation 1 ). model-specific features are extracted from the fine-tuned mberts, by feeding unlabeled pivot language text as input. development and evaluation mbert models in the meta-dev set are used to experiment with different model and language features. evaluation is performed using models in the meta-test set. we use the leave-one-language-out setup for each task during evaluation. for each target language, we rank models using the learned scoring function, select the highest scoring model, and report results in ## evaluation below we report model selection results on mberts in the meta-test set for each of the five tasks. significance compared to the en-dev is indicated with * * (@xmath3.05) -all tests are computed using the paired bootstrap procedure @xcite . ## analysis in section 6, we empirically demonstrated that our learned scoring function, s(•), consistently selects better models than the standard approach (en-dev), and is comparable to small amounts of labeled target language data. section 7.1 presents additional analysis of our approach, exploring the impact of various modeling choices with {ar, de, es, nl, zh}. in addition, analysis of generalization beyond mbert and across tasks capability are present in appendices a.3 and a.4. ## conclusion in this paper, we presented a machine learning approach to model selection for zero-shot crosslingual transfer, which is appropriate when small amounts of development data are available in one or more pivot languages, but not in the target language. we showed that our approach improves over the standard practice of model selection using source language development data. experiments on five well-studied nlp tasks show that by inspecting internal representations, our method consistently selects better models. lms also achieves comparable results to the more expensive alternative of annotating small amounts of target-language development data. finally, we demonstrated that lms selects better models for low-resource languages, such as quechua and maori, that are not included during pretraining. a.4 can multi-task learning help? our setting does not assume access to the labeled data in the target language for a particular task. however, labeled data in the target language may be available for a relevant auxiliary task, which could help the scoring function learn to better estimate whether a model is a good match for the target language. to test whether an auxiliary task in the target language might help to select a better model for the target task, we train the lms on two tasks: re and arl. gold rankings on the models are then computed for each language using the pivot languages' dev sets. also, another "silver" ranking is computed for each language using the auxiliary task. the scoring function is then trained to rank mbert models for both tasks. to differentiate the two tasks, a variant of the scoring function, s(m, l, t), which concatenates a randomly initialized task embedding with the language embedding is adopted. in