Don’t Use E nglish Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings (2020)

Abstract: Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning, where multilingual BERT is fine-tuned on one source language and evaluated on a different target language. However, published results for mBERT zero-shot accuracy vary as much as 17 points on the MLDoc classification task across four papers. We show that the standard practice of using English dev accuracy for model selection in the zero-shot setting makes it difficult to obtain reproducible results on the MLDoc and XNLI tasks. English dev accuracy is often uncorrelated (or even anti-correlated) with target language accuracy, and zero-shot performance varies greatly at different points in the same fine-tuning run and between different fine-tuning runs. These reproducibility issues are also present for other tasks with different pre-trained embeddings (e.g., MLQA with XLM-R). We recommend providing oracle scores alongside zero-shot results: still fine-tune using English data, but choose a checkpoint with the target dev set. Reporting this upper bound makes results more consistent by avoiding arbitrarily bad checkpoints.

## introduction zero-shot and zero-resource cross-lingual nlp has seen significant progress in recent years. the discovery of cross-lingual structure in word embedding spaces culminated in the work of @xcite , which showed that unsupervised word translation via adversarial mappings is competitive with supervised techniques. concurrent work in machine translation also showed that it is possible to achieve non-trivial bleu scores without any bitext @xcite . self-supervised multilingual contextual embeddings like mbert @xcite and xlm @xcite have shown remarkably strong performance on cross-lingual named entity recognition, text classification, dependency parsing, and other tasks (e.g., @xcite @xcite . much of this recent work has demonstrated that mbert performs very well on zero-shot tasks, superseding prior techniques as the baseline for zeroshot cross-lingual transfer learning. by zero-shot, we mean that no parallel text or labeled data from the target language was used during model training, fine-tuning, or hyperparameter search. in this setting, models are trained on labeled (usually english) text and tested on target (non-english) text. standard practice prohibits the use of target language data for model selection; the final model is chosen using the english dev set only. even though the authors report english accuracies which are basically identical, their target language performances are very different. given that each experiment starts with the same pre-trained mbert model and mldoc dataset, it is clear that these cross-lingual results are not reproducible. for the listed target languages, the highest accuracy is up to 3 points better than the next best and up to 17 points better than the worst. we investigate this reproducibility issue in both mldoc and xnli @xcite , which is another major dataset for evaluating cross-lingual transfer. the variations in published baselines on these and other datasets are summarized in sec. 6. in section 3, we show that the final zero-shot accuracies between and within independent mbert training runs are highly variable. variations over different random seeds are similar in magnitude to those in poor correlation between en dev and target test accuracy, combined with high variance between independent runs, means that published zero-shot accuracies are somewhat arbitrary. in addition to zero-shot results, we recommend reporting oracle results, where one still fine-tunes using en data, but uses the target dev set for checkpoint selection. ## experimental setup we use cased mbert (110m parameters) for all of our experiments. we illustrate the reproducibility issues in zero-shot cross-lingual transfer through the document classification task in mldoc foot_0 and the natural language inference task in xnli. 2 for the mldoc and xnli experiments, we used their full en training sets (10k and 393k examples, re-spectively). unless stated otherwise, we use their en development sets (1k and 2.5k examples) and non-en test sets (4k and 5k examples). when finetuning mbert, we used a constant learning rate of 2 × 10 -6 with a batch size of 32 for mldoc, and a constant learning rate of 2 × 10 -5 with a batch size of 32 for xnli. checkpoints were saved at regular intervals: one checkpoint for every 2% of the training corpus processed. models were trained until convergence based on en dev accuracy. we used the mbert implementation in glu-onnlp @xcite and an aws p3.8xlarge instance for our mldoc and xnli runs, which completed in ∼3 and ∼12 hours respectively on a single nvidia v100 gpu. ## between-run and within-run variation running mbert fine-tuning under different random seeds yields highly variable results, similar to what we observed in first, we observed significant variation between independent runs ( indeed, for all of the mldoc languages and for 7 of the 14 xnli languages, the variation across 10 runs exceeds 2.5% (absolute), even though the en dev accuracy only varies within a narrow 0.9% range. to put this variation in context, we note that the mldoc and xnli test sets are relatively large, so a 2.5% difference in accuracy would be statistically significant (at the 5% significance level using the usual test of proportions), which means that a paper claiming 'state-of-the-art' performance on these zero-shot tasks may be reporting strong results because of the large between-run variation, rather than a genuine improvement due to their proposed technique. in other words, using the en dev accuracy for choosing the final model for zero-shot transfer leads to inconsistent results across different runs. the en dev accuracy is similar for each of our independent experiments, but the target test accuracy for each experiment fluctuates in a wide band. ## english dev accuracy and its relationship with zero-shot accuracy experimenters use the en dev set for model selection under the assumption that zero-shot performance improves as en dev performance improves. we show that this assumption is often false. we compare the abilities of en dev and target dev to predict directional changes in target test accuracy. in we randomly sample pairs of checkpoints where the change in target test accuracy for the pair is at least 0.5%, and compute the proportion for which the en dev accuracy changed in the same direction. ## catastrophic forgetting the strange phenomenon in in figure 1 , we plotted the en dev accuracy and the target test accuracy over time, for the language with the highest directional agreement (es, 0.59) and for the language with the lowest directional agreement (ja, 0.42) for mldoc (see we conclude that en dev accuracy cannot detect when mbert is improving on the en training data at the expense of non-en languages, and should not (solely) be used to assess zero-shot performance. ## published results at the time of writing, over 70 papers have been published on the mldoc and xnli corpora. several new datasets (e.g., @xcite @xcite for zero-shot crosslingual evaluation have been released and aggregated into benchmark suites like xglue @xcite and xtreme @xcite . there are also more recent multilingual contextual embeddings (e.g., xlm-r large; @xcite which is trained on significantly more data (2.5 terabytes). notably, @xcite uses the target dev sets jointly for checkpoint selection. as with