E nglish Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too (2020)

Abstract: Intermediate-task training—fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task—often improves model performance substantially on language understanding tasks in monolingual English settings. We investigate whether English intermediate-task training is still helpful on non-English target tasks. Using nine intermediate language-understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on the XTREME benchmark. We see large improvements from intermediate training on the BUCC and Tatoeba sentence retrieval tasks and moderate improvements on question-answering target tasks. MNLI, SQuAD and HellaSwag achieve the best overall results as intermediate tasks, while multi-task intermediate offers small additional improvements. Using our best intermediate-task models for each target task, we obtain a 5.4 point improvement over XLM-R Large on the XTREME benchmark, setting the state of the art as of June 2020. We also investigate continuing multilingual MLM during intermediate-task training and using machine-translated intermediate-task data, but neither consistently outperforms simply performing English intermediate-task training.

## introduction zero-shot cross-lingual transfer involves training a model on task data in one set of languages (or language pairs, in the case of translation) and evaluating the model on the same task in unseen languages (or pairs). in the context of natural language understanding tasks, this is generally done using a pretrained multilingual language-encoding model ⇤ ⇤ equal contribution. intermediate-task training (stilts; @xcite consists of fine-tuning a pretrained model on a data-rich intermediate task, before fine-tuning a second time on the target task. despite its simplicity, this two-phase training setup has been shown to be helpful across a range of transformer models and target tasks @xcite , at least within english settings. in this work, we propose to use intermediate training on english tasks to improve zero-shot cross-lingual transfer performance. starting with a pretrained multilingual language encoder, we perform intermediate-task training on one or more english tasks, then fine-tune on the target task in english, and finally evaluate zero-shot on the same task in other languages. intermediate-task training on english data introduces a potential issue: we train the pretrained multilingual model extensively on only english data before evaluating it on non-english target task data, potentially causing the model to lose the knowledge of the other languages that was acquired during pretraining @xcite . to mitigate this issue, we experiment with mixing in multilingual mlm training updates during the intermediate-task training. in the same vein, we also conduct a case study where we machine-translate intermediate task data from english into three other languages (german, russian and swahili) to investigate whether intermediate training on these languages improves target task performance in the same languages. concretely, we use the pretrained xlm-r (conneau et al., 2020) encoder and perform experiments on 9 target tasks from the recently introduced xtreme benchmark @xcite , which aims to evaluate zero-shot cross-lingual transfer performance across diverse target tasks across up to 40 languages each. we investigate how training on 9 different intermediate tasks, including question answering, sentence tagging, sentence completion, paraphrase detection, and natural language inference impacts zero-shot cross-lingual transfer performance. we find the following: ## approach we follow a three-phase approach to training, illustrated in figure 1 : (i) we use a publicly available model pretrained on raw multilingual text using mlm; (ii) we perform intermediate-task training on one or more english intermediate tasks; and in phase (ii), our intermediate tasks have english input data. in section 2.4, we investigate an alternative where we machine-translate intermediate-task data to other languages, which we use for training. we experiment with both single-and multi-task training for intermediate-task training. we use target tasks from the recent xtreme benchmark for zero-shot cross-lingual transfer. ## related work sequential transfer learning using pretrained transformer-based encoders @xcite has been shown to be effective for many text classification tasks. this setup generally involves finetuning on a single task @xcite or multiple tasks @xcite @xcite , sometimes referred to as the intermediate task(s), before finetuning on the target task. we build upon this line of work, focusing on intermediate-task training for improving cross-lingual transfer. early work on cross-lingual transfer mostly relies on the availability of parallel data, where one can perform translation @xcite or project annotations from one language into another @xcite . for dependency parsing, @xcite use delexicalized parsers trained on source languages and labeled training data for parsing target-language data. @xcite proposes a parser selection method to select the single best parser for a target language. for large-scale cross-lingual transfer outside nlu, @xcite train a single multilingual neural machine translation system with up to 7 languages and perform zero-shot translation without explicit bridging between the source and target languages. @xcite expand this approach to cover over 100 languages in a single model. recent works on extending pretrained transformer-based encoders to multilingual settings show that these models are effective for cross-lingual tasks and competitive with strong monolingual models on the xnli benchmark @xcite @xcite . more recently, @xcite showed that cross-lingual transfer performance can be sensitive to translation artifacts arising from a multilingual datasets' creation procedure. finally, @xcite propose adapter modules that learn language and task representations for cross-lingual transfer, which allow adaptation to languages not seen during pretraining. ## conclusion we evaluate the impact of intermediate-task training on zero-shot cross-lingual transfer. we investigate 9 intermediate tasks and how intermediate-task training impacts the zero-shot cross-lingual transfer to the 9 target tasks in the xtreme benchmark. overall, intermediate-task training significantly improves the performance on bucc and tatoeba, the two sentence retrieval target tasks in the xtreme benchmark, across almost every intermediate-task configuration. our best models obtain 5.9 and 23.9 point gains on bucc and tatoeba, respectively, compared to the best available xlm-r baseline scores @xcite . we also observed gains in question-answering tasks, particularly using squad v1.1 and v2.0 as intermediate tasks, with absolute gains of 2.1 f1 for xquad, 0.8 f1 for mlqa, and 10.4 for f1 ty-diqa, again over the best available baseline scores. we improve over xlm-r by 5.4 points on average on the xtreme benchmark. additionally, we found multi-task training on all 9 intermedi-ate tasks to slightly outperform individual intermediate training. on the other hand, we found that neither incorporating multilingual mlm into the intermediate-task training phase nor translating intermediate-task data consistently led to improved transfer performance. while we have explored the extent to which english intermediate-task training can improve crosslingual transfer, a clear next avenue of investigation for future work is how the choice of intermediateand target-task languages influences transfer across different tasks. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791-4800, florence, italy.