Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies (2023)

Abstract: When we transfer a pretrained language model to a new language, there are many axes of variation that change at once. To disentangle the impact of different factors like syntactic similarity and vocabulary similarity, we propose a set of controlled transfer studies : we systematically transform the language of the GLUE benchmark, altering one axis of crosslingual variation at a time, and then measure the resulting drops in a pretrained model’s downstream performance. We find that models can largely recover from syntactic-style shifts, but cannot recover from vocabulary misalignment and embedding matrix re-initialization, even with continued pretraining on 15 million tokens. Moreover, good-quality tokenizers in the transfer language do not make vocabulary alignment easier. Our experiments provide insights into the factors of cross-lingual transfer that researchers should most focus on when designing language transfer scenarios.

## introduction what makes it hard for neural networks to learn new languages? large language models (llms) require vast datasets for pretraining, making it challenging to train llms from scratch for lowresource languages @xcite @xcite . for such languages, an appealing approach is to transfer knowledge from an llm trained for a highresource language, especially since pretrained models can transfer knowledge across even extreme shifts @xcite . a range of methods have been explored to enable such crosslingual transfer of english llms, using techniques such as adaptive pretraining @xcite , and embedding retraining @xcite . to better understand the factors affecting in chinese, "oolong" can refer to an unexpected change or development. * equal contribution. † corresponding author. our controlled studies consist of transferring an english model to a language that is transformed from english on just one axis of variation. realistic transfer scenarios involve languages that differ across multiple axes of variation at one time. our experiments serve to disentangle these effects, and identify the issues that practitioners should most focus on when doing cross-lingual transfer learning. we examine three factors that are salient in a transfer learning context: we test the effect of these factors on transfer learning both by 1) directly fine-tuning on t-english versions of the glue benchmark, as well as 2) continuing masked language model pre-training on 15 million tokens of t-english wikitext. in all cases, we find that word identity alignment provides the greatest stumbling block for transfer learning. reinitializing or shuffling the rows of the embedding matrix has a very negative effect on downstream learning which we cannot reverse in the low-data regime that we are simulating. if the embedding matrix is reinitialized and a new tokenizer is used, the effect of reinitialization overshadows any effect that the quality of the new tokenizer might cause. in the case of syntactic word-order transformations, we find that even in the low-data transfer learning regime, the models we test can adapt to word order shifts as long as vocabulary information is kept. we run experiments on roberta, deberta, and xlm-r in order to test transfer learning beyond the training set languages for both monolingual and multilingual models. our method allows us to disentangle the effects of correlated factors by inspecting them one at a time. foot_0 2 related work as self-supervised pretraining advances the state of nlp in high-resource languages, research into widening these successes beyond high-resource languages has become widespread and important. methodologies for best transferring a monolingual or multilingual model to an unseen language are widely explored. @xcite and @xcite , showcase the positive effects of pretraining on closer and related languages to the target language, even if this is less data than larger pretrained models, in part because of the possibility of shared vocabulary @xcite . our experiments build off previous efforts that try to enable crosslingual transfer from pretrained monolingual llms to new languages @xcite @xcite @xcite . with respect to vocabulary sharing and adaptation, @xcite show that training a multilingual model with a massive vocabulary that separates out languages outweighs the benefits of vocabulary sharing between language @xcite , while in the transfer regime @xcite showcase the importance of maintaining vocabulary overlap. techniques mapping subword embeddings to their new synonyms, or keeping subwords in the same script across languages, prove effective for cross-lingual transfer @xcite @xcite . the importance of embedding intialization statistics is discussed in @xcite . results on the importance of syntactic shifts remain broad, with work on multilingual training suggesting that syntactic shifts are significant compared to vocabulary effects @xcite , and that syntactic structure plays a role in developing parallel multilingual encodings (dufter and schütze, 2020), while @xcite show intersecting effects of vocabulary and word order shifts. understanding the direct relationship between the effect of syntactic shifts and the effect of vocabulary and tokenizer shifts remains an important problem in understanding transfer learning. our work creates a framework for decomposing and disentangling the difficulties of transfer in controlled studies, giving researchers pointers for what aspects of language variation make transfer difficult. ## methods our methodology consists of taking a pretrained model, and transferring to a t-english: a systematically transformed version of english data that differs from english on one axis of variation. the different t-englishes that we use are described and motivated below, and examples are in ## results we present the main results of our transfer experiments. our experimental details (e.g. hyperparameter choices) with a per-task breakdown of tglue performance as well as additional results on deberta and xlm-r are included in appendix a. ## conclusions in this paper, we propose a paradigm to study crosslingual transfer through transformations which simulate and disentangle the linguistic changes across languages. our results suggest that solving the embedding alignment problem is the "high-order bit" for crosslingual transfer: it has the largest impact on finetuning performance and is the least improved by continued pretraining. thus, future progress on solving this problem in large-scale transformers may have outsized impact.