WECHSEL : Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models (2022)

Abstract: Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method – called WECHSEL – to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.

## introduction large lms based on the transformer architecture @xcite have become increasingly popular since gpt @xcite and bert @xcite were introduced, prompting the creation of many large lms pretrained on english text @xcite @xcite . there is a tendency towards training larger and larger models @xcite while the main focus is on the english language. recent work has called attention to the costs associated with training increasingly large lms, including environmental and financial cost @xcite . if training large lms for english is already costly, it is prohibitively expensive to train new, similarly powerful models to cover other languages. one approach to address this issue is creating massively multilingual models @xcite @xcite trained on a concatenation of texts in many different languages. these models show strong natural language understanding capabilities in a wide variety of languages, but suffer from what @xcite call the curse of multilinguality: beyond a certain number of languages, overall performance decreases on monolingual as well as cross-lingual tasks. consistent with this finding, @xcite observe that monolingual lms often outperform massively multilingual models. this might be attributed to superior quality of monolingual tokenizers over their multilingual counterparts @xcite . it is thus desirable to train monolingual models in more languages. training monolingual models in non-english languages is commonly done by training a new model with randomly initialized parameters @xcite @xcite . however, to train a model with capabilities comparable to that of an english model in this way, presumably a similar amount of compute to what was used to train the english model would be required. to address this issue, we introduce wechsel, 1 a novel method to transfer monolingual language models to a new language. wechsel uses multilingual static word embeddings between the source language and the target language to initialize model parameters. wechsel first copies all inner (non-embedding) parameters of the english model, and exchanges the tokenizer with a tokenizer for the target language. next, in contrast to prior work doing random initialization @xcite , the token embeddings in the target language are initialized such that they are close to semantically similar english tokens by mapping multilingual static word embeddings to subword embeddings. the latter step is particularly important considering that token embeddings take up roughly 31% of the parameters of roberta @xcite and roughly 33% of the parameters of gpt2 @xcite . intuitively, semantically transferring embeddings instead of randomly initializing one third of the model should result in improved performance. our parameter transfer provides an effective initialization in the target language, requiring significantly fewer training steps to reach high performance than training from scratch. as multilingual static word embeddings are available for many languages @xcite , wech-sel is widely applicable. we conduct our experiments on roberta and gpt-2 as representative models of encoder and decoder language models, respectively. we transfer the english roberta model to four languages (french, german, chinese and swahili), and the english gpt-2 model to the same four plus another four very low-resource languages (sundanese, scottish gaelic, uyghur and malagasy). we evaluate the transferred roberta models on named entity recognition (ner), and natural language inference (nli) tasks in the respective languages. the transferred gpt-2 models are evaluated in terms of language modelling perplexity (ppl) on a held-out set. we compare wechsel with randomly initialized models (denoted as fullrand), as well as the recently proposed transinner method which only transfers the inner (non-embedding) parameters (de vries and nissim, 2021). all mentioned models are trained under the same conditions (around 4 days on a tpuv3-8). we also compare our model with models of comparable size trained from scratch under significantly larger training regimes, in particular camembert (martin et al., 2020) (french), gbert base @xcite (german), and bert base -chinese @xcite . results show that models initialized with wechsel outperform randomly initialized models and models initialized with transinner across all languages and all tasks, for both roberta and gpt-2. in addition, strong performance is reached at a fraction of the training steps of other methods. our contribution is summarized as follows. in the following, we review related work in section 2. we introduce the wechsel method in section 3, followed by explaining the experiment setup in section 4. we show and discuss results in section 5. ## related work large language models. training language models is usually done in a self-supervised manner i. e. deriving labels from the training text instead of needing explicit annotations. one optimization objective is masked language modelling @xcite , where randomly selected tokens in the input are replaced by a special [mask] token, and the task is to predict the original tokens. another common objective is causal language modelling (clm), where the task is to predict the next token. these two objectives highlight a fundamental distinction between language models: models can be trained as encoders (e.g. with mlm) or as decoders (e.g. with clm). instead of words, the vocabulary of recently proposed language models commonly consists of subwords @xcite @xcite . ## methodology to initialize the model in the target language, we copy the inner (non-embedding) parameters from the source model. our goal, then, is given the tokenizer t s in the source language with vocabulary u s , the corresponding token embeddings e s , and a tokenizer t t in the target language with vocabulary u t , to find a good initialization of the embeddings e t by using e s . to this end, we use existing bilingual word embeddings enriched with subword information, containing a set of words and subword n-grams in the source and target language and their aligned vectors. we denote the set of words and n-grams in the source and target language as v s and v t respectively, and the aligned static embeddings as w s and w t . in appendix d we consider an alternative method if no subword information is available in the bilingual word embeddings. first, independently for both languages, we compute static subword embeddings for tokens in the tokenizer vocabulary in the same semantic space as the static word embeddings (section 3.1). this results in subword embeddings u s and u t for the source and target language, respectively. next, we use u s and u t to compute the semantic similarity of every subword in u s to every subword in u t . using these semantic similarities, we initialize the embeddings in e t through a convex combination of embeddings in e s (section 3.2). by applying wechsel, the vectors of e t are in the same semantic space as e s , where a subword in the target language is semantically similar to its counterpart(s) in the source language. these steps are summarized in figure 1 and explained in more detail in the following. ## experiment design we evaluate our method by transferring the english roberta @xcite and the english gpt-2 model @xcite to french, german, chinese and swahili. we refer to these languages as medium-resource languages. in addition, we study the benefits of our method on four lowresource languages, namely sundanese, scottish gaelic, uyghur and malagasy. we evaluate wechsel-roberta by finetuning on xnli @xcite , and on the balanced train-dev-test split of wikiann @xcite to evaluate nli and ner performance, respectively. the hyperparameters used for fine-tuning are reported in appendix b. gpt-2 is evaluated by perplexity (ppl) on a heldout set from the same corpus on which the model was trained on. due to the difficulty of extrinsic evaluation on low-resource languages, we only train gpt-2 models in these languages, and evaluate their performance intrinsically via language modelling perplexity on a held-out set. we use the pretrained models roberta base with 125m parameters, and the small gpt-2 variant with 117m parameters provided by huggingface's transformers @xcite in all experiments. since under limited training regimes such as ours, using a smaller corpus does not in general degrade performance @xcite , we use a subset of 4gib from the oscar corpus for german, french and chinese. for the other languages, we use data from the cc-100 corpus @xcite which contains 1.6gib, 0.1gib, 0.1gib, 0.4gib and 0.2gib for swahili, sundanese, scottish gaelic, uyghur and malagasy, respectively. to obtain aligned word embeddings between the source and the target language we use monolingual fasttext word embeddings foot_1 @xcite . we align these embeddings using the orthogonal procrustes method @xcite with bilingual dictionaries from muse foot_2 @xcite for french, german and chinese and a bilingual dictionary from freedict foot_3 @xcite for swahili. for the low-resource languages, we use bilingual dictionaries scraped from wiktionary. foot_4 model tokens trained on factor wechsel-roberta 65.5b 1.0x transinner-roberta 65.5b 1.0x fullrand-roberta 65.5b 1.0x camembert 419.4b 6.4x gbertbase 255.6b 3.9x bertbase-chinese 131.1b 2.0x we choose temperature τ = 0.1 and neighbors @xmath0. we train tokenizers in the target languages using a vocabulary size of 50k tokens and byte-level bpe @xcite . after applying wechsel, we continue training roberta on the mlm objective and gpt-2 on the clm objective. we compare against two baseline methods. all models are trained for 250k steps with the same hyperparameters across all languages (reported in appendix b). training one model takes around 4 days on a tpuv3-8. for wechsel and fullrand we use a learning rate (lr) schedule with linear warmup from zero to the peak lr for the first 10% of steps, followed by a linear decay to zero. for transinner, we perform two warmup phases from zero to peak lr, once for the first 10% of steps for training embeddings only, then again for the remaining steps while training the entire model. in addition to the mentioned baselines trained under this setting, we compare the results of results are averaged over 3 runs. we report scores before training (score@0), after 10% of steps (score@25k) and after training (score@250k). we also report results from fine-tuning prior monolingual models and xlm-r (score (more training)), all trained on more tokens than our models. for each language, the best results in every column are indicated with underlines. the overall best results including the comparison with existing monolingual/multilingual models of comparable size are shown in bold. roberta models with existing comparable models trained from scratch with more training effort. we consider the total number of tokens the model has encountered in the target language, computed as the product of batch size × sequence length × train steps (shown in ## results we present our results on transferring roberta and gpt-2 from english to other languages, followed by analyzing training behavior. in appendix c, we provide a qualitative assessment of how well subword tokens are mapped between the source and the target languages. ## conclusion we introduce wechsel, an effective method to transfer monolingual language models to new languages. wechsel exploits multilingual static word embeddings to compute an effective initialization of subword embeddings in the target language. we conduct experiments by transferring roberta and gpt-2 models from english to french, german, chinese and swahili, as well as english gpt-2 to four low-resource languages. the evaluation results show that the transferred roberta and gpt-2 models are more efficient and effective than strong baselines, and consistently outperform prior monolingual models that have been trained for a significantly longer time. wechsel facilitates the creation of effective monolingual lms for new languages with medium to low resources, particularly in computationally-limited settings. in addition, our work provides strong evidence towards the hypothesis by @xcite that deep monolingual language models learn abstractions that generalize across languages.