FOCUS : Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (2023)

Abstract: Using model weights pretrained on a high-resource language as a warm start can reduce the need for data and compute to obtain high-quality language models for other, especially low-resource, languages. However, if we want to use a new tokenizer specialized for the target language, we cannot transfer the source model’s embedding matrix. In this paper, we propose FOCUS - **F**ast **O**verlapping Token **C**ombinations **U**sing **S**parsemax, a novel embedding initialization method that effectively initializes the embedding matrix for a new tokenizer based on information in the source model’s embedding matrix. FOCUS represents newly added tokens as combinations of tokens in the overlap of the source and target vocabularies. The overlapping tokens are selected based on semantic similarity in an auxiliary static token embedding space. We focus our study on using the multilingual XLM-R as a source model and empirically show that FOCUS outperforms random initialization and previous work on language modeling and on a range of downstream tasks (NLI, QA, and NER). We publish our model checkpoints and code on GitHub.

## introduction research on large language models is advancing rapidly with powerful new models being published at a break-neck pace (e.g., @xcite @xcite . although multilingual models have been released, many of the world's languages are not covered. multilingual models have also been shown to have subpar performance on under-resourced languages @xcite . therefore, it is crucial to develop methods that harness these advances and make them available for further languages, especially low-resource ones. a promising line of work in this regard focuses on crosslingual transfer of transformer models pre- target vocabulary (fasttext embedding) set weights according to distances pretrained vocabulary (pretrained model embedding) initialize as weighted mean from pretrained embeddings find similar tokens in target and pretrained vocabulary overlap trained on high-resource languages. crosslingual transfer directly copies the pretrained weights in the transformer layers to the target language model. subsequently, the model is further adapted to the target language by continued pretraining on unlabeled target language text using the original selfsupervised pretraining objective. this sort of training regimen is also known as language adaptive pretraining (lapt; @xcite . however, the pretrained model's embedding matrix cannot be directly transferred if we use a new tokenizer for the target language @xcite . using appropriate tokenizers has been shown to be important for the model's performance on downstream tasks @xcite and is crucial if the source and target language use different scripts. we present focus, an embedding initialization method that allows us to transfer information from the source model's pretrained embedding matrix to a new embedding matrix for the target language's tokenizer. focus is illustrated in figure 1 . the key idea is to use overlapping tokens between both tokenizers as anchor points and represent new target language tokens as a weighted mean of overlapping tokens' embeddings. this enables us to initialize the new embedding matrix in the same semantic space as the pretrained embedding ma-trix. we empirically show in extensive experiments across a range of different high-resource and lowresource target languages that focus outperforms various strong baselines both in language modeling as well as on downstream tasks (natural language inference, question answering, and named entity recognition). in our experiments, we focus on the multilingual xlm-r @xcite as a source model and specialize it for a single language. focus is particularly well-positioned to take advantage of multilingual source models due to their larger vocabulary and the fact that they have already been pretrained to a certain extent on many potential target languages. additionally, we show that focus still improves significantly over random initialization even if only minimal vocabulary overlap is available. 2 in previous work, a common approach has been to adapt multilingual models to target languages while simply keeping or extending the original vocabulary @xcite @xcite . when extending the vocabulary, focus can also be applied to initialize embeddings just for the new tokens. however, we advocate considering the setting of full vocabulary replacement. only a fraction of the multilingual vocabulary is actually used for any single language, so by fully replacing the large multilingual vocabulary with a language-specific smaller vocabulary we can enable faster training times and smaller models. xlm-r's vocabulary has 250k tokens, and replacing this with a languagespecific 50k token vocabulary reduces the model size quite dramatically, by over 55%. 3 in our experiments, training with a language-specific 50k token vocabulary is 40% faster than extending the original 250k token vocabulary. 4 we summarize the contributions of our paper as follows: ## focus our goal is to initialize embeddings for tokens in a new, language-specific target vocabulary in the same semantic space as the source model's embeddings. in this study, we mainly focus on the multilingual source model xlm-r although fo-cus can in principle also be applied to monolingual source models. 5 we copy all embeddings of shared tokens between source and target tokenizer for our new embedding matrix. if the target language was already part of the source model's pretraining corpus, this takes advantage of target language tokens with pretrained embeddings in the source model's tokenizer. in any case, we take advantage of shared named entities, symbols, numbers, punctuation, and shared words resulting from code-switching between the target and pretrained vocabularies. additional target language tokens not present in the source model are represented as a linear combination of embeddings of semantically similar shared tokens. unlike previous work on embedding initialization, this requires neither bilingual dictionaries nor an alignment of embedding spaces across different languages @xcite . next, we formally describe focus. ## experimental setup we perform experiments using xlm-r as our multilingual source model, due to its popularity and widespread use. 8 we use the base variant for all experiments. our language-specific tokenizers are trained in the same way as xlm-r for comparability, specifically sentencepiece tokenization @xcite with the unigram algorithm @xcite . we use hugging-face tokenizers and a vocabulary size of 50k tokens for all languages. ## results we present downstream task results for nli and qa in ## related work we now discuss further related work apart from the studies introduced in section 1. @xcite adapted xlm-r to up to 20 african languages at the same time instead of specializing on a single language. @xcite and @xcite used resources with much higher language coverage than web-scraped monolingual texts (the bible and lexicons, respectively) to adapt pretrained multilingual models to unseen languages. @xcite transliterated unseen languages into latin script to improve the results when using an existing pretrained vocabulary. ## conclusion we propose focus, a novel embedding initialization method for the monolingual specialization of language models with a language-specific tokenizer. focus uses the vocabulary overlap between source and target languages to effectively transfer the pretrained embeddings to the new target tokenizer's embedding matrix. in a series of experiments across a diverse set of languages and several different tasks, we show that focus outperforms other available embedding initialization methods without requiring additional resources like bilingual dictionaries. focus can provide a good initialization even if only a minimal vocabulary overlap is available and when the target language has not been part of the source model's pretraining. we release our code and model checkpoints on github. 11