An Empirical Comparison of Vocabulary Expansion and Initialization Approaches For Language Models (2024)

Abstract: Language Models (LMs) excel in natural language processing tasks for English but show reduced performance in most other languages. This problem is commonly tackled by continually pre-training and fine-tuning these models for said languages. A significant issue in this process is the limited vocabulary coverage in the original model’s tokenizer, leading to inadequate representation of new languages and necessitating an expansion of the tokenizer. The initialization of the embeddings corresponding to new vocabulary items presents a further challenge. Current strategies require cross-lingual embeddings and lack a solid theoretical foundation as well as comparisons with strong baselines. In this paper, we first establish theoretically that initializing within the convex hull of existing embeddings is a good initialization, followed by a novel but simple approach, Constrained Word2Vec (CW2V) , which does not require cross-lingual embeddings. Our study evaluates different initialization methods for expanding RoBERTa and LLaMA 2 across four languages and five tasks. The results show that CW2V performs equally well or even better than more advanced techniques. Additionally, simpler approaches like multivariate initialization perform on par with these advanced methods indicating that efficient large-scale multilingual continued pretraining can be achieved even with simpler initialization methods.

## introduction language models are adept at a wide spectrum of natural language processing (nlp) tasks @xcite @xcite @xcite . however, the best-performing language models work well for english but have inferior capabilities in other languages. a common method to improve the capabilities of other languages is to continually pre-train and finetune the english model for other languages @xcite . this approach builds upon the capabilities acquired through large-scale english pre-training and focuses on aligning the english and other language spaces, making efficient re-use of compute and data resources @xcite . one of the major challenges for llm adaptation is the lack of vocabulary coverage in the original model's tokenizer for the new language. this would mean the inability to represent the new language if the vocabulary is totally different or inefficient tokenization with high fertility in the case of inadequate vocabulary representation. a solution is to expand the tokenizer to incorporate new vocabulary and then perform continual pre-training on monolingual data from the new language to adapt the model to the new language @xcite @xcite . in this scenario, an important question is: how do we initialize the embeddings of the new vocabulary items? various methods have been proposed in the literature for the initialization of the new token embeddings, from simple random initialization @xcite to the mean of embeddings @xcite to sophisticated methods such as ofa among others @xcite @xcite that learn the new embeddings as a function of existing embeddings using external resources and tools like cross-lingual word-vectors and bilingual dictionaries. however, there is no theoretical basis for what constitutes a good initialization. furthermore, in existing works, comparisons with simple, naive initialization methods across different model sizes are missing. in this paper, we theoretically define and analyze the properties of a good initialization. we prove that initializing embeddings of new vocabulary embeddings to be in the convex hull of original embeddings ensures that the greedy generation of the existing language(s) is not impacted by the new vocabulary items on initialization. based on these insights, we propose a simple learnable initialization approach which we dub as constrained word2vec (cw2v) which ensures initializations in the convex hull without needing cross-lingual embeddings. we conducted a comparative analysis of cw2v alongside 5 existing initialization strategies including ofa on two models containing varying parameters, namely roberta (125m) and llama2 (7b), examining their impact through 5 downstream tasks across 4 languages. our analysis of various initialization methods demonstrates that cw2v achieves better if not comparable performance with the previous best methods. additionally, we find that simpler methods like multivariate or mean initialization, which ensure new embeddings remain within the convex hull, are comparable with more advanced approaches such as ofa. ## related work multilingual models: to create a multilingual model for specific languages, one method is to train the model from scratch on the target languages using mlm and clm objectives @xcite . however, this requires significant computational resources and data. a more efficient approach is to adapt an existing pretrained language model (plm) @xcite @xcite to the desired target language. there are two ways to adapt a plm to a new language. the first is to fully adapt the model to the new language, replacing the source tokenizer and focusing only on the new language's performance @xcite . the second is to keep the original language support and add the new language, ensuring the model still performs well on the source language @xcite . in this work, we focus on extending the language support of the plm rather than replacing it. we do this by extending the source tokenizer, which requires effectively initializing the model's embedding layer and lm head for the added tokens in the vocabulary. gies. methods like fvt @xcite and @xcite use the mean of source plm embeddings, while wechsel @xcite , ramen @xcite , focus (dobler and de melo, 2023), and ofa @xcite utilize external cross-lingual word vectors and source embeddings. however, these approaches rely on static embeddings. in contrast, we propose initialization strategy that learns new embeddings from the source plm model and doesn't require static embeddings. continual pre-training: a good initialization strategy provides a solid start for adapting a plm to a new language by effectively initializing the new tokens in the embedding and lm head layers. however, to fully adapt the extended model to the new language, continued pre-training (cpt) @xcite @xcite is essential. therefore, we performed cpt on target languages post initialization. ## methodology we describe the core methodology in this work followed by theoretical proofs of good initializations which motivate our own initialization approach, namely, constrained word2vec. ## experimental setting we now describe the models we focus on, the languages, downstream tasks and datasets, and implementation details. ## conclusion in this work, we establish that effective embedding initialization for an expanded vocabulary in language models can be achieved within the convex hull of source vocabulary embeddings. we introduce a data-driven initialization method, constrained word2vec (cw2v), which learns the target embeddings by constraining them in the convex hull of the source embeddings. our comparison of various initialization methods reveals that constrained word2vec performs on par with other advanced techniques. additionally, we find that simple methods like multivariate and mean, which ensure new embeddings lie within the convex hull of source embeddings, perform comparably well to more complex approaches. this indicates that efficient large-scale multilingual continued pretraining can be possible even with simpler methods, provided they are good initialization strategies.