Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for E stonian (2024)

Abstract: Adapting multilingual language models to specific languages can enhance both their efficiency and performance. In this study, we explore how modifying the vocabulary of a multilingual encoder model to better suit the Estonian language affects its downstream performance on the Named Entity Recognition (NER) task. The motivations for adjusting the vocabulary are twofold: practical benefits affecting the computational cost, such as reducing the input sequence length and the model size, and performance enhancements by tailoring the vocabulary to the particular language. We evaluate the effectiveness of two vocabulary adaptation approaches—retraining the tokenizer and pruning unused tokens—and assess their impact on the model’s performance, particularly after continual training. While retraining the tokenizer degraded the performance of the NER task, suggesting that longer embedding tuning might be needed, we observed no negative effects on pruning.

## introduction adapting multilingual pretrained language models to specific languages can enhance both their efficiency and performance @xcite . the adaptation generally involves continuously training the full model on language-specific data. this approach can be expensive and unsuitable for lessrepresented languages. in this study, we explore how modifying the vocabulary of a multilingual language model to better suit the estonian language affects its downstream performance. compared to the previous works @xcite @xcite , we focus on training newly initialized embeddings rather than the specific initialization approaches. the motivations for adjusting the vocabulary are twofold: our goal is to evaluate the effectiveness of two vocabulary adaptation approaches-retraining the tokenizer and pruning unused tokens-and assess their impact on encoder models' performance in the estonian language, particularly after continual training, which is evaluated by fine-tuning on the named entity recognition task. in the retraining approach, we train a new tokenizer on the estonian national corpus (enc), foot_0 and use the resulting vocabulary to replace/adapt the encoder-based multilingual deberta model @xcite . we first train the new embeddings with other model parameters frozen, and then continue training the full model with the masked language modeling loss. finally, we fine-tune the model with new/adapted vocabulary on the estonian ner dataset @xcite to evaluate the effect of vocabulary optimization. in the second approach, we simply prune the model's initial vocabulary to only keep the tokens that are present in the enc, and experiment with continuing the training of either only the embeddings or all model parameters. ## related work adapting a tokenizer to a new domain or language has been mainly done in two ways: modifying an existing tokenizer or training a new tokenizer on the domain data. the main focus of previous works has been on the embedding initialization methods for new or extended vocabulary, which is not needed in cases of vocabulary pruning. the vocabulary pruning has been previously explored by @xcite . the main motivation was that mbert @xcite , for instance, allocates more than 51% of its parameters to the embeddings layer, yet only a fraction of the vocabulary is used for a single given language. accordingly, the proposed approach is to create monolingual models from multilingual mbert by shrinking the vocabulary of the original model. to select the tokens to keep for a given monolingual model, the authors collected token frequency statistics from the wikipedia of the target language, and used these frequencies to filter out the tokens that appeared in less than 0.05% lines. as a result, the monolingual models retained up to 23.8% of the original vocabulary. tokenizer extension for bpe models is usually done by first training a new tokenizer and then adding non-overlapping tokens to the existing tokenizer's vocabulary @xcite . @xcite investigated extending an existing tokenizer and found that a correctly implemented vocabulary extension does not negatively affect downstream performance. @xcite also studied the vocabulary extension of llms, finding that a larger extension requires more pre-training data for optimal results. gee et al. ( 2022 ) introduced a method for fast vocabulary transfer (fvt) to train a domain-specific tokenizer. the embeddings for tokens shared between the new and original tokenizers were copied. the embeddings for new tokens were obtained by averaging their sub-token embeddings from the original tokenizer. the model was then further pre-trained on in-domain data using the masked language modeling (mlm) loss before fine-tuning on target tasks. dagan et al. (2024) applied the fvt @xcite for llms and found that the tokenizer choice impacts the effectiveness and down-stream performance of llms. specifically, they found that adapting the model to a new tokenizer requires tens of billions of tokens of retraining to outperform the original tokenizer. while our training enc training corpus is far smaller, containing only few billions of tokens, we are experimenting with encoder models that are much smaller than llms. ## methodology the overall methodology of optimizing a model's vocabulary to estonian entails two main steps: 1) modifying the content of the vocabulary and adjusting the embeddings accordingly, and 2) continuing the training of the whole model to align it better with the new vocabulary. as the base multilingual model we select mdeberta v3-the multilingual version of deberta v3 @xcite -the model which is considered the sota encoder model at the time of the writing. we compare two methods to modify the vocabulary of the mdeberta v3 multilingual language model for estonian. the first method involves training a new tokenizer on the estonian national corpus (enc), while in the second method we simply prune the model's original vocabulary to remove tokens that are not used in the tokenized enc. retraining the tokenizer we retained all the original settings (such as special tokens and preand post-processing steps) from the base mde-berta v3 model and retrained the underlying sen-tencepiece tokenizer. for new tokens introduced by the retrained tokenizer, we initialized their embeddings using the mean of the original embedding matrix, while for tokens present in both the original and new tokenizers, we copied the existing embeddings. we also adjust the token to id mapping and resize the embedding matrix. to align the newly initialized embedding vectors with the rest of the model, we first train the model on the training corpus with only the embeddings unfrozen using the masked language modeling (mlm). ## experimental setup training data for training the new tokenizer, and training and validation of the mlm objective, we employed the estonian national corpus (enc). 2 the corpus contains approximately 16m documents with documents coming from different domains such as old and contemporary literature, academic texts, wikipedia pages and discussions, as well crawled web pages. we performed light deduplication on the corpus resulting in ca 3.4b tokens and randomly split it into train, validation, and test, with both validation and test sets containing 1% of the documents. ## results in addition to the deberta baseline, we also compare with various other models, including both estonian-specific estbert (tanvir et al., 2020), xlm-roberta base @xcite , an estonian-specific estroberta finetuned from the xlm-roberta 3 and another estroberta model trained from scratch. 4 tokenizer efficiency we first present the impact on the models' size and tokenizer efficiency in ## discussion while replacing the vocabulary of the mdeberta model with a smaller estonian-specific vocabulary led to more efficient input tokenization, the results on the downstream ner task suffered even after both embedding layer training and subsequent full model training with lora. first, suboptimal embedding initialization approach likely played a role in the observed outcome. secondly, it is likely that a single epoch of embedding tuning was insufficient to match the performance of the base model. the subsequent lora mlm training resulted in slightly reducing the gap between the base model and the model with the retrained tokenizer, however it also remained insufficient to recover the original model's performance. we presume that training for longer, both the embeddings and the lora parameters, would further reduce that gap. in contrast, we observe that the vocabulary pruning has no observable negative effect on the downstream task. meanwhile, tuning of the embeddings appears to have little to no effect on the downstream task, which suggests that such tuning is redundant in case of pruning. surprisingly and similarly to embeddings tuning, continued training with lora had no observable benefit for the pruned model, despite the gains in the mlm accuracy. finally, we observed that the relation between the mlm accuracy and f1 on ner is not transparent. while we acknowledge that mlm accuracy scores with different vocabularies are not directly comparable, the absence of the effect on the ner result in the presence of a notable improvement in the mlm accuracy in the pruned model is puzzling. ## conclusion in this study, we explored two options for optimizing the vocabulary of a multi-lingual model for the estonian language. in summary, we found that replacing the tokenizer with a retrained languagespecific version noticeably degrades model performance on the downstream ner task, and one epoch of embedding layer training on a 3.4b word corpus did not suffice to restore it. while lora offers efficient way for further training continuation, a single epoch was insufficient to mitigate the negative impact of the tokenizer replacement. on the other hand, pruning unused tokens proved to be an effective method to reduce vocabulary size without compromising performance.