VRCP : Vocabulary Replacement Continued Pretraining for Efficient Multilingual Language Models (2025)

Abstract: Building large language models (LLMs) for non-English languages involves leveraging extensively trained English models through continued pre-training on the target language corpora. This approach harnesses the rich semantic knowledge embedded in English models, allowing superior performance compared to training from scratch. However, tokenizers not optimized for the target language may make inefficiencies in training. We propose Vocabulary Replacement Continued Pretraining (VRCP), a method that optimizes the tokenizer for the target language by replacing unique (solely available) vocabulary from the source tokenizer while maintaining the overall vocabulary size. This approach preserves the semantic knowledge of the source model while enhancing token efficiency and performance for the target language. We evaluated VRCP using the Llama-2 model on Japanese and Chinese corpora. The results show that VRCP matches the performance of vocabulary expansion methods on benchmarks and achieves superior performance in summarization tasks. Additionally, VRCP provides an optimized tokenizer that balances token efficiency, task performance, and GPU memory footprint, making it particularly suitable for resource-constrained environments.

## introduction recent advancements in large language models (llms) based on transformer architectures @xcite have brought significant progress to the field of nlp. models such as @xcite and llama-2 @xcite have predominantly been trained on extensive english corpora, leaving a gap in the availability of models optimized for non-english languages. this disparity is due to the relative scarcity of highquality, large-scale corpora for many non-english languages compared to english. consequently, this limits the potential improvements based on the scal- ## experiments we evaluated the effectiveness of our proposed method using experiments with japanese and chinese corpora. we evaluated the methods based on three key aspects: ## related works several studies have explored enhancing model performance for low-resource languages by expanding vocabulary and embedding layers in continued pretraining. for example, @xcite and @xcite expanded the vocabulary and embedding layers for mbert @xcite , which improved performance by incorporating lowresource language corpora. these methods typically involve expanding the vocabulary size, which increases the number of model parameters and the gpu memory footprint. this is because expanding the vocabulary requires adding corresponding embedding vectors. in decoder-only models like llama-2 @xcite , embeddings must be placed both after the input and before the output. therefore, expanding the vocabulary by n new tokens with an embedding dimension of d results in an increase of 2×d×n new model parameters. in contrast, our proposed method maintains the source vocabulary size by focusing on vocabulary replacement. this preserves the semantic knowledge of the source model while optimizing for the target language. @xcite highlight the significant impact of tokenization on multilingual models, especially the importance of language-specific token coverage for word-level tasks. their study provides guidelines for selecting tokenizers before expensive model pre-training. our research aligns with these insights by addressing the optimization of tokenizers for multilingual models without expanding the vocabulary size. this is particularly beneficial in resourceconstrained environments. we provide an alternative strategy that leverages source model knowledge while adapting to the target language, complementing the guidelines suggested by @xcite . ## conclusion we introduced vocabulary replacement continued pretraining (vrcp), a method that optimizes tokenizers for non-english languages without increasing vocabulary size. vrcp replaces low-frequency english tokens with high-frequency target language tokens, leveraging the semantic knowledge of english models while enhancing token efficiency for the target language. our experiments with japanese and chinese corpora demonstrated that vrcp matches the performance of traditional vocabulary expansion methods and excels in tasks requiring text generation, such as summarization. this improved performance in generation tasks suggests that vrcp can be effectively applied to language models integrated into retrieval-augmented generation (rag) frameworks, where generating coherent and contextually accurate summaries is critical. additionally, vrcp avoids additional gpu memory costs by maintaining the original vocabulary size, making it suitable for resource-constrained environments.