From E nglish-Centric to Effective Bilingual: LLM s with Custom Tokenizers for Underrepresented Languages (2025)

Abstract: In this paper, we propose a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language. The method includes vocabulary expansion, initialization of new embeddings, model training and evaluation. We performed our experiments with three languages, each using a non-Latin script—Ukrainian, Arabic, and Georgian.Our approach demonstrates improved language performance while reducing computational costs. It mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar. Additionally, we introduce new metrics to evaluate language quality, revealing that vocabulary size significantly impacts the quality of generated text.

## introduction the discovery of the transformer architecture @xcite has opened doors for creating large language models (llms) with billions of parameters, trained on datasets of trillions of tokens. one of the notable features of the llms is cross-lingual language understanding (xlu), which allows models to possess multilingual capabilities. however, the xlu ability is restricted by the so-called curse of multilinguality, which refers the difficulties and constraints encountered in creating multilingual llms. studies showed that a substantial drop in performance occurs as the number of languages increases, due to the model's limited capacity to adequately capture and represent the nuances of each language @xcite . the efforts to examine and address the problem have highlighted two key factors: the composition of the dataset and vocabulary composition @xcite . some studies @xcite suggest that the natural limitations on the model capacity, vocabulary and training dataset sizes along with differences in language structures do not allow the creation of the ultimate multilingual model to perform equally in many languages, favoring the creation of custom models targeted at specific languages instead. the most obvious yet often overlooked consequence of low language representation in a model's vocabulary is a much higher cost of language processing. a sentence in ukrainian requires about 3 times more tokens for the gpt-4 model @xcite than the same sentence in english due to higher tokenization fertility (see section 6.1). three times higher fertility means three times smaller context window, three times higher memory usage, and nine times higher computation cost due to attention's quadratic dependence on the sequence length. on the other hand, high computational costs are not the only ramifications of a poor vocabulary. recent studies @xcite indicate that representation in an llm vocabulary of a specific language directly relates to the performance of the model in that language @xcite . in particular, it may be a reason for the generation of non-existing words, codeswitching @xcite , and broken grammar. languages that use a non-latin alphabet are particularly affected by poor vocabulary representation since they cannot rely even on the overlapping tokens with better represented languages. an insufficient training dataset affects the performance of llms as much as it does any other deep learning model. the model might generate a response in the wrong language, probably the one it is most familiar with, such as english @xcite . in this work, exposing the model to additional data in the target language via continual pre-training helped mitigate these effects. in this paper, we present a model-agnostic resource-effective method to create a base bilingual llm that supports english and another language. by addressing the above-mentioned issues of dataset and vocabulary composition, we make sure to improve its language capabilities along with boosting its computational efficiency. we illustrate our method in three languages with non-latin alphabets: ukrainian, georgian, and arabic. the contributions of our work are as follows: ## related work the shortcomings of existing multilingual llms have motivated numerous scholars and practitioners to address the insufficient performance of underrepresented languages. perhaps the most fundamental approach is to design and train a model from scratch, as demonstrated by @xcite . while this method offers maximal flexibility, it is highly demanding in terms of effort and computational resources. more commonly, available open-source llms are used as a starting point, leveraging transfer learning and building on available weights @xcite . this can still involve significant architectural changes compared to other methods, as seen in the solar model @xcite . despite utilizing transfer learning, such approaches often require pre-training on vast datasets, sometimes reaching trillions of tokens. a number of publications @xcite ; "hemanth kumar"; @xcite suggest a more lightweight approach, where the model's vocabulary is extended by 10,000-20,000 tokens, entailing the extension of the embedding layer and the language modeling head, while leaving the rest of the architecture unchanged. this method reduces the required training dataset to hundreds, or even tens, of billions of tokens, while still delivering notable improvements in the model's language abilities and computational efficiency. finally, instruction fine-tuning @xcite @xcite offers a highly resource-efficient alternative by skipping the base model composition step. while this approach can yield some improvements, it does not enhance the model's factual knowledge or address tokenization issues. our approach, in contrast, maintains the overall vocabulary size and keeps the model architecture intact. to create a bilingual model, we extend the vocabulary of the target language at the expense of other languages in the model, except english. this allows us to reduce the pre-training dataset to as little as 2 billion tokens while still improving the model's factual knowledge, enhancing the dataset, and achieving visible improvements in target language generation. ## methodology our proposed pipeline for training of bilingual llms supporting english and a target language l consists of the following steps: each step will be explained in more detail in the following subsections. ## datasets vocabulary extension datasets the monolingual language-specific tokenization models t l have been trained on monolingual datasets. for the ukrainian language we've trained on the publicly available ubertext 2.0 @xcite , that contains 3.274b words and consists of 8.59m texts. to train an arabic tokenizer we have used a private dataset of non-fiction books of 430 million words based on (acrps). for arabic, we integrated one more additional preprocessing step. as an arabic word could correspond to several words in another language transmitting the same meaning, it is the best practice to perform light stemming to allow the models to pick the similarity of the semantics of the main parts of words @xcite . for example, we consider (english translation: the) as a separate token when it prefixes a word. we processed attached pronouns and gender specifiers in similar way. for our experiments with georgian we have used the georgian section of the public oscar dataset (oscar), which contains 171.9m words. this dataset has been used for both tokenizer training and continual pre-training of the english-georgian mistral model for token adoption experiments. continual pre-training datasets for continual pre-training we created parallel datasets, consisting of both english and target language. for ukrainian and arabic, we considered wikipedia parallel dataset dump from june 20th 2024 archive dump 1 . for ukrainian, the size of the datasets is approximately 2b tokens. the total number of articles was 2.1m (791,336 in ukrainian and 1,327,709 in english). the total number of ukrainian tokens was 1.02b and the total number of english tokens was 1.05b. for arabic, the size of the datasets is approximately 1.8b tokens. the total number of articles was 2.1b (1.2b in arabic and 882,534 in english). the total number of arabic tokens was 621.51m and the total number of english tokens was 1.1b. for georgian token adoption experiments, we trained a model on parallel corpora from the same dump. the dataset was much smaller due to a sparsity of resources in georgian. it contained 107,123 and 169,602 articles in english and georgian, respectively. the total number of tokens was approximately 395.2m (219.88m in english and 175.32m in georgian). the articles were shuffled to create the training dataset with equal representation of the target language (arabic or ukrainian) and english. to determine the amount of tokens, we used the gemma 1 https://huggingface.co/collections/polyagent/ parallel-datasets-6707e4197a737319934d2a48 2 tokenizer. to evaluate the results, we used flores-200 (team, 2022) dataset for corresponding languages. the dataset is a collection of parallel translation corpora for 200 distinct languages, including ukrainian and arabic. we selected 500 text samples per language from the "devtest" split of the dataset in arabic and ukrainian. each text was separated into tokens by space, and only initial 3 tokens were kept as a model input. finally, these inputs were provided to the model to generate a completion with a maximum generated sequence length of 128. for ukrainian inputs, we obtained 1,500 tokens and 1,098 unique tokens. for arabic inputs, we obtained 1,500 tokens and 1,000 unique tokens. ## experimental setup we continually pre-trained bilingual models on the next token prediction task on the parallel corpora utilizing huggingface @xcite instructions for 8x80gb gpus. to launch training, we used the skypilot framework @xcite . in order to isolate the effects of extended vocabulary and additional pre-training we have conducted the same pre-training for the vanilla models and then compared the performances. for hyper-parameter optimization we used grid search. the selected set of hyper-parameters can be found in our github repository. ## evaluation metrics since we work on the base completion model, we focused mainly on the metrics that reflect the text completion performance: tokenizer fertility, code switching score, non-existing words ratio, and manually evaluated grammar correctness score. ## discussion the obtained results highlight a subject that has been largely overlooked, particularly in the context of generative llms: the impact of vocabulary size and composition an on the quality of generated text. our experiments with the vanilla model pretraining demonstrated that the effects of training on additional data can be mitigated via the vocabulary extension. additional pre-training on the target language corpus can noticeably increase text quality, particularly in addressing issues like codeswitching and the generation of non-existent words. however, handling more complex linguistic features, such as grammar, requires vocabulary extension. ukrainian and arabic tokens are represented differently in the original model's vocabulary, resulting in distinct yet complementary outcomes for the two languages. while for ukrainian a substantial 29.6% improvement was obtained with the extended vocabulary, the severely underrepresented arabic achieves a much higher 90.5% improvement. this effect was confirmed with another round of training at a lower learning rate for the english-ukrainian models, which showed a 35% improvement utilizing the model vocabulary extension. we propose the following explanation for this phenomenon: a poor vocabulary results in tokens that contain only one or a few characters, conveying very little specific semantic meaning. as a result, the model is forced to rely heavily on context during training and inference. this increases the noisiness of the data and prevents the model from learning nuanced meanings or effectively constructing complex grammatical structures. unfortunately, a static and limited vocabulary with fixed token-to-embedding mappings is a limitation of the standard transformer architecture. this makes it challenging to create a transformerbased llm that is equally proficient in multiple distinct languages. some methods that utilized char-based (canine @xcite ), patch-based (megabyte @xcite ), or bytebased @xcite transformers were suggested. they often suffer from longer sequences, reduced linguistic abstraction, and increased computational cost, which can hinder downstream performance compared to efficient bpe-based tokenization. for this reason, we advocate training bilingual models, which are both cost-effective and proficient in their target languages. ## conclusions in this work, we introduced a model-agnostic, costeffective method for developing bilingual base completion llms that support english and a target language, including low-resource or underrepresented languages. our approach, centered on vocabulary extension and efficient embedding initialization, was validated by creating two bilingual llms: english-ukrainian and english-arabic. moreover, we conducted experiments with georgian tokenization and explored token adoption process during the training of a english-georgian model. georgian has a unique underreprsentation in the mistral tokenizer. we demonstrated that extending the vocabulary of a pre-trained model enhances its performance in target language while maintaining its english performance. specifically, the grammar correctness results indicate that pre-training alone provides only limited improvement. the comparison between ukrainian and arabic further emphasizes the limitations of poor vocabulary for the underrepresented language. expanding the tokenizer's vocabulary with target language tokens reduced tokenizer fertility, resulting in lower computational costs and improved processing efficiency. finally, retaining the original english tokens in the custom tokenizer while adding new language-specific tokens lead to preservation of the model's english performance on the mmlu benchmark, while also improving its performance in the target language from perspective of grammar, code switching and non-existant word ratios. our approach promotes a more equitable and inclusive nlp ecosystem, contributing to the revitalization of underrepresented languages. by lowering the barrier to developing more literate and grammatically capable models, we believe our work also paves the way for enhanced economic viability of using llms in non-english languages.