Exploring Design Choices for Building Language-Specific LLM s (2024)

Abstract: Despite rapid progress in large language models (LLMs), their performance on a vast majority of languages remains unsatisfactory. In this paper, we study building language-specific LLMs by adapting monolingual and multilingual LLMs. We conduct systematic experiments on how design choices (base model selection, vocabulary extension, and continued pretraining) impact the adapted LLM, both in terms of efficiency (how many tokens are needed to encode the same amount of information) and end task performance. We find that (1) the initial performance of LLM does not always correlate with the final performance after the adaptation. Adapting an English-centric models can yield better results than adapting multilingual models despite their worse initial performance on low-resource languages. (2) Efficiency can easily improved with simple vocabulary extension and continued pretraining in most LLMs we study, and (3) The optimal adaptation method (choice of the base model, new vocabulary size, training data, initialization strategy) is highly language-dependent, and the simplest embedding initialization works well across various experimental settings. Together, our work lays foundations on efficiently building language-specific LLMs by adapting existing LLMs.

## introduction the predominance of english data on the internet, combined with a financial interest in englishcentric applications, has led to the development of primarily monolingual llms @xcite @xcite which exhibit significantly higher proficiency in english compared to other languages. even when llms support other languages, their performance lags behind -both in terms of end task performance * authors contributed equally and efficiency, measured by the amount of tokens required to encode information @xcite . prior work has mainly focused on building multilingual models that cover a broad spectrum of languages @xcite @xcite , or building language-specific llms from scratch @xcite @xcite . training a new language-specific llm from scratch can be expensive, both in terms of compute and data requirements. therefore, recent efforts have focused on adapting existing, high-performing llms @xcite , which consist of a two-stage process: (1) adapting a model's tokenizer with tokens from the target language @xcite to improve efficiency, and (2) updating model parameters through continued pre-training (cpt) to improve end task performance. for this straightforward adaptation recipe (illustarted in figure 1 ), many design choices can affect the final performance, providing trade-offs between efficiency and downstream performance. we focus on three major design choices -the choice of base llm, the size of augmented vocabulary, and the amount of continued pretraining data. we empirically evaluate how these design choices impact the final task performances on four diverse languages (hindi, turkish, arabic and tamil) on multilingual benchmarks containing up to 7 tasks. our goal is to provide guidance on how to build language-specific lms based on our experimental results spanning seven base lms. 1 we summarize our main findings here: together, we lay foundation for studying language-specific adaptation of existing llms, enabling llm access to a wider population. cross-lingual transfer recent efforts explore cross-lingual transfer of pre-trained english llms to new languages. @xcite transliterate indic languages to the latin script to transfer linguistic capabilities. recent work @xcite showed that training on millions of target language tokens without vocabulary extension can match the end task performance of state-of-the art model trained on billions of tokens. however, this comes at the cost of inference efficiency. most similar to ours, a line of work @xcite @xcite perform vocabulary extension, showing it can improve generation efficiency on the target language. however, to the best of our knowledge, there is no unified perspective on the design choices such as the base model, vocabulary size when working with limited compute and data resources, and our study is the first that explores these aspects on multiple languages. previous approaches have studied ways to optimally initialize embeddings for cross-lingual transfer. some methods have only been applied to encoder-only models @xcite , others typically use external lexicons @xcite , focus primarily on latin scripts (de vries and nissim, 2021), or require training a secondary model/embeddings separately @xcite . we compare few initialization approaches, showing that the simple mean embedding initialization works well. ## method: adapting llm to a target language we introduce a straightforward adaptation process -we will first generate language-specific tokens that will be added to the base vocabulary ( §3.1) of the model. then, we continue training lms with language modeling objective on the target language corpus such that they can make use of new tokens efficiently ( §3.2). ## results we report performances on two axis -efficiency (fertility) and end task performance. ## analysis can multilingual models benefit from the same adaptation recipe? figure 3 illustrates the change in performance (∆spbleu) on the flores generation task observed when adapting various monolingual and multilingual models for hindi (hi) and tamil (ta). here, we fix the adaptation recipe: we train on 100k examples, both with and without vocabulary augmentation (∼10k tokens). we compare four kinds of base models -large monolingual (mistral-7b, llama-2-7b), small monolingual (phi-2), large multilingual (xglm-7.5b, bloom-7.1b, gemma-7b), and small multilingual (gemma-2b). the lack of end task performance gains: phi-2 and gemma-7b larger monolingual models (llama and mistral) improve up to 12× over their base variants. here, the more capable english model (mistral) performs better after adaptation for both mid and low resource settings. the smaller, yet highly competitive, monolingual model i.e. phi-2 @xcite exhibits minimal improvement, particularly on tamil. phi-2 is the only model in our comparisons that was trained on a curated, high quality monolingual data. this suggests that adapting monolingual model trained on a highly curated data might be more challenging. smaller multilingual models like gemma-2b still exhibit up to 56% relative improvement, and for models with extended vocabulary, ∆@xmath0. training a smaller model with more data vs. a larger model with less data. as shown in figure 4 , this choice depends heavily on the target language, compute budget, and the type of base model. for shorter training periods (<30h), the smaller multilingual model (gemma-2b) is highly effective for both mid-resource (hindi) and lowresource (tamil) languages, making it a viable choice in scenarios where computational resources are limiting factors. while gemma-7b achieves the highest performance across all tasks and languages, it consumes more computational overhead with longer training times (>40h), llama-2-7b surpasses gemma-2b on generation tasks for hindi (figure 4 ), and across all tasks for tamil. a larger target vocabulary size of 50k seems to hinder performance in the stronger english model (mistral) as opposed to a smaller vocabulary size of 10k (figure 3 ). llama-2-7b offers a good balance between adaptation computational efficiency and performance. this makes llama-2-7b suitable for low-resource languages, particularly on generation tasks, when longer training periods are possible, but with a trade-off of increased inference time. size of augmented vocabulary can be scaled proportionally to cpt data we compare performance of our model variants with increasing target vocabulary size in ## conclusion in this paper, we presented a systematic study on the design choices involved in adapting llms to specific target languages. we explored the strategic choices in the adaptation process, such as the necessity of vocabulary extension, the addition of a large pool of language-specific tokens, and the criticality of initializing new parameters effectively. we contextualize these choices with respect to the end goal, such as the target language and a specified compute budget. through comprehensive experiments on up to four languages and seven base language models, we showed that the efficiency of the model on a target language can be improved by a simple vocabulary augmentation step followed by further training. we further show that under the same adaptation strategy, adapted smaller multilingual models can be as good as their larger counterparts, and larger monolingual lms can perform almost as good as multilingual lms with relatively minimal amounts of target language data.