S amba L ingo: Teaching Large Language Models New Languages (2024)

Abstract: Despite the widespread availability of LLMs, there remains a substantial gap in their capabilities and availability across diverse languages. One approach to address these issues has been to take an existing pre-trained LLM and continue to train it on new languages. While prior works have experimented with language adaptation, many questions around best practices and methodology have not been covered. In this paper, we present a comprehensive investigation into the adaptation of LLMs to new languages. Our study covers the key components in this process, including vocabulary extension, direct preference optimization and the data scarcity problem for human alignment in low resource languages. We scale these experiments across 9 languages and 2 parameter scales (7B and 70B). We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing language experts, outperforming all prior published baselines. Additionally, all evaluation code and checkpoints are made public to facilitate future research.

## introduction new state of the art large language models are being released at a breakneck speed, yet their training data, tokenizer, and evaluations remain primarily centered around a few popular languages such as english, chinese, french and arabic. in principle, the way to create large language models for specific languages is to pre-train models from scratch @xcite . however, it is difficult to obtain a large amount of compute resources and a vast quantity of data in diverse languages. researchers have tackled this problem by training monolithic multi-lingual models that cover a wide range of languages @xcite 1 fork of lm-evaluation-harness @xcite with new multilingual benchmarks: lm-evaluation-harness 2 all sambalingo checkpoints: sambalingo checkpoints @xcite @xcite . these models can still struggle to achieve uniformly good results across all languages due to various factors such as the curse of multilinguality @xcite and the scarcity of pre-training data in many languages @xcite . recently, adapting english centric models to new languages has gained prominence @xcite @xcite @xcite . the resulting models can outperform large multilingual models and even language specific models pre-trained from scratch. adaptation requires various design choices around the tokenizer, data, alignment and evaluation strategies. this paper aims to provide a comprehensive study to help inform these decisions, outlining a clear protocol to adapt a pre-trained model to a new language. we show that our methodology works by training models across 9 languages and 2 parameter scales (7b and 70b) and comparing them against publicly available models. figure 1 and 2 show that our methodology can lead to better models than existing state of the art models in these languages. the key studies and contributions include: perplexity (lower is better) 1.0 1.5 2.0 2.5 3.0 3.5 japanese thai hungarian arabic russian turkish bulgarian serbian slovenian llama 7b bloom-7b xglm-7.5b best open source expert sambalingo 7b sambalingo 70b 2023) , arabic: jais-13b @xcite , hungarian: puli-gptrio @xcite , russian: saiga-7b @xcite , bulgarian: mgpt-bulgarian @xcite . we could not find serbian, slovenian and turkish languages models with low enough perplexity that would fit the graph so we chose to omit them here to ensure readability. but sub word averaging accelerates training loss convergence (5.1.2) -the quality of the base checkpoint on english benchmarks can improve downstream language adaptation results (5.3) there has been a significant body of work around open-source multi-lingual models @xcite @xcite . our work differs from the aforementioned studies as we solely focus on adapting pre-trained llms to new languages and not on pre-training from scratch. notably, these multilingual open-source models tend to be pretrained on significantly fewer tokens than the base models we adapt from. as the models in this work tend to outperform these multilingual models, this presents a promising path forward for obtaining the state of the art in new languages. ## adaptation methodology we present our methodology to adapt large languages models to a new language, with state of the art results in 9 target languages: arabic, thai, turkish, japanese, hungarian, russian, bulgarian, serbian and slovenian. we select these languages because they provide a mix of high resource and lower resources languages with diverse character sets and linguistic patterns. we additionally limit the scope of the languages studied in this paper to languages with easily available text datasets from culturax @xcite . see section 4 for evaluation results on the final checkpoints produced by this methodology, and section 5 for ablations justifying our methods. we use the term initial language to describe the original language that the base model was trained on (in this case, english) and the term target language as the new language this model is being adapted to. ## ablations in this section, we present ablations of our design decisions in section 3. section 5.1 presents experiments motivating the modifications we make to the base model's tokenizer and how we initialize its new embeddings. section 5.2 ablates the amount of target language data and use of machine translated data in the dpo phase of our methodology. finally, section 5.3 looks at the impact of the quality of the base model. ## limitations our work has several limitations, including the need for extensive data from the target language, which is often unavailable for many languages. we study 9 diverse languages, but further research is required to address multilingual data scarcity and generalize our recipe. due to compute and time constraints, our ablation studies focus on around 3 languages each, assuming similar results for other languages, although linguistic diversity and data availability may affect this. additionally, we evaluate our chat-based model using gpt-4 as a judge, and while this has been shown to strongly correlate with human preferences in english, we are uncertain how well this works in other languages. we acknowledge that publicly releasing llms is risky because they can inadvertently generate harmful or biased content, compromise privacy, and be exploited for malicious purposes such as spreading misinformation. moreover, while our models are adapted to other languages and cultures, the english base model, data biases, and use of translation may prevent them from fully capturing the nuances of cultures and languages from around the world. ## conclusion we present a methodology to adapt pretrained llms to new languages. the methodology encompasses both continuous pretraining and alignment to human preferences in the target language. we present experimental results to justify our design choices and scale our methodology to 9 typologically diverse languages and 2 parameter scales. we make our evaluation scripts and final checkpoints publically available to facilitate future research, and we hope this work outlines a clearer path towards attaining state of the art language models in every language.