Adapting Open-Source Generative Large Language Models for Low-Resource Languages: A Case Study for T urkish (2024)

Abstract: Despite advancements in English-dominant generative large language models, further development is needed for low-resource languages to enhance global accessibility. The primary methods for representing these languages are monolingual and multilingual pretraining. Monolingual pretraining is expensive due to hardware requirements, and multilingual models often have uneven performance across languages. This study explores an alternative solution by adapting large language models, primarily trained on English, to low-resource languages. We assess various strategies, including continual training, instruction fine-tuning, task-specific fine-tuning, and vocabulary extension. The results show that continual training improves language comprehension, as reflected in perplexity scores, and task-specific tuning generally enhances performance of downstream tasks. However, extending the vocabulary shows no substantial benefits. Additionally, while larger models improve task performance with few-shot tuning, multilingual models perform worse than their monolingual counterparts when adapted.

## introduction the performance of proprietary generative large language models (llms) is better than open-source ones in most cases as this article is written @xcite , though there are efforts to develop open-source generative llms in terms of high performance and human ethics alignment @xcite @xcite . the progress is more significant in the english language compared to other languages as the aforementioned open-source models are mostly trained by english corpora @xcite . to make natural language processing technology more inclusive and accessible globally, research and development should be dedicated to the techniques that improve the performance of large language models in low-resource languages. monolingual @xcite @xcite @xcite and multilingual pretraining @xcite @xcite of generative llms are two main solutions for representing low-resource languages. however, monolingual pretraining is too costly due to hardware requirements for generative llms @xcite . on the other hand, multilingual llms have uneven performance across different languages mostly due to imbalanced training corpus @xcite . our proposed solution is to adapt open-source generative llms for low-resource languages, illustrated in figure 1 . in this regard, this study examines how to adapt open-source llms for low-resource languages in a systematic way. we focus on the benefits of using different methodologies, both individually and together, including continual training, supervised fine-tuning, and vocabulary extension, to adapt generative llms for low-resource languages. for the sake of efficiency, we use llama @xcite in the experiments. we select the turkish language as a low-resource language. we therefore refer to the model family used in this study as llamaturk. the model size and language selection are affordable when the number of experiments is considered in this study foot_0 . also, llama is trained mostly with english data, which can provide better investigation for adapting non-english languages. the turkish language can be categorized under low-resource languages when training corpus of open-source generative llms are considered @xcite ), yet the recipes given in this study can also be used for other low-resource languages since the methods are independent of language itself. we further examine adaptation in terms of two more aspects: model size and multilinguality. model size is important for scalability and performance @xcite . we provide an analysis of the adaptation of llama-7b and 13b in this respect. moreover, multilingual llms, such as bloom @xcite , yi @xcite , aya @xcite , and mala @xcite , can provide an opportunity to adapt low-resource languages easier than englishdominant ones due to multilingual corpus and vocabulary. since bloom and yi do not involve turkish in training and aya is larger than mala in terms of model parameters, we use mala for an analysis of multilingual llms. the main contributions of this study can be summarized as follows. we (i) analyze the adaptation of generative llms for low-resource language systematically to understand advantages and disadvantages in terms of continual training, instruction fine-tuning, task-specific fine-tuning, and vocabulary extension, (ii) investigate model size and multilingual models for adaptation, and (iii) publish all resources including source codes, datasets, and generative models reported in the experiments 2 . ## related work generative llms are either proprietary or opensource. although proprietary llms have currently outstanding performance @xcite , there are also efforts to develop competitive open-source models @xcite . the majority language of open-source generative llms is english. their pretraining text corpus mostly includes text in the english language. for adapting llms pretrained with english data for low-resource languages, the following methods are examined. (i) the training phase is continued using non-english raw data to learn the language properties of the new language @xcite @xcite @xcite . (ii) the knowledge of large language model is transferred by supervised fine-tuning on a non-english instruction or downstream-task dataset @xcite @xcite @xcite @xcite . (iii) the vocabulary of large language model is extended to include non-english tokens @xcite . these methods are employed in different studies and languages, resulting in a lack of understanding advantages and disadvantages of each in a controlled experimental framework. different from these studies, we provide a comprehensive experimental setup on the benefits of different methodologies for adapting generative llms for low-resource languages. moreover, we focus on model size and multilingual models for adaptation. ## adaptation methods in this section, we explain the methods to adapt open-source generative llms for low-resource languages in detail. ## experiments in this section, we evaluate the performance of different methods to adapt generative large language models for low-resource language. we particularly conduct both intrinsic and extrinsic evaluations in order to understand the performance of both language comprehension and downstream tasks. we also run benchmark llm evaluation by using appropriate datasets. this section further involves the results of using varying model sizes and applying multilingual models for the adaptation. ## conclusion this study examines different methods for adapting english-dominant generative large language models to low-resource languages. the results show that continual training with raw text can improve perplexity. vocabulary extension has no significant impact on adaptation performance under a small scale of continual training set. another factor can be 8-bit quantization with lora that trains a subset of model weights for new vocabulary token embeddings @xcite . we also find that the adaptation with generalpurpose instruction tuning has promising results in both perplexity and accuracy scores, while downstream task performance can be boosted by taskspecific tuning. furthermore, adapting a larger model with 13b parameters improves task performance with few-shot tuning. however, we observe no significant improvements by adapting a multilingual model. in future work, we plan to adapt other opensource language models such as llama2 @xcite and gemini @xcite to generalize our results to different models. other adaptation methods can also be studied such as modification of model architecture since different model layers and tokenization algorithms might change the outcomes.