Bridging the Bosphorus: Advancing T urkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking (2024)

Abstract: Large Language Models (LLMs) are becoming crucial across various fields, emphasizing the urgency for high-quality models in underrepresented languages. This study explores the unique challenges faced by low-resource languages, such as data scarcity, model selection, evaluation, and computational limitations, with a special focus on Turkish. We conduct an in-depth analysis to evaluate the impact of training strategies, model choices, and data availability on the performance of LLMs designed for underrepresented languages. Our approach includes two methodologies: (i) adapting existing LLMs originally pretrained in English to understand Turkish, and (ii) developing a model from the ground up using Turkish pretraining data, both supplemented with supervised fine-tuning on a novel Turkish instruction-tuning dataset aimed at enhancing reasoning capabilities. The relative performance of these methods is evaluated through the creation of a new leaderboard for Turkish LLMs, featuring benchmarks that assess different reasoning and knowledge skills. Furthermore, we conducted experiments on data and model scaling, both during pretraining and fine-tuning, simultaneously emphasizing the capacity for knowledge transfer across languages and addressing the challenges of catastrophic forgetting encountered during fine-tuning on a different language. Our goal is to offer a detailed guide for advancing the LLM framework in low-resource linguistic contexts, thereby making natural language processing (NLP) benefits more globally accessible.

## introduction the remarkable advancements in large language models (llms) have revolutionized the field of natural language processing (nlp) @xcite @xcite @xcite . however, addressing models that diverge from an englishcentric framework poses considerable challenges, particularly in low-resource languages. while certain languages like turkish aren't categorized as under-resourced, there's a limited number of research groups focusing on them @xcite . consequently, these languages lag in advancing cutting-edge systems because of the absence of solid and open-source base llms together with standardized benchmarks to evaluate their capabilities. recognizing this gap, our work is motivated by aiming to leverage turkish llms. we meticulously demonstrate two distinct methodologies: we first tried to adapt two base llms, mistral-7b @xcite and gpt2-xl @xcite to turkish. secondly, we trained a family of decoder models entirely from scratch in varying sizes. to adhere the turkish llms to human instructions and extend their reasoning capabilities, we designed a novel turkish instruction-tuning (it) dataset, designed to enhance the reasoning abilities of turkish llms by following the self-instruct framework @xcite . one of the key challenges with turkish llms is evaluating their accuracy on different tasks in a reproducible and fair manner while ensuring dataset quality. many reasoning datasets have been directly machine-translated from english without any validation, leading to biased and inaccurate results. to address this, we introduce three turkish datasets: truthfulqa-tr, for assessing a model's tendency to reproduce common falsehoods, arc-tr, a set of grade-school science questions, and gsm8k-tr for evaluating the mathematical reasoning capabilities of the models. we carefully translated by using state-of-the-art tools and validated all samples with multiple annotators, cleaning them as needed. we detailed the translation and annotation processes. our contributions are as follows: by detailing the development of specialized datasets and methodologies, we offer a comprehensive guide for building llms for languages with limited resources. additionally, our contributions substantially enrich the field by providing critical resources that will support future research in turkish language processing and the broader area of natural language processing (nlp) for under-resourced languages. ## datasets the initial step in building a base llm involves pretraining it on a vast corpus of text with a nexttoken-prediction objective @xcite . this corpus comprises trillions of words gathered from the internet and is characterized by its large volume but often compromised in quality due to the noise in the raw internet data. following the pretraining, the model undergoes fine-tuning with high-quality prompt-response pairs which focuses on improving the model's reasoning capabilities @xcite . in the end, the goal is to achieve a supervised-finetuned (sft) model that is aligned with the desired response behavior or domain expertise. this section describes the corpora utilized in the pretraining phase (section 2.1) and the development process of the turkish it dataset (section 2.2). ## methodology creating an llm for under-resourced languages, like turkish, often poses challenges primarily due to the scarcity of publicly available data especially if you have limited computational resources. regarding these, we followed two different strategies to build a turkish series of llms: (i) further training state-of-the-art base models on turkish data, which was initially unfamiliar with turkish (i.e., not trained on turkish data), (ii) pretraining a model from scratch, following gpt2 scales on a vast amount of text data defined in section 2.1. ## evaluations 4.1 bits-per-character (bpc) evaluations auto-regressive language modeling is trained on optimizing the negative log-likelihood (nll) of the data in the training set and the effectiveness of the model is then calculated on the unseen test data. furthermore, the most common metric to evaluate these models is perplexity, which measures the uncertainty of an llm in predicting the next token in a sequence and is derived by taking the exponential average of the nll. however, as various tokenizers can divide each sentence into differing numbers of tokens, nll and ppl may produce incomparable results for models utilizing different tokenizers. to tackle this, we use bits-per-character (bpc), which is another critical metric derived from nll, used for evaluating the performance of llms at character-level. further details on the calculation of these metrics are given in the appendix in section f. consequently, our comparisons mainly relied on bpc, which normalizes the impact of tokenization differences. for the bpc evaluation, we utilized the test set of the trnews-64 corpus @xcite , comprising 5,000 samples. results. we present the bpc results of different models evaluated on trnews-64 in the last column of ## conclusion our work advances the development of turkish llms, presenting a new series of models both trained from scratch (hamza) and also adapted from other base llms (hamza m istral and hamza gp t 2-xl ), together with new instruction tuning dataset and a meticulously crafted turkish llm leaderboard. in our analysis, we noted that the base llms exhibited catastrophic forgetting of their primary language knowledge during continued pretraining. additionally, through the creation of a novel turkish llm evaluation benchmark, we have identified a significant performance gap between current turkish llms and their english counterparts, underscoring the need for further improvements in turkish language modeling. for more detailed discussions on limitations and future work, please refer to appendix b. our fully open-source work and detailed observations play a pivotal role in the field of turkish language modeling, providing insights on construction methodologies and offering a comparative framework for evaluating performance, thereby paving the way for future advancements.