Atlas-Chat: Adapting Large Language Models for Low-Resource M oroccan A rabic Dialect (2025)

Abstract: We introduce Atlas-Chat, the first-ever collection of LLMs specifically developed for dialectal Arabic. Focusing on Moroccan Arabic, also known as Darija, we construct our instruction dataset by consolidating existing Darija language resources, creating novel datasets both manually and synthetically, and translating English instructions with stringent quality control. Atlas-Chat-2B, 9B, and 27B models, fine-tuned on the dataset, exhibit superior ability in following Darija instructions and performing standard NLP tasks. Notably, our models outperform both state-of-the-art and Arabic-specialized LLMs like LLaMa, Jais, and AceGPT, e.g., our 9B model gains a 13% performance boost over a larger 13B model on DarijaMMLU, in our newly introduced evaluation suite for Darija covering both discriminative and generative tasks. Furthermore, we perform an experimental analysis of various fine-tuning strategies and base model choices to determine optimal configurations. All our resources are publicly accessible, and we believe our work offers comprehensive design methodologies of instruction-tuning for low-resource languages, which are often neglected in favor of data-rich languages by contemporary LLMs.

## introduction transformer-based large language models have revolutionized nlp research and beyond, demonstrating exceptional performance in both natural and formal language generation @xcite , and exhibiting advanced reasoning capabilities in arithmetic, symbolic, and logical tasks @xcite . despite their success and the frequent release of new, superior open models exemplified by llama @xcite and @xcite , these breakthroughs have † these authors contributed equally. † correspondence: guokan.shang@mbzuai.ac.ae 1 https://hf.co/mbzuai-paris/atlas-chat-9b been concentrated in a few data-rich languages @xcite , assuming access to hundreds of billions or even a dozen trillions of tokens for training, often neglecting underrepresented languages. in this work, we explore the challenges of introducing llms for low-resource dialectal arabic (da). the arabic language has a rich history and profound cultural significance, featuring an intricate script, extensive lexicon, and complex grammar, making it a unique linguistic entity. although interest in developing arabic-specialized models has recently been growing, notably led by models like jais @xcite , acegpt @xcite , and allam @xcite , these efforts primarily focus on bilingualism by balancing english and modern standard arabic (msa), while often neglecting or excluding da. however, msa differs significantly from da in terms of morphology, syntax, and other linguistic features. moreover, various arabic dialects also differ considerably from one another. in fact, arabic dialects collectively have more native speakers than msa, as da serves as the primary mode of communication in daily life across various arabic-speaking regions @xcite . this asymmetry is due in large part to the fact that da poses challenges not encountered with msa. some are related to the lack of essential components for model development-namely, training data, benchmarks, and suitable evaluation metrics-but others stem from the very nature of the linguistic characteristics involved in da itself more generally. we take moroccan arabic, also known as darija, as the focus of our work. despite being spoken by 40 million people 2 , darija remains low-resource. this is because msa is used in official domains in morocco, while darija, a blend of msa, amazigh, french, and spanish, is the vernacular widely spoken in daily life. although darija, previously only an oral language, has recently developed a written form through the proliferation of social networks and increased access to technology, it still lacks standardization and established grammatical or syntactic rules due to its recent emergence @xcite . moreover, darija can be represented in two forms: arabic script or latin script (also known as arabizi). for example, the darija translation of "how are you?" can be written as: "kidayr?" or " ". these challenges underscore the need for models tailored to this linguistic context. to the best of our knowledge, we are the first to introduce modern llms specifically developed for moroccan arabic, as well as for da in general. we first constructed the darija-sft-mixture foot_1 dataset, consisting of 458k instruction samples, by consolidating existing darija language resources, creating novel datasets both manually and synthetically, and translating english instructions under strict quality control. we then developed a comprehensive evaluation suite including benchmarks: darijammlu, darijahellaswag, darijaalpacaeval, and darijabench, to assess llm capabilities in real-world knowledge, following darija instructions, and performing traditional nlp tasks such as translation, summarization, and sentiment analysis. in the end, atlas-chat models foot_2 , fine-tuned from the gemma 2 models @xcite on our instruction dataset, exhibit superior ability in darija, surpassing both state-of-the-art and arabic-specialized llms like llama, jais, and acegpt, according to automatic metrics and simulated win rates. additionally, we conduct an experimental analysis of various fine-tuning strategies and base model choices to determine final configurations. we provide some examples by chatting with our models in appendix d. all our resources are publicly accessible, and we believe our work offers comprehensive design methodologies of instruction-tuning for low-resource languages. and code data. containing 116b arabic tokens-25% of which were translated from english-jais was designed to enhance performance in both arabic and english tasks, trained on a mixture of the two languages in a 1:2 ratio. however, this approach may suffer from localization issues. acegpt @xcite aims to address localization issues by pre-training llama 2 @xcite 7b and 13b models on 30b and 10b token mixtures, respectively, of arabic and english data, with the arabic portion dominating the dataset. the models were then fine-tuned on arabic instructions and aligned with arabic values and culture using rlaif @xcite . they further introduced the arabic cultural and value alignment dataset, comprising 8,000 yesno questions. allam (bari et al., 2024) demonstrated that second-language acquisition can steer the model towards a new language without catastrophic forgetting, even with random initialization of weights. they hypothesize that low-resource languages are diluted in large volumes of highresource languages, and pre-train a 7b model from scratch on 4t english tokens, followed by training on a 1.2t mixture of arabic and english. regarding darija, @xcite is currently the only "llm" dedicated to the moroccan arabic dialect. the model was trained on ∼100m tokens. however, darijabert is encoder-only, and no decoder-only models have been developed for darija. arabic benchmarks for llms. several arabic benchmarks have been created for various tasks. arabicmmlu @xcite is an arabic adaptation of the original mmlu benchmark @xcite , consisting of 14k multiple-choice questions across 40 tasks in msa. the benchmark covers a wide range of subjects, including history, mathematics, science, and linguistics, reflecting educational levels from eight different countries. larabench @xcite , a benchmark designed for evaluating msa llms on several practical nlp tasks, such as sentiment analysis, named entity recognition, and machine translation, spanning 33 tasks across 61 datasets encompassing ∼ 296 data points. the open arabic llm leaderboard (oall) foot_3 aggregates various native and translated arabic benchmarks to evaluate models' performance across tasks such as reading comprehension, reasoning, and more. ## data overview in developing atlas-chat, we chose to use instruction-tuning on a base model rather than training from scratch. this decision was primarily driven by the fact that training an llm from the ground up requires extensive data, which is not readily available for darija, a low-resource dialect. for the same reason, our training process does not include the additional continual pre-training phase typically seen in many language adaptation efforts. however, to mitigate this limitation, we designed a synthetic instruction dataset (see section 5.3) that, to some extent, mimics the next-word prediction task over a relatively longer context, typically performed during (continual) pre-training. moreover, recent studies show that multilingual llms often exhibit a bias toward internally solving tasks in english, even when trained on multiple languages @xcite , and perform best with english prompts, followed by mixed prompts, while non-english prompts significantly underperform @xcite . this observation led us to limit the scope of our work to a monolingual llm, making atlas-chat darija-centric. we focus on developing a model that accurately understands prompts written in darija, generates darija content, respects its cultural context, and remains accessible and adaptable for native speakers. therefore, we directed our efforts towards creating an extensive and diverse darija dataset for instruction-tuning. ## translated english instruction datasets finally, we broadened our instruction-tuning data by translating english datasets into darija, to cover a wider array of scenarios, domains, and tasks. we began by reviewing the most widely used datasets for fine-tuning state-of-the-art models to ensure that our translation efforts would lead to meaningful improvements. after careful consideration, we decided to focus on the tülu-v2-mix @xcite 16 dataset for several reasons. it offers a comprehensive dataset composition, including samples from some of the most widely used datasets, such as flan and sharegpt, for fine-tuning state-of-the-art models. appendix b.1 presents descriptions of each of these datasets and describes how the subset was sampled. the dataset mixture was meticulously designed based on ablation studies of both human-annotated and aigenerated data, with a focus on complexity and diversity. models fine-tuned on it showed significant improvements in overall performance on key benchmarks compared to those trained on individual datasets. we adopted the user-assistant message format from tülu-v2-mix (see appendix b.2) to structure our entire darija-sft-mixture dataset. to ensure quality, we first filtered out instructions from tülu-v2-mix that are either inappropriate for typical darija speakers or could lose meaning or coherence when translated, such as scientific content, translation tasks, and non-english samples. we then experimented with several opensource and closed-source models for english-to-darija translation, including nllb @xcite , gpt, and others. our results showed that closed-source models consistently outperformed open-source alternatives, with claude 3.5 sonnet emerging as our final choice. finally, we implemented several post-processing measures to correct errors introduced by the automatic translation. all details are provided in appendix b.3. ## training details in this section, we outline the training details and present the experimental analysis of various finetuning strategies and base model choices that informed our final settings. base model selection. initially, we considered the two arabic models: jais and acegpt (as allam is not open-weights). later, we included gemma 2 based on positive feedback from arabic llm community, as it can serve as a strong starting point for arabic fine-tuning tasks. we also compared the performance differences between fine-tuning on an instruction-tuned model and a base model. our results indicate that continual fine-tuning of instruction-tuned gemma 2 models (gemma-2-2b-it, 9b-it 17 , and 27b-it) yields significantly higher scores than other settings on our dataset. training framework. we also investigated the performance differences between full fine-tuning and parameter-efficient approaches. results indicate that the latter, with low-rank adaptation (lora) @xcite , proved to be more effective, whereas full fine-tuning resulted in catastrophic forgetting @xcite . this is supported by the recent work of @xcite , that shows lora exhibits a desirable form of regularization: it better maintains the base model's performance on tasks outside the target domain, and it also helps maintain more diverse generations. hyperparameters. lora was set with rank 256 and alpha 128. we run the training for 3 epochs, and set the learning rate to 5e-5 with warmup ratio of 3%, and per_device_train_batch_size to 4, 17 https://hf.co/google/gemma-2-9b-it with gradients accumulated over 4 steps.the maximum input context length was configured to 2048. we used bfloat16 to optimize training speed. the loss is computed only on the responses, not on the prompts of instructions. the atlas-chat models were trained on 8 nvidia a100 80 gb gpus in parallel, utilizing fsdp strategy on aws sagemaker. ## evaluation benchmarks to evaluate llm performance in darija, we developed a comprehensive suite that includes benchmarks such as darijammlu, darijahellaswag, darijaalpacaeval, and darijabench. additionally, we evaluated using an existing benchmark, belebele. all our custom benchmarks are integrated into a fork foot_14 of the lm-evaluation-harness repository @xcite to ensure reproducibility and foster future model comparison. darijammlu foot_15 . it is constructed by translating two major benchmarks into darija from english and msa: massive multitask language understanding (mmlu) @xcite 20 and arabicmmlu @xcite 21 , whose subsets that were either too technical (beyond typical user needs) or culturally inappropriate for the moroccan context were excluded. the remaining samples were translated into darija using claude 3.5 sonnet. the benchmark consists of 22,027 multiple-choice questions, with the number of choices ranging from 2 to 5. the subsets we selected are listed in c.4. darijahellaswag foot_18 . hellaswag foot_19 @xcite is a multiple-choice dataset designed to evaluate machine reading comprehension and commonsense reasoning. it presents complex scenarios where models must select the most plausible continuation of a passage from four options, challenging nuanced language understanding and contextual inference. using claude 3.5 sonnet, we translated the hellaswag validation set into darija. belebele_ary. belebele @xcite foot_20 is a multiple-choice machine reading comprehension dataset designed to evaluate both monolingual and multilingual models across 122 languages. each question is paired with a brief passage and offers four multiple-choice answers. for our work, we specifically used the ary_arab (indicating moroccan arabic) subset of belebele. darijaalpacaeval foot_21 . claude 3.5 sonnet was prompted to translate and culturally adapt the al-pacaeval dataset @xcite into darija, to evaluate the instruction-following capabilities and cultural alignment of llms in darija. the dataset consists of 805 instructions, focusing on culturally relevant content tailored to the moroccan context. more details about the dataset creation and evaluation method can be found in appendix c.3. darijabench foot_22 . in addition to the above benchmarks, we evaluated with the test sets from the native darija datasets (see section 4). typically, 10% of each subset is reserved for testing, unless the original source provides a pre-defined separate test set. the test sets for the three tasks collectively are referred to as darijabench. ## results evaluation measures. we employed accuracy to evaluate models on multiple-choice benchmarks, including darijammlu, darijahellaswag, bele-bele_ary, and the discriminative sentiment analysis task within darijabench. for translation and summarization tasks, we adopted the conventional bleu @xcite and rouge-1/l @xcite , respectively. however, since these metrics are based on n-grams, they are not well-suited for assessing darija. for example, the same word in darija can be written in multiple ways ("how are you?" = " " = " " = " ") due to the lack of standardization (e.g., diacritics, agglutinations, borrowings), making them overly rigid in cases where slight variations still convey the same meaning. to gain a more fine-grained insight, we also included chrf @xcite , operating at the level of character n-grams. in addition, to capture higher-level semantic similarity, we also used bertscore @xcite , with darijabert as the reference model for summarization, and multilingual bert foot_23 for translation. these evaluations were conducted in a zero-shot setting using greedy decoding, and some in a few-shot setting. the number of few-shot examples was chosen based on relevant literature and standard practices. for summarization evaluation, we also employ the llm-as-a-judge approach @xcite , where a model judges the preferred summary between a reference and a generated one, based on predefined criteria. we report the win-rate, defined as the percentage of instances where the generated summary is chosen over the reference. detailed information on the judge model, prompt, bias mitigation, and selection criteria is in appendix c.5. darijaalpacaeval employs the same approach as llm-as-a-judge, where we choose jais-13b-chat, the first arabic-specialized llm, as the reference. for these two evaluations, we applied the default sampling-based decoding. baseline models. we compared atlas-chat with instruction-tuned models from new jais series (including the -family models trained from scratch and the -adapted ones based on llama 2), along with acegpt, llama 3.1, 3.2, and gemma 2 (our base model). given that atlas-chat features 2b, 9b, and 27b sizes, we extended our comparison to the closest larger-sized model above 27b when available, while included all smaller-sized ones. the strong zero-shot performance of atlas-chat is further enhanced by the larger-sized atlas-chat-9b, which consistently outperforms other baseline models with parameters less than or equal to 13b, achieving the highest scores in 14 out of 16 metrics. its strength is especially evident in translation as it leads in all three metrics, chrf, bleu, and bertscore, by a significant margin. moreover, the model excels in darijammlu, darijahellaswag, belebele_ary, darijaalpacaeval, and sentiment analysis, surpassing larger models like acegpt-13b-chat and jais-family-13b-chat. our largest model, atlas-chat-27b, consistently outperforms competitors, including jaisfamily-30b-8k-chat and gemma-2-27b-it. in darijammlu, darijahellaswag, belebele_ary, and darijaalpacaeval, it achieves zero-shot performance gaps of 10.07%, 12.76%, 1.51%, and 10.00%, respectively, over the highest-performing competitor. similarly, in translation and summarization tasks, atlas-chat-27b demonstrates significant zero-shot performance advantages over its closest competitor, with substantial performance improvements over all evaluation metrics. few-shot performance. atlas-chat demonstrated further improvements when moving from the zeroshot to the few-shot setting, with the effect being particularly pronounced for the 9b and 27b models, especially on the belebele_ary benchmark. however, this enhancement in few-shot performance is not observed for the atlas-chat-2b model, despite consistently outperforming competitors. ## conclusion we presented atlas-chat, the first collection of llms specifically developed for moroccan darija. we constructed a comprehensive instruction dataset by consolidating native, synthetic, and translated resources. we also introduced several benchmarks, including both discriminative and generative tasks. atlas-chat models showed superior performance in following darija instructions and executing standard nlp tasks, outperforming both state-of-theart and arabic-specialized llms. our work highlights the potential of targeted llm development for underrepresented languages and offers design methodologies of instruction-tuning that can be applied to similar language adaptation challenges.