Empirical Evaluation of Pre-trained Language Models for Summarizing M oroccan D arija News Articles (2025)

Abstract: Moroccan Dialect (MD), or “Darija,” is a primary spoken variant of Arabic in Morocco, yet remains underrepresented in Natural Language Processing (NLP) research, particularly in tasks like summarization. Despite a growing volume of MD textual data online, there is a lack of robust resources and NLP models tailored to handle the unique linguistic challenges posed by MD. In response, we introduce .MA_v2, an expanded version of the GOUD.MA dataset, containing over 50k articles with their titles across 11 categories. This dataset provides a more comprehensive resource for developing summarization models. We evaluate the application of large language models (LLMs) for MD summarization, utilizing both fine-tuning and zero-shot prompting with encoder-decoder and causal LLMs, respectively. Our findings demonstrate that an expanded dataset improves summarization performance and highlights the capabilities of recent LLMs in handling MD text. We open-source our dataset, fine-tuned models, and all experimental code, establishing a foundation for future advancements in MD NLP. We release the code at https://github.com/AzzedineAftiss/Moroccan-Dialect-Summarization.

## introduction moroccan dialect (md), commonly known as "darija," is the primary spoken variety of arabic in morocco, coexisting with berber in some regions. approximately 91% of moroccans communicate in darija @xcite . with the rise of digital resources, md textual data available online is rapidly growing @xcite , creating a need for effective automatic summarization to help users extract key information efficiently. while extensive work has focused on widely spoken languages, such as english, limited research exists on md @xcite , es-pecially in sequence-to-sequence (seq2seq) tasks like summarization. challenges in md research include a lack of comprehensive corpora, limited linguistic resources, complex syntax that challenges nlp models, and unique vocabulary not present in standard arabic lexicons. although existing datasets, such as goud.ma @xcite , offer foundational resources, the evolving nature of the md necessitates additional, robust datasets. motivated by these challenges, we introduce good.ma_v2, an expanded version of the goud.ma dataset, containing over 50,000 articles with titles across 11 categories, aiming to enhance model robustness for md data. additionally, we explore llms for summarizing md text, analyzing various seq2seq models in addition to evaluating recent causal llms in a zero-shot prompting setting, thus contributing valuable insights into their performance on md. the emergence of llms has led to remarkable nlp advancements @xcite . however, adapting these models for low-resource languages, including the md, remains underexplored. recent efforts have attempted to adapt arabic-specific models (e.g., @xcite , @xcite , dziribert @xcite ) and multilingual models (e.g., mbart @xcite , mt5 @xcite ) for dialectal arabic @xcite @xcite . more recent models, such as gpt-4 @xcite and llama 3 @xcite , offer advanced nlp capabilities but have not yet been adapted to the md. in this paper, we conduct an empirical study of llms on goud.ma_v2 specifically curated for abstractive summarization. we demonstrate that expanding the dataset with additional samples improves summarization performance. our approach includes fine-tuning encoder-decoder models and adapting recent llms for zero-shot prompting, pro-viding comprehensive insights into the effectiveness of llms for md text summarization. our dataset, fine-tuned models, and all code used in our experiments are open-sourced. the main contributions of this paper are as follows: the rest of this paper is organized as follows: section 2 reviews related work on arabic text summarization, with a focus on arabic dialect summarization. section 3 details the dataset collection process. section 4 describes the experimental settings, while section 5 presents the experimental results. finally, section 7 discusses the conclusions and limitations of this work. ## related work arabic dialect processing has gained attention due to the linguistic diversity and widespread use of dialects in the arabic-speaking world. unlike modern standard arabic (msa), arabic dialects, such as moroccan darija, exhibit unique lexical, syntactic, and phonological variations @xcite , presenting challenges for nlp tasks due to limited labeled data and resources. early work in arabic dialect nlp focused on tasks like classification @xcite @xcite , identification @xcite @xcite , and translation @xcite . however, studies on dialectal summarization, particularly for md, remain sparse. @xcite introduced one of the first md summarization datasets, with articles paired with titles as reference summaries. arabic text summarization has advanced with methods like clustering, minimum redundancy-maximum relevance (mrmr), and graphbased approaches @xcite . deep learning techniques, such as seq2seq architectures with lstms and attention mechanisms, have also been explored @xcite . recent transformer-based models, such as arabart @xcite , @xcite , and arabert @xcite , pre-trained on large arabic corpora, have shown strong performance in arabic summarization tasks. additionally, multilingual models like mbart @xcite and mt5 @xcite , pre-trained on diverse language corpora, have demonstrated cross-lingual effectiveness, making them suitable for low-resource dialects, including moroccan darija. recently, models like @xcite and dziribert @xcite have been pretrained on north african dialectal arabic, specifically to address moroccan and algerian dialects. darijabert, focused on moroccan darija, incorporates dialect-specific vocabulary, bridging the gap between msa and regional dialects, thus enhancing contextual understanding compared to general arabic models. the emergence of advanced llms, such as gpt-4o mini @xcite , llama 3 @xcite have brought attention to their capabilities in domain-specific tasks in zero-shot or few-shot settings. these models exhibit strong reasoning and text generation capabilities without fine-tuning task-specific data. however, applying them to dialectal summarization has limitations, as their training data generally lacks comprehensive coverage of specific dialects, such as moroccan darija. finetuning remains essential to optimize performance for dialectal tasks. our work builds upon these prior studies by applying and comparing arabicspecific, multilingual, and causal llms for md summarization using both zero-shot and fine-tuning methods. to our knowledge, this is the first work that evaluated llms for moroccan darija abstractive summarization, contributing valuable insights to nlp research for dialectal arabic. ## experiment settings the main objectives of this study are twofold: (1) to expand the dataset to capture evolving dialectal variations and hence improve the performance of text summarization models, and (2) to evaluate the effectiveness of large language models in summarizing moroccan darija text. we conducted an empirical study to assess the performance of various llms, including pre-trained causal models and fine-tuned seq2seq models, on the good.ma_v2 dataset. in this section, we present the implementation details and a brief description of the models used for comparison. ## experimental results in this section, we present a comparative analysis of the pre-trained language models used on the good.ma_v2 dataset. ## results the results of our experiment are presented in on the other hand, the extractive summarization baselines (lead-2, textrank, sumrank) and causal language models (gpt-4o mini, llama 3, mistral nemo) achieved comparatively lower performance. the extractive baselines struggled to produce coherent summaries, as they simply concatenate sentences in an unsupervised manner, often leading to disconnected and inconsistent outputs. similarly, causal llms like gpt-4o mini and llama 3, which were employed using zero-shot prompting, did not perform as well as fine-tuned arabic-specific and multilingual models. this is likely due to the lack of fine-tuning, which limits their adaptability to moroccan darija summarization tasks. moreover, relying exclusively on rouge and bertscore metrics may not provide a comprehensive assessment of generative models. as noted by @xcite , automatic evaluation metrics like rouge and bertscore primarily measure n-gram overlap or embedding similarity, which may not fully capture the creative and contextually nuanced outputs generated by large language models (llms). for instance, llms like gpt-4o-mini and llama 3 are capable of rephrasing summaries in ways that differ from the reference text but still convey the intended meaning. for the mistral nemo model, we observed difficulties in understanding arabic texts and dialects, which led to hallucinations in the resulting summaries, ad-ditionally, during the experiment, we found that mistral nemo sometimes generated the summary in french or spanish. to mitigate this, we applied post-processing to translate these summaries into arabic using the google translate api 6 . this translation step may have impacted mistral's overall performance. to illustrate the qualitative differences across models, we provide example summaries generated by each model alongside the reference summary in ## conclusion in this study, we conducted an empirical evaluation of various pre-trained language models for abstractive summarization of moroccan darija text, comparing the performance of arabic-specific encoderdecoder models, multilingual models, and causal language models. our findings demonstrate that the multilingual model mbart, which is pre-trained on a diverse set of languages, generally achieved superior performance compared to the other models. the zero-shot application of causal language models, including gpt-4o mini, llama 3, and mistral nemo, showed potential; however, results indicated that fine-tuning would be necessary to achieve contextually accurate and fluent summaries in md. moreover, while automatic evaluation metrics like rouge and bertscore provided useful quantitative insights, they may not fully reflect qualitative aspects such as readability, fluency, and consistency-attributes that are crucial in summarization tasks. future research could further explore this area by fine-tuning causal language models for md summarization and incorporating human evaluation to gain a more comprehensive understanding of summary quality.