ABDUL : A New Approach to Build Language Models for Dialects Using Formal Language Corpora Only (2025)

Abstract: Arabic dialects present major challenges for natural language processing (NLP) due to their diglossic nature, phonetic variability, and the scarcity of resources. To address this, we introduce a phoneme-like transcription approach that enables the training of robust language models for North African Dialects (NADs) using only formal language data, without the need for dialect-specific corpora.Our key insight is that Arabic dialects are highly phonetic, with NADs particularly influenced by European languages. This motivated us to develop a novel approach in which we convert Arabic script into a Latin-based representation, allowing our language model, ABDUL, to benefit from existing Latin-script corpora.Our method demonstrates strong performance in multi-label emotion classification and named entity recognition (NER) across various Arabic dialects. ABDUL achieves results comparable to or better than specialized and multilingual models such as DarijaBERT, DziriBERT, and mBERT. Notably, in the NER task, ABDUL outperforms mBERT by 5% in F1-score for Modern Standard Arabic (MSA), Moroccan, and Algerian Arabic, despite using a vocabulary four times smaller than mBERT.

## introduction nads, including moroccan, algerian, and tunisian, introduce additional complexities. influenced by berber languages and colonial languages such as french and spanish, these dialects display notable phonetic variability, including vowel inconsistency and the adoption of phonemes absent in msa, such as /p/ and /v/ @xcite . in addition, their lexicons are enriched by extensive borrowing from french and spanish and often incorporating them with phonetic modifications @xcite . in this article, we introduce a phoneme-like transcription approach that bridges formal arabic with dialectal varieties through linguistic normalization. inspired by the buckwalter @xcite transliteration system, our method simplifies and adapts transliteration by clustering phonetically similar sounds, improving alignment with dialectal phonetic patterns. to highlight consonants and long vowels (e.g., the "ā" in the word kitāb for "book" which is pronounced with an extended duration of the vowel /a/), this approach deliberately omits diacritization and even removes preexisting diacritics from the text, reducing phonetic variability @xcite . by transforming arabic script into a standardized phoneme-like latin representation, this preprocessing pipeline promotes cross-script and crossdialect generalization, allowing for the development of robust nlp models trained solely on formal language data. in this article, we will focus exclusively on transliterating msa to handle arabic dialects, with the future goal of including french and code-switched text, given their significance in nads. ## linguistic justification nads are low-resource languages with no formal or standardized grammatical rules, relying mainly on direct phonetic transcription. alongside msa vocabulary, they feature extensive lexical borrowings from french, spanish, turkish, and italian, reflecting the historical and colonial influence of these languages in the region. the lexical resemblance between algerian arabic (alg) and msa has been quantitatively analyzed using computational methods. abukwaik et al. @xcite employed latent semantic indexing (lsi) to assess lexical overlap between msa and various arabic dialects, reporting an lsi similarity score of 0.68 for algerian arabic. this score indicates a moderate lexical divergence, suggesting that while some vocabulary is shared, directly applying msa-trained models to algerian arabic could result in significant tokenization mismatches. harrat et al. @xcite found that approximately 20% of algerian dialectal words originate from arabic, while 34% are derived from msa. studies estimate that loanwords make up around 30-40% of the vocabulary in these dialects, particularly in technical, educational, and governmental contexts @xcite . in @xcite , the authors argue that significant variations between and msa occur in vocalization, along with the omission or modification of certain letters, particularly the hamza foot_0 . despite the influence of foreign lexicons, nads preserve core linguistic structures from msa. however, in terms of pronunciation, menacer et al. @xcite found that 46% of msa-derived words in nads exhibit phonetic variations compared to their standard msa counterparts. another key characteristic of nads is their strong dependence on consonantal structures for lexical and semantic distinctions, as vowel patterns vary significantly across regions @xcite . given these linguistic properties, abdul leverages stable consonantal structures, which serve as robust subword units for training nlp models, reducing variability caused by inconsistent usage of vowels. ## related work nads are low-resource languages that lack formalized grammatical rules and primarily rely on phonetic transcription. in this work, we propose a novel paradigm for training language models for nads using only formal language corpora, eliminating the need for dialect-specific datasets. to evaluate the effectiveness of our approach, we compare it against several key baselines in arabic nlp, particularly those designed for arabic dialects: beyond these baselines, our approach is further inspired by the study "consonant is all you need" (al-shaibani and ahmad, 2023), which highlights the benefits of reducing reliance on vowels for more efficient nlp models. this work demonstrates how selectively omitting certain lexical features can lead to smaller vocabularies, lower computational complexity, and improved training efficiency. these insights align with our dediacritization and consonant-centric transcription strategy, reinforcing the scalability and effectiveness of our method. through a rigorous comparative analysis, we aim to underscore the advantages of our approach. by pretraining a language model from scratch on data processed via our pipeline, we establish a fair and consistent benchmark to demonstrate the benefits of phoneme-like transcription for arabic dialectal nlp. our work contributes to the broader goal of improving low-resource language modeling through linguistically informed methodologies. ## methodology to effectively adapt formal arabic resources for dialectal nlp, we develop a preprocessing pipeline that normalizes phonetic variability while preserving linguistically significant features. in the following, we outline the key steps in our phoneme-like transcription process. ## datasets in this section, we describe the datasets used for pretraining and benchmarking abdul, covering msa and nads. our selection includes a largescale corpus in msa for pretraining and multiple dialect-specific datasets for downstream tasks, ensuring a comprehensive evaluation across emotion classification and named entity recognition (ner). ## results the performance of abdul is evaluated on two tasks: emotion classification and named entity recognition (ner), across multiple arabic variants. the results demonstrate abdul's ability to generalize effectively across dialects while maintaining competitive performance against specialized models. ## limitations and future work while abdul demonstrates strong performance in dialectal nlp tasks, several limitations remain. currently, our approach does not support latinscript arabizi dialects, which are widely used in informal settings. expanding abdul to handle arabizi is a key part of our future work. additionally, we plan to investigate how vocabulary size impacts performance, as well as how different formal languages used in pretraining (e.g., french, spanish, and english) influence the model's ability to generalize across dialects. overall, the results are low for state-of-the-art models, including abdul. the task will be to test other architectures to improve the results and not settle for the current ones. finally, we aim to expand abdul's applicability to a broader set of nlp tasks, including machine translation and text generation, to further assess its scalability and effectiveness in diverse linguistic contexts. as a long-term objective, we seek to build the first large language model (llm) for arabic dialects, leveraging the high availability and quality of formal languages data to address the lowresource status of arabic dialects and advance the field of dialectal arabic nlp.