A lcla M : A rabic Dialect Language Model (2024)

Abstract: Pre-trained Language Models (PLMs) are integral to many modern natural language processing (NLP) systems. Although multilingual models cover a wide range of languages, they often grapple with challenges like high inference costs and a lack of diverse non-English training data. Arabic-specific PLMs are trained predominantly on modern standard Arabic, which compromises their performance on regional dialects. To tackle this, we construct an Arabic dialectal corpus comprising 3.4M sentences gathered from social media platforms. We utilize this corpus to expand the vocabulary and retrain a BERT-based model from scratch. Named AlcLaM, our model was trained using only 13GB of text, which represents a fraction of the data used by existing models such as CAMeL, MARBERT, and ArBERT, compared to 7.8%%, and 21.3%, respectively. Remarkably, AlcLaM demonstrates superior performance on a variety of Arabic NLP tasks despite the limited training data. AlcLaM is available at: https://github.com/amurtadha/Alclam.

## introduction pre-trained language models (plms) utilizing selfsupervised learning techniques, such as bert @xcite and roberta @xcite , have become pivotal in advancing the field of natural language processing (nlp) through transfer learning. these models have significantly enhanced performance across a variety of nlp tasks by leveraging vast amounts of textual data and extensive computational resources. however, the necessity for large corpora and the substantial computational demand, often requiring weeks of training time @xcite @xcite , has primarily confined the development of such models to the english language and a few other major languages. this limitation has sparked an increased interest in creating multilingual models capable of understanding and processing multiple languages simultaneously. innovations such as mbert @xcite , xlm-roberta @xcite and labse @xcite aim to address this gap. despite these efforts, the performance of these multilingual models typically lags behind their monolingual counterparts. this discrepancy is largely due to smaller, language-specific vocabularies and less comprehensive language-specific datasets @xcite @xcite @xcite . furthermore, while languages with similar structures and vocabularies may benefit from shared representations @xcite ), this advantage does not extend to languages such as arabic. arabic's unique morphological and syntactic structures share little in common with the morphosyntactic frameworks of more abundantly represented latinbased languages. to address this, various arabicspecific plms have been developed, including @xcite , arbert @xcite , and camel @xcite . these models significantly enhance arabic nlp tasks over multilingual models. however, they are predominantly trained on modern standard arabic (msa) datasets. this focus on msa introduces two primary limitations: first, there is reduced recognition of dialectal tokens, which vary widely across different arabic-speaking regions; second, there is a biased weighting towards msa tokens in the models, which may not accurately reflect the linguistic nuances present in everyday arabic usage. in this paper, we first introduce a new corpus of 3,372,744 arabic dialectal texts, meticulously sourced from social media platforms such as youtube and facebook. second, we outline the procedure for pretraining the transformer model @xcite specifically for the arabic language, which we dub alclam. note that we only train alclam on 13gb text due to computational resources limitation. finally, we assess alclam's performance on three arabic nlu downstream tasks, each distinct in nature: (i) arabic dialect identification (did), (ii) sentiment analysis (sa), and (iii) hate speech and offensive language detection. despite the limited training data, our experimental results demonstrate that alclam attains state-of-the-art results on most datasets, surpassing several baseline models, including previous multilingual and single-language approaches. in summary, our contributions are twofold. ## related work pre-trained language models (plms) using a selfsupervised masking objective, such as bert @xcite and roberta @xcite , have significantly advanced nlp. these models have multilingual versions, including mbert @xcite , xlm-roberta @xcite and labse @xcite . additionally, models featuring different objectives or architectures, such as albert @xcite , t5 @xcite , its multilingual variant mt5 @xcite , and gpt-3 (brown et al., 2020), llama @xcite , palm (chowdhery et al., 2023), gpt-4 (openai, 2023), and ro-former @xcite have been introduced. non-english plms have also been developed. these include bertje for dutch @xcite , camembert @xcite and flaubert @xcite for french, phobert for vietnamese (nguyen and nguyen, 2020), as well as models for finnish by @xcite , for polish by @xcite , and for swedish by @xcite . @xcite have created monolingual lms using wikipedia data for 42 languages. for arabic, msa-based plms includes arabert (antoun et al., 2020) ara-bicbert @xcite , arbert @xcite . another line of research involves pre-training models on a combination of msa and dialectal data, such as mdbert @xcite and camel @xcite . our contributions to this field include a comprehensive arabic dialectal corpus spanning various dialects and the development of an arabic plm. our model, named alclam, enhances the representation of linguistic diversity in arabic nlp. ## methodology in this paper, we develop alclam, an arabic dialect language representation model that enhances the performance on several arabic nlp tasks. this model builds upon the bert architecture, a stacked bidirectional transformer encoder @xcite . recognized as the foundation for many state-of-the-art results in various nlp tasks across multiple languages, bert's architecture has proven highly effective. below, we detail the dialectal corpus used for alclam's pretraining, the pretraining setup, and the fine-tuning process. ## conclusion in this paper, we present alclam, a novel bertbased model trained specifically to address the challenge of arabic dialectal variation. leveraging a carefully curated corpus sourced from social media platforms, alclam its alternatives across various arabic nlp tasks, despite being trained on significantly less data. for future work, expanding the dialectal vocabulary without increasing inference costs, inspired by chinese character modeling.