ALD i: Quantifying the A rabic Level of Dialectness of Text (2023)

Abstract: Transcribed speech and user-generated text in Arabic typically contain a mixture of Modern Standard Arabic (MSA), the standardized language taught in schools, and Dialectal Arabic (DA), used in daily communications. To handle this variation, previous work in Arabic NLP has focused on Dialect Identification (DI) on the sentence or the token level. However, DI treats the task as binary, whereas we argue that Arabic speakers perceive a spectrum of dialectness, which we operationalize at the sentence level as the Arabic Level of Dialectness (ALDi), a continuous linguistic variable. We introduce the AOC-ALDi dataset (derived from the AOC dataset), containing 127,835 sentences (17% from news articles and 83% from user comments on those articles) which are manually labeled with their level of dialectness. We provide a detailed analysis of AOC-ALDi and show that a model trained on it can effectively identify levels of dialectness on a range of other corpora (including dialects and genres not included in AOC-ALDi), providing a more nuanced picture than traditional DI systems. Through case studies, we illustrate how ALDi can reveal Arabic speakers’ stylistic choices in different situations, a useful property for sociolinguistic analyses.

## introduction arabic is spoken by more than 420 million people all over the world @xcite , and exists in a state of diglossia, in which two variants of the language co-exist in arabic-speaking communities @xcite . modern standard arabic (msa) is the standardized variant, which is taught in schools and used in formal communications and as a common language across all arab countries. however, many local variants of dialectal arabic (da) are used for daily communicationmainly in speech and speech-like text such as social media. they can diverge from msa and each other in phonology, morphology, syntax, and semantics ## the arabic level of dialectness (aldi) we define the level of dialectness of a sentence as the extent by which the sentence diverges from the standard language, which can be based on any of the cues described above. this definition is consistent with the crowd-sourced annotation of the arabic online commentary (aoc) dataset @xcite , where annotators labeled user comments on arabic newspaper articles by their dialect and their level of dialectness. however, the original and subsequent work only used the dialect labels, and the dialectness annotations have not previously been analyzed in detail. ## case studies (aldi in practice) the same speaker can adapt different styles according to various social and linguistic factors @xcite . the aldi of speech is one example of an intraspeaker variation in arabic. in this section, we provide two case studies analyzing the transcribed speeches of three different arab presidents. we highlight how quantitatively estimating the aldi can help in revealing different speaking styles. ## conclusion we presented aldi, a linguistic variable that quantifies the level of dialectness of an arabic sentence. we release aoc-aldi, a dataset of arabic comments annotated with their aldi scores. a bertbased regression model fine-tuned on aoc-aldi showed superior performance compared to existing baselines that are based on lexicons and di models. our analysis shows that the model generalizes to various arabic dialects. in addition, the model provides a nuanced distinction of dialectal features, which token and sentence di models can not perform. lastly, we presented multiple case studies demonstrating the effectiveness of aldi in revealing new insights in arabic text. for future work, we aim to explore the possible applications of aldi for text analysis, especially for sociolinguistics and computational social science studies. moreover, we aim to apply the level of dialectness work to other languages that have the same phenomena of arabic, such as swiss-german.