AL - QASIDA : Analyzing LLM Quality and Accuracy Systematically in Dialectal A rabic (2025)

Abstract: Dialectal Arabic (DA) varieties are under-served by language technologies, particularly large language models (LLMs). This trend threatens to exacerbate existing social inequalities and limits LLM applications, yet the research community lacks operationalized performance measurements in DA. We present a framework that comprehensively assesses LLMs’ DA modeling capabilities across four dimensions: fidelity, understanding, quality, and diglossia. We evaluate nine LLMs in eight DA varieties and provide practical recommendations. Our evaluation suggests that LLMs do not produce DA as well as they understand it, not because their DA fluency is poor, but because they are reluctant to generate DA. Further analysis suggests that current post-training can contribute to bias against DA, that few-shot examples can overcome this deficiency, and that otherwise no measurable features of input text correlate well with LLM DA performance.

## introduction large language models (llms) have transformed natural language processing (nlp) for many languages @xcite . however these technologies often lack support for minority dialects and language varieties @xcite @xcite @xcite . arabic, the fourth most spoken macro-language in the world with 420m speakers and official status in 26 countries @xcite , has a diversity of such varieties. despite its prominence, arabic has been historically set back in nlp due to its script, which lacked digital support until the 1980s @xcite . many nlp tools support arabic now, but often only modern standard arabic (msa); the numerous dialectal arabic (da) varieties are often neglected. there are 28 microlanguages designated as arabic in iso 639-3, foot_0 of which msa is only one. in the arab world, msa is used only in narrow circumstances, while local da varieties are predominant @xcite . msa has no native speakers according to ethnologue; foot_1 arabic speakers speak their da varieties natively (as l1) and later learn msa as an l2 @xcite . many who lack educational resources are not proficient in msa @xcite . arabic varieties are diverse and differ both phonologically, morphologically, syntactically, semantically, and lexically (habash, 2010; @xcite . according to @xcite , moroccan arabic or darija (ary) and egyptian arabic (arz) are as mutually intelligible as spanish and romanian. many llms today are proficient in msa but reluctant to model da; users wishing to converse in da often have to perform extensive prompt acrobatics to coerce the llm to use the dialect. 3 beeven when interacting with msa-proficient users, replying to informal da inputs with formal msa sounds unnatural and limits an llm's uses. llm pre-training data is diverse and likely includes content in many arabic dialects. however, post-training uses existing human-labeled data, which is typically msa, and newly collected data that commonly adopt an official tone corresponding to a formal msa register. current llm behavior suggests that models have the capability to understand and model da, but that they default to msa regardless, which could be caused by biases introduced in post-training. we thus hypothesize: hypothesis 1. current post-training methods make llms more reluctant to model da. hypothesis 2. post-trained llms understand da better than they generate it. arabic-speaking communities are aware of llms' da shortcomings, but lack a standard operationalized definition of llm da proficiency. to this end, we present an evaluation suite of da proficiency along different dimensions. we contribute: and further analysis suggests the following: we release software and data to conduct al-qasida evaluations, as well as supplementary material, on our project repository. 5 ## related work and background there exist notable prior works on benchmarking nlp for da varieties. arabench @xcite is a benchmark for mt between arabic varieties and english that predates llms' widespread popularity. dialectbench @xcite is a benchmark of 10 text-based traditional nlp taskssuch as parsing, part-of-speech tagging, and named entity recognition-that covers a large number of dialects and varieties in various languages, including multiple da varieties. @xcite compares mt5 @xcite with novel arat5 de-noising language models across mt, summarization, paraphrasing, and question generation tasks. dolphin @xcite is a comprehensive arabic nlp benchmark that evaluates da, but does not define da proficiency. aradice @xcite is an llm benchmark much like ours that focuses on accuracy and cultural appropriateness in da. our work differs from these in its purpose: to define llm da proficiency. to our knowledge, ours is the first evaluation to measure this comprehensively. ## methodology to be proficient in da, an llm requires different competencies that we define as the following: ## evaluation and results we detail our evaluation of nine llms for eight country-level da varieties: kuwait, saudi arabia, syria, palestine, sudan, egypt, algeria, and morocco. these constitute two varieties from each of four arabic dialectal regions (see figure 1 ). ## follow-up experiments and analysis after conducting our primary evaluations, which produced adi2 scores and both automatic and manual mt scores for each llm, we analyzed llm outputs further. our motivation for this was (1) to explore inexpensive remedies to llms' da deficiencies, and (2) to test hypothesis 3 and explore how user input features elicit llm behaviors. we restrict this analysis to command-r+, gpt-4o, and llama 3.1 for depth and focus. we first explored few-shot learning-an inexpensive mitigation approach-to improve da modeling of the least performative of these llms: command-r+. we curated five in-context promptcompletion examples for each of for da varieties, for both monolingual and cross-lingual tasks, by translating the few shot examples used by @xcite in their related experiments. figure 7 shows that few-shot examples improve adi2 across tasks and genres for command-r+. we then analyzed relationships between prompt attributes and performance. we collected features for each prompt and completion of the nontranslation tasks: target da variety; prompt length; prompt template; genre or data source; llm used; number of few-shot examples; dialectness of input (for monolingual setting); and location of the dialect request (for cross-lingual) or inserted generic utterance (for monolingual) in the prompt. we fit decision tree regressors on adi2 score using these features and max_depth 3, shown in figure 8 . the predominant dichotomous features are the desired da variety and llm used. others include number of few-shot examples and dialectness of input text. for feature-level correlations with adi2 score, we performed spearman's rank tests for numerical features and anova tests for categorical features to find ρ and η 2 coefficients. we display these only for the relationships where @xmath0.001, in dav pt loc gen llm len n η 2 = ρ = mono. .115 .017 .012 .038 -.09 .12 cross. .131 .003 .067 .32