Are Dialects Better Prompters? A Case Study on A rabic Subjective Text Classification (2025)

Abstract: This paper investigates the effect of dialectal prompting, variations in prompting scrip t and model fine-tuning on subjective classification in Arabic dialects. To this end, we evaluate the performances of 12 widely used open LLMs across four tasks and eight benchmark datasets. Our results reveal that specialized fine-tuned models with Arabic and Arabizi scripts dialectal prompts achieve the best results, which constitutes a novel state of the art in the field.

## motivations large language models (llms) have gained significant attention for their ability to perform various nlp tasks, including text classification. studies have demonstrated that these models achieve high accuracy on a wide range of classification tasks by leveraging prompt engineering and customized prompts @xcite @xcite . however, the results for subjective classification remain a challenge. for example, @xcite have shown that llms perform poorly in classifying highly subjective content, such as irony and emotions, where contextual subtleties play a crucial role. fine-tuning these models on large affective datasets has been shown to improve performance for tasks involving nuanced emotional understanding @xcite . additionally, studies indicate that llms tend to perform better when prompts are either in english or translated to english compared to when they are in the target language of the testset @xcite @xcite @xcite . this is particularly salient for low-resource languages such as dialectal arabic and its regional variants, where similar declines were observed @xcite @xcite @xcite . improvements have, however, been observed with code-mixed prompts in few-shot settings, i.e., instructions in english and examples in msa @xcite . when it comes to subjective text classification in arabic, three main challenges remain: (1) taskspecific transformer fine-tuned models consistently outperform llms when prompted in zero or fewshot settings @xcite , (2) the results when using native language prompts in msa vs. english were not conclusive (e.g., increase in subjectivity detection but decrease in sentiment analysis @xcite ), and (3) llms struggle with more nuanced subjective tasks like detecting sarcasm and emotions @xcite . in this paper, we aim to address these challenges by investigating, for the first time as far as we know, the effectiveness of dialect-specific prompting in improving subjective text classification in arabic. our work makes contributions in both advancing prompt engineering methodologies and studying the effects of linguistic variation in prompt design, focusing on the following dimensions: ## results to reduce randomness and enhance generalizability, all models are evaluated on the same test set and run three times with different seed of examples, if any; we report the averaged macro-f1 across runs. in-dataset results. the main findings are as follows (cf. for instance, llama and its derivative as well as acegpt excel in gulf arabic, while jais in levantine. additionally, mutlilingual models often confuse dialects (e.g., gulf with levantine or msa) and tend to switch to msa in uncertain cases. in-dataset da results. cross-dataset results. we measured transfer learning of our best models across sa and ed tasks (the two others containing one dataset each). for sa (see fine-tuning transfer results. models finetuned on a specific dialect exhibit varying levels of robustness when tested on different dialects (see yellow cells in tables 9 and 10 in appendix d.2). in particular, models initially trained on larger data from msa and levantine/gulf exhibit greater adaptability when fine-tuned on other regionally specific dialects (e.g., tunisian, algerian arabizi), see for e.g., the sem17 (tn ar ) model that achieved the highest performance in sa, followed by the twifil-sent (dz abz ) model -both built on acegpt 8b . a similar trend is observed in ed, demonstrating superior generalization, therefore maintaining a competitive score comparable to the best in-dataset fine-tuning. ## discussions comparison with state of the art. we acknowledge that while dialectal prompting improves performance over msa prompting, the magnitude of these improvements varies across dialects. overall, the improvements range from 1% (twifil-sent dz ) to 7% (idat m sa+m ixed ) as shown in the first line of for the other datasets, it is important to mention that direct comparison with reported results is unfair as we are using different experimental settings due to differences in the main task (e.g., sem18 emotion shared task is a multi-label classification and not a multiclass classification), number of classes (e.g., the latest reported results @xcite on twifil-emo dz (resp. sem18 m sa+m ixed ) multiclass task use a set of 6 emotion classes and not the whole set composed of 10 (resp. 11) classes as we do here), or evaluation setting (e.g., in the flood corpus, the train/test split is done by crisis events vs. random in our case). to allow a better comparison, we therefore designed similar baseline transformer models that we finetuned on each dataset following the same train/test split ratio across datasets. additionally, we conducted statistical significance testing to assess the impact of dialectal prompting compared to msa prompting. results reveal that dialect-specific prompting significantly outperforms msa prompting (at @xmath2.05) on six out of eight datasets, providing strong empirical support for incorporating dialectal variation in prompt design. mixed dialectal data. our findings reinforce the advantage of using dialectal arabic for prompting, especially when fine-tuned to align with the dataset's dominant script. we have however to consider cases where datasets are composed of mixed dialectal data. in this case, selecting a single dominant dialect for prompting can be challenging. however, most dialects share a substantial overlap with msa, which serves as a linguistic backbone for many arabic dialects. maghrebi dialects. they are among the least represented, as they incorporate more amazigh and french words. despite this, they are quite similar which benefits fine-tuning. for example, in tsac t n , using algerian prompts yielded the best results. similarly, in sem17 m sa+m ixed , msa performed best before fine-tuning, but post-finetuning, tunisian showed greater consistency, followed by algerian, especially using acegpt. this suggests that fine-tuning amplifies dialect-specific nuances, enabling the model to better adapt to unique characteristics of dialectal data. notably, maghrebi data within the dataset, which might have been misclassified when using msa. this is illustrated in the following example from twifilemo dz : ## conclusion this study examined the ability of 12 open source llms to handle arabic dialects in both arabic and arabizi scripts to improve subjective text classification. we highlighted the impact of arabic-centric llms fine-tuning and prompt design on models performances, showing that dialectal prompts generally outperformed msa and english prompts after fine-tuning. transfer learning, on the other hand, demonstrated limited effectiveness. while models pre-trained on msa and levantine/gulf showed some ability to adapt to other regionally specific varieties (e.g, maghrebi), their performances remained significantly lower compared to in-dataset configuration. we believe our study will open the door to future directions in prompting llms for low-resource languages, beyond english or translated prompts.