Testing the Boundaries of LLM s: Dialectal and Language-Variety Tasks (2025)

Abstract: This study evaluates the performance of large language models (LLMs) on benchmark datasets designed for dialect-specific NLP tasks. Dialectal NLP is a low-resource field, yet it is crucial for evaluating the robustness of language models against linguistic diversity. This work is the first to systematically compare state-of-the-art instruction-tuned LLMs—both open-weight multilingual and closed-weight generative models—with encoder-based models that rely on supervised task-specific fine-tuning for dialectal tasks. We conduct extensive empirical analyses to provide insights into the current LLM landscape for dialect-focused tasks. Our findings indicate that certain tasks, such as dialect identification, are challenging for LLMs to replicate effectively due to the complexity of multi-class setups and the suitability of these tasks for supervised fine-tuning. Additionally, the structure of task labels—whether categorical or continuous scoring—significantly affects model performance. While LLMs excel in tasks like machine reading comprehension, their instruction-following ability declines in simpler tasks like POS tagging when task instructions are inherently complex. Overall, subtle variations in prompt design can greatly impact performance, underscoring the need for careful prompt engineering in dialectal evaluations.

## introduction natural language processing (nlp) systems have traditionally focused on high-resource languages, leaving dialectal variations underexplored @xcite . in this work, we address this gap by evaluating large language models (llms) on task-specific benchmark datasets curated for various dialects. dialectal tasks often lack the resources available for standard languages, but 1 code repository: https://github.com/ffaisal93/ dialectbench they provide critical insights into a model's robustness across linguistic diversity @xcite . to our knowledge, no prior studies have systematically assessed llm performance on dialectfocused nlp tasks. we compare llms such as @xcite and aya-101 @xcite with state-of-the-art multilingual encoder models like mbert @xcite and xlm-r @xcite to establish new baselines and identify areas where llms either excel or fall short. our contributions: we make several key contributions to the understanding of llm performance in dialect-specific tasks: overall, this study contributes to a deeper understanding of llm behavior in low-resource, dialectrich environments and emphasizes the need for tailored approaches when working with dialectal nlp tasks. task selection: we experiment with seven tasks from the dialectbench task collections. these tasks are: ## experimental setup this section outlines the selected language models for evaluation, along with the training and evaluation configurations. ## investigating dialect identification failure including explanation-prompt yields no improvement to investigate further the challenges faced by llms in dialect identification task, we conducted an ablation study on prompt-engineering to improve dialect identification performance. the experiment involved presenting varying numbers of example sentences @xmath0, 3, 10, 30, and 50 examples) per city-level dialect to gpt-4 and subsequently prompting it to generate refined instructions for the classification task (presented in fig. 2 ). we then used these refined prompts to evaluate the performance of aya-101. aya-101's strong bias toward sudanese arabic in our initial setup, we began with a detailed set of 26 city-level arabic dialects. to simplify analysis and improve model interpretability, we grouped these dialects into broader regional categories, such as maghreb, gulf arabic, levantine arabic, and egyptian arabic, as reported in we observe, aya-101, without additional explanations, exhibits a strong tendency to misclassify a wide range of dialects as sudanese arabic, despite sudanese arabic representing only a small fraction (200 instances) of the dataset. this misclassification does not align with the true label distribution, where maghreb (1400 instances), gulf arabic (1200), and levantine arabic (1000) are among the most represented dialects. aya-101's errors are predominantly concentrated within maghreb and gulf arabic groups, leading to a significant over-prediction of sudanese arabic. when provided with a longer prompt including additional explanations, aya-101 demonstrates improved differentiation, particularly in distinguishing levantine and egyptian arabic from other groups. however, this extended prompting introduces a new issue: a portion of predictions are left blank, marked as "no prediction", indicating instances where aya-101 fails to respond with a specific classification. this is a significant limitation, as such non-responses reduce the model's effective prediction rate. furthermore, aya-101 continues to show substantial within-group confusion, especially among dialects within the gulf and maghreb regions, even with additional explanation. in comparison, gpt-4 demonstrates the most robust performance across dialects. it closely aligns with the true label distribution and shows higher accuracy in identifying key groups such as maghreb, levantine arabic, and modern standard arabic. although gpt-4 still exhibits within-group misclassification-such as confusing gulf arabic with iraqi arabic-it effectively differentiates between dialects overall. this indicates that, while longer prompts with explanations enhance aya-101's performance to some extent, gpt-4's inherent understanding of dialectal distinctions remains significantly stronger. ## related work the evaluation of language models has been a critical component in advancing natural language processing (nlp). evaluation benchmarks are necessary to provide standardized, reproducible comparisons across models, ensuring that improvements in architecture or training result in tangible performance gains on a variety of tasks @xcite . popular benchmarks such as xtreme @xcite and glue @xcite are designed to assess models, primarily focusing on standard language varieties and tasks like text classification and structural prediction. with the development of large language models (llms), recent benchmarks have expanded to include reasoning capabilities and expert domain knowledge. examples include benchmarks like superglue @xcite , bigbench @xcite , and mmlu @xcite , which evaluate models on complex reasoning, knowledge-intensive tasks, and multi-domain expertise. these benchmarks are increasingly multilingual, but they still largely overlook dialectal and non-standard language varieties across diverse tasks. efforts in dialectal nlp have emerged, such as the arabic dialect corpus madar @xcite and resources developed through the vardial workshop @xcite , such as dsl-tl @xcite and dialect-copa @xcite . however, these datasets remain largely scattered, and no unified benchmark exists to comprehensively evaluate language models on dialectal and non-standard varieties across multiple languages and tasks. di-alectbench @xcite attempts to address this by aggregating dialectal datasets using a standardized approach with glottocode mapping for language clusters and varieties. however, it primarily evaluates smaller encoder models and does not comprehensively explore dialectal tasks using recent advancements in large language models. structured studies that leverage llms to evaluate a broad range of dialectal tasks remain largely unexplored. ## conclusion in this study, we evaluated the performance of encoder-based models and llms on various dialect-specific nlp tasks. our results indicate that while llms such as gpt-4 and aya-101 excel in tasks like topic classification and natural language inference, they struggle with complex instructions and formatting, particularly in parts of speech (pos) tagging and dialect identification. in contrast, fine-tuned encoder models outperform llms in highly structured tasks such as pos tagging and extractive question answering. these findings suggest that while llms have potential, taskspecific fine-tuning or hybrid approaches are still necessary for effectively handling nuanced, low-resource dialects.