E n D ive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Models (2025)

Abstract: The diversity of human language, shaped by social, cultural, and regional influences, presents significant challenges for natural language processing (NLP) systems. Existing benchmarks often overlook intra-language variations, leaving speakers of non-standard dialects underserved. To address this gap, we introduce EnDive (English Diversity), a benchmark that evaluates seven state-of-the-art (SOTA) large language models (LLMs) across tasks in language understanding, algorithmic reasoning, mathematics, and logic. Our framework translates Standard American English datasets into five underrepresented dialects using few-shot prompting with verified examples from native speakers, and compares these translations against rule-based methods via fluency assessments, preference tests, and semantic similarity metrics. Human evaluations confirm high translation quality, with average scores of at least 6.02/7 for faithfulness, fluency, and formality. By filtering out near-identical translations, we create a challenging dataset that reveals significant performance disparities—models consistently underperform on dialectal inputs compared to Standard American English (SAE). EnDive thus advances dialect-aware NLP by uncovering model biases and promoting more equitable language technologies.

## introduction language diversity, shaped by social and cultural factors, presents significant challenges for nlp systems. while english serves as a global lingua franca, its dialects exhibit substantial variation that often goes unaddressed in language technologies @xcite . this oversight perpetuates discrimination against dialect speakers in critical domains like education and employment * lead author @xcite , exacerbated by llms' predominant focus on sae @xcite . recent studies reveal systemic biases in llm processing of non-standard dialects @xcite -from toxic speech misclassification of african american vernacular english tweets @xcite to parsing errors in chicano and jamaican english @xcite . similar issues plague indian and singaporean english due to morphological divergences @xcite , highlighting an urgent need for inclusive nlp systems @xcite . existing benchmarks like glue @xcite and superglue @xcite fail to capture dialect variation, while specialized datasets (svamp, mbpp, folio) @xcite @xcite remain sae-centric. recent large-scale efforts such as di-alectbench @xcite and aradice @xcite broaden coverage to hundreds of varieties or to arabic dialects, but they still leave cross-dialect reasoning largely unexplored. while frameworks like multi-value @xcite address dialect representation through rule-based lexical substitutions, their synthetic approach fails to capture authentic syntactic patterns. this limitation is particularly acute in reasoning tasks, where surface-level translations preserve logical meaning but lose dialect-specific pragmatic markers essential for fair evaluation. to address these gaps, we introduce endive (english diversity), a benchmark that evaluates seven llms across 12 natural language understanding (nlu) tasks translated into five underrepresented dialects selected for their linguistic distinctiveness and sociocultural significance: • african american vernacular english (aave): 33m speakers with distinct syn-tax/phonology (lippi-green, 1997) • indian english (inde): 250m speakers blending local/colonial influences (kachru, 1983) • jamaican english (jame): diaspora language with mesolectal variation (patrick, 1999) • chicano english (chce): spanish-influenced variety in us hispanic communities @xcite • colloquial singaporean english (collsge): multicultural creole with asian substrates @xcite our methodology combines linguistic authenticity with strategic filtering to create robust dialect evaluations. using verified text samples in the target dialects from ewave @xcite for few-shot prompting, we translate sae datasets into target dialects while preserving sociolinguistic nuance. to eliminate superficial transformations, we apply bleu-based filtering @xcite , removing translations with @xmath0.7 against their sae sources-retaining only substantive linguistic variations that challenge llms' dialect understanding. we compare our translations against multi-value's rule-based translations @xcite through fluency assessments, semantic similarity metrics, and llm preference tests. additionally, we have native speakers assess our translations to ensure linguistic authenticity and original content meaning are preserved across all five dialects. ## results and discussion we evaluate seven sota llms across five dialects and twelve tasks: gemini 2.5 pro (deep-mind, 2025), o1 (openai, 2024b), claude 3.5 sonnet @xcite , gpt-4o and gpt-4omini (openai, 2024a), deepseek-v3 (deepseek-ai, 2024), and llama-3-8b instruct (meta, 2024). as shown in performance gaps between sae and dialectal inputs. all seven models demonstrate consistent performance drops when evaluated on dialectal inputs compared to sae prompts. the average gap ranges from approximately 2.69% to 12.37%, with smaller models like gpt-4o-mini showing the steepest decline-most notably a 12.37% drop on inde (74.26% cot vs. 86.63% sae cot). in dialectal question sae equivalent gold predicted model(s) the committee done approved the bill, right? did the committee approve the bill? yes no claude 3.5 sonnet, o1 ain't no one gone to that party, huh? did anyone go to that party? no yes claude 3.5 sonnet, llama-3-8b instruct she been had that promotion, right? has she had that promotion? yes no gpt-4o-mini he don't work on mondays, right? does he work on mondays? no yes deepseek-v3, claude 3.5 sonnet reasoning-only models lead, but gaps remain. o1 and gemini 2.5 pro, both reasoningnative models, deliver the strongest performance across all dialects and prompting conditions. o1 consistently scores above 88.5% on dialectal inputs and surpasses 93% under sae cot prompting. gemini 2.5 pro shows similar strength, with dialectal scores ranging from 88.70% to 89.72% and sae cot scores consistently above 92%. despite their robust performance, both models still exhibit performance gaps of 3-5 points between dialectal and sae inputs. this indicates that architectural advances and reasoning capabilities alone are insufficient to fully close the dialect gap. mid-tier models show mixed robustness. gpt-4o, deepseek-v3, and claude 3.5 sonnet form a middle tier in performance, generally scoring between 79%-83% on dialectal inputs. claude 3.5 sonnet has scores ranging from 79.61% on inde to 81.15% on collsge, and performance gaps exceeding 9 points in multiple dialects. these patterns suggest that while mid-sized models benefit from cot prompting, they remain vulnerable to dialectal shifts, especially in settings involving complex linguistic variation. smaller models are more sensitive to dialect shift. gpt-4o-mini and llama-3-8b instruct, the two smallest models in the benchmark, consistently yield lower accuracies and exhibit wider gaps between dialectal and sae inputs. gpt-4o-mini records 74.53% on aave compared to 78.27% with sae cot, while llama-3-8b instruct shows similarly notable drops-particularly on chce and collsge. these findings underscore the disproportionate impact of dialectal variation on smaller or less instruction-tuned models. dialectal disparities persist across model tiers. despite strong aggregate performance from top models, consistent accuracy gaps across dialects reveal a clear and persistent issue: models struggle to generalize beyond standard american english. mid-sized and smaller systems are especially vulnerable, but even the best-performing models exhibit measurable degradation across dialects. this points to a central conclusion-current language models, regardless of scale, exhibit dialectal bias. full per-task results are provided in appendix e. ## conclusion this paper introduces endive, a benchmark for evaluating llms on dialectal robustness across 12 diverse nlp tasks and five underrepresented english dialects. our results show that llms consistently underperform on non-standard dialects compared to sae, highlighting significant unfairness and limitations in current language technologies. moving forward, we aim to expand endive to additional dialects and refine translation methodologies to further bridge the gap in dialect-aware nlp. by establishing this benchmark, we encourage future research into fairer, more robust intralanguage technologies that serve all linguistic communities. all code and datasets are available at endiveee.github.io. furthermore, we tested seven widely used llms. however, given the rapid pace of development in the field, it is infeasible to evaluate every emerging model. we hope endive will serve as a resource for future studies examining fairness and robustness across a broader range of llms as they emerge. we also acknowledge that while gpt-4o was used to generate dialectal translations, it underperforms on dialectal reasoning tasks. nonetheless, we distinguish between generation and comprehension: gpt-4o, when prompted with verified fewshot examples, produces fluent and faithful outputs, as validated by native speakers (section 4.7). still, llm-generated text may not fully capture deeper linguistic and cultural nuances. comparing these translations to naturally occurring dialect corpora would further strengthen our evaluation of realism. we faced limitations with bleu score filtering as well. for chce, the number of remaining translations was extremely low because multi-value struggled to generate diverse translations and many were further filtered out due to bleu score thresholds. as a result, there were too few data points to evaluate chce translations against multi-value. a similar issue arose with humaneval for aave and collsge, where limited translations prevented reliable evaluation of metrics for these dialects. finally, while our results highlight significant performance disparities in dialectal inputs, this study does not deeply investigate the underlying causes of these discrepancies or propose direct mitigation strategies. understanding these biases and developing equitable nlp solutions remain important areas for future research. despite these limitations, we believe endive provides a valuable framework for advancing dialect-aware nlp evaluation.