That Ain’t Right: Assessing LLM Performance on QA in A frican A merican and W est A frican E nglish Dialects (2025)

Abstract: As Large Language Models (LLMs) gain mainstream public usage, understanding how users interact with them becomes increasingly important. Limited variety in training data raises concerns about LLM reliability across different language inputs. To explore this, we test several LLMs using functionally equivalent prompts expressed in different English sublanguages. We frame this analysis using Question-Answer (QA) pairs, which allow us to detect and evaluate appropriate and anomalous model behavior. We contribute a cross-LLM testing method and a new QA dataset translated into AAVE and WAPE variants. Early results reveal a notable drop in accuracy for one sublanguage relative to the baseline.

## introduction large language models (llms) are increasingly embedded in daily life, assisting users with both professional and personal tasks. despite global use, llms are trained primarily on english-over 90% of which is standard american english (sae) @xcite -resulting in potential mismatches with user inputs @xcite . popular llms largely train on sae, with only about 7% of training data coming from other languages @xcite . limited geographic variation can lead to misunderstanding, or hallucinations when users employ english dialects that deviate from sae. consequently, llms do not perform equally well across speakers of different dialects. these dialects differ in vocabulary, grammar, and pronunciation, often shaped by culture. over 30 major english dialects are spoken regularly in the u.s., and over 150 are spoken worldwide @xcite . african american vernacular english (aave) and west african pidgin english (wape) are two major dialects spoken by millions globally. however, they are rarely included in llm training this study focuses on question answering (qa) as a focused task for evaluating how well llms respond to prompts written in aave and wape, compared to sae. by translating a qa dataset into these dialects and analyzing model responses, we aim to assess if llm performance degrades with dialect input, ensuring consistent behavior across different forms of english. by testing llm performance on aave and wape, this study highlights where model adjustments may be needed in order to support broader consistency across a wider range of users. ## background and related work this study examines discrepancies in llm performance with aave and wape, building on emerging work that probes the consistency and coverage of llms in non-standard dialects of english. prior research informs our approach by demonstrating how others evaluate llms on different dialects, guiding how we think about achieving more consistent behavior across a broader range of inputs. ## methodology dialects express similar intents in ways llms may interpret inconsistently. we simulate this variation by creating equivalent prompts (see figure 1 ) and comparing performance against sae, the baseline given its dominance in training data. ## results a fleiss' kappa of 0.889 reflects strong inter-rater agreement. this provides greater confidence in the results for the evaluations that were not shared among raters. ## discussion the following observations arise from notable llm behaviors in response to the provided prompts. these patterns suggest directions for future experimentation and deeper analysis. ## conclusion and future work this study explores how llms respond to prompts written in aave and wape compared to sae. by translating a qa dataset written in sae into aave and wape and evaluating llm performance, we assess how accuracy varies across dialects. results reveal notable performance gaps-particularly wape-indicating discrepancies in how llms handle non-standard varieties of english. given widespread llm usage, these findings matter, as many users communicate in dialects not well reflected in training data. when llms struggle with these variations, they risk misunderstanding entire user communities. improving performance on dialects like aave and wape strengthens model generalization across varied inputs. several directions for future work emerge from this study's limitations and findings. one priority is to expand the set of dialects and regional varieties tested, which would help determine whether the current findings are generalizable. another area for exploration involves distinguishing between a model's understanding of a prompt and the correctness of its response, treating comprehension and accuracy as separate parameters. in some cases, llms misunderstand a prompt and respond incorrectly. in others, they appear to grasp the core the meaning of a prompt but still generate an incorrect answer. this distinction is not captured by current metrics, so future work will include an additional evaluation layer to track these two forms of hallucination.