Analyzing Dialectical Biases in LLM s for Knowledge and Reasoning Benchmarks (2025)

Abstract: Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying “standard” American English language questions as non-”standard” dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-”standard” English questions. We find that individual grammatical rules have varied effects on performance, but some are more consequential than others: three specific grammar rules (existential “it”, zero copula, and y’all) can explain the majority of performance degradation observed in multiple dialects. We call for future work to investigate bias mitigation methods focused on individual, high-impact grammatical structures.

## introduction large language models are essential to natural language processing applications, achieving strong performance across numerous tasks. however, language model learning is highly sensitive to the style of the data used in training @xcite . this can lead to fairness issues for speakers of non-"standard" varieties of american english (such as african american english, chicano english, and non-native english speakers), who may write text in their corresponding spoken dialects @xcite @xcite @xcite ; in turn, these written variants are likely underrepresented in training corpora relative to "standard american english" (sae). these are particularly important english-speaking populations to study as they have already been found to suffer from llm underperformance in benchmark tasks @xcite , and are often correspondingly minoritized in broader societal contexts. an example of differential validity on such tasks is if llms disproportionately respond with inaccurate responses to prompts written in african american english, but respond correctly to prompts written in sae. this is a real concern, as llms are increasingly involved in highstakes scenarios from education to hiring. while llm underperformance of individual varieties of english has been studied at a high level in real-world conversational contexts @xcite , it remains understudied the extent to which (a) underperformance for different english dialects is an issue in more basic nlp tasks (i.e., multiple choice rather than open response questions), and (b) grammatical rules defining english dialects might be drivers of llm response differences. these questions have historically been difficult to answer because one would need to find corpora of comparable text typified in different dialects; however, prior work automating dialectal translation @xcite allows us to generate text data in different dialects based on grammatical rules. we apply this translational tool to fill a gap in the literature by answering two research questions: rq1: do llms underperform when answering multiple choice questions that are typed in a written dialect (african american english, appalachian english, chicano english, indian english, singaporean english, and southern english) versus answering questions typed in sae? rq2: can llm underperformance in certain dialects be decomposed into underperformance stemming from multiple specific grammatical rules? answering these questions is important: not only can we quantify biases in fundamental llm tasks, but we can further break down these quantities by grammatical rules, which can inform model developers of whether these rules should be a focus of improvement for multidialectal llms, thereby helping to drive mitigation of identified biases. prior work addressing these questions has either focused on linguistic analyses of model underperformance for individual grammatical rules, such as the habitual be (martin and tang, 2020) or zero copula @xcite common to african american english @xcite , or has focused on studying biases in overall dialects without considering individual grammatical rules @xcite @xcite . in contrast, we study a wide range of grammatical rules used across multiple dialects, which can in turn be used to inform model improvements across multiple dialects: given the high number of shared or similar grammatical features across dialects, we may expect that technical improvements on specific grammatical rules can yield performance improvements across dialects through transfer learning. ## methods we begin with three multiple choice benchmark question answering (qa) datasets commonly used for benchmarking: boolq @xcite containing 9,427 real google user queries, sciq @xcite containing 11,679 science exam questions, and mmlu @xcite containing 14,042 questions spanning 57 subtopics from accounting to religion. all datasets contain both questions and multiple choice answers in "standard american english" (sae). we opt to focus on multiple choice questions as we expect these tasks to be relatively easy for llms in sae; uncovering dialectal underperformance in these tasks could indicate the extent to which dialectal biases persist across not just difficult-but also prima facie easier-tasks, and can potentially serve as a lower bound on quality-of-service harms across llm question answering tasks. next, we generate grammatically-perturbed variants of each question in these datasets-leaving answers and support material unchanged-by using the multi-value package @xcite . multi-value is based on the electronic world atlas of varieties of english (ewave), a linguistic database of morphosyntactic variation in spontaneous spoken english @xcite , and allows users to input sae text and generate dialectal variants based on sets of grammatical rules from ewave. the creators of multi-value additionally recruited speakers of several dialects (includ-ing african american english, chicano english, indian english, and appalachian english) to validate multi-value dialectal translations @xcite . this package allows us to perturb sae text (a) by individual grammar rules, (b) by multiple grammar rules of our choosing, and (c) using the default set of grammar rules ascribed to specific dialects (e.g. appalachian, singaporean, etc.). for example, for the sentence "she is always studying," we could apply the specific zero copula grammatical rule to obtain the string "she always studying," or apply all rules for the african american english dialect (in this case, both zero copula and habitual be) to obtain the string "she always be studying." multi-value applies rules probabilistically based on documented dialect pervasiveness, with uncommon dialectal grammar rules up-weighted for stress-testing @xcite . for the six dialects of interest in our study, we use the multi-value default grammatical rules and transformation frequencies when generating dialectal perturbations. for individual and groups of grammar rules, we set the transformation frequencies to 100% such that grammar rule transformations are always applied when the corresponding grammatical structure appears. we perturb only question texts and not reference or answer texts. we include examples of individual rule transformations in we then compare the performance of three common llms on both original (sae) and grammatically-perturbed variants of the three qa datasets. we choose to use gemma-2b, mistral 7b, and gpt4o-mini due to their popularity in realworld applications, and to encapsulate a range of model sizes in our evaluations. * we used the default prompts in lm eval harness @xcite available for these datasets in a zero-shot setting. for all three llms, we find performance on the unperturbed qa datasets to be comparable to their technical reports @xcite @xcite . we then compare performance for each of the three datasets, for each of the dialectal and grammatical variants of each dataset, and for each of the three llms. to calculate accuracy, we first ## discussion our results indicate that llms have trouble parsing certain grammatical concepts, leading to fairness concerns for non-"standard" dialects as quantified by underperformance in knowledge and reasoning benchmark tasks, consistent with findings of underperformance on more explicitly cultural-specific tasks @xcite @xcite . our results may be an underestimate of the severity of dialectal underperformance in qa tasks: recent work has found greater performance decreases for human-written aave texts relative to multi-value perturbed texts @xcite , and multiple choice responses are less variable than open-ended tasks @xcite . our findings could potentially be driven by an underrepresentation of dialectal english in training data; we hope that pinpointing specific grammatical rules associated with underperformance will allow practitioners to update models for bias mitigation going forward. more work needs to be done to explore avenues for such bias mitigation: some researchers have found that in-prompt "translating" to sae still leaves significant performance gaps @xcite , and others have proposed training loras to map hundreds of grammatical structures to sae equivalents to mitigate dialectal degradation @xcite . going forward, it will be also be important to extend our findings from multiple choice qa tasks to open-ended responses, especially given the increasing concerns of biases in not only valid llm responses, but also in llm-generated hallucinations @xcite . given that the demographics that often use non-"standard" english have been surveyed to be more likely to rely on llms @xcite , it is especially important to ensure that there is a focus on improving llm performance across dialectsespecially from those demographics currently underserved in society and correspondingly underrepresented in training data. looking toward future work in mitigating biases, our findings suggest that practitioners and researchers building llms for multi-dialectal users could target future model improvements by training models on qa pair variants-perturbed with even just a handful of important, distinct grammar rules-to yield less disparate performance across multiple dialects.