Linguistic Bias in C hat GPT : Language Models Reinforce Dialect Discrimination (2024)

Abstract: We present a large-scale study of linguistic bias exhibited by ChatGPT covering ten dialects of English (Standard American English, Standard British English, and eight widely spoken non-”standard” varieties from around the world). We prompted GPT-3.5 Turbo and GPT-4 with text by native speakers of each variety and analyzed the responses via detailed linguistic feature annotation and native speaker evaluation. We find that the models default to “standard” varieties of English; based on evaluation by native speakers, we also find that model responses to non-”standard” varieties consistently exhibit a range of issues: stereotyping (19% worse than for “standard” varieties), demeaning content (25% worse), lack of comprehension (9% worse), and condescending responses (15% worse). Moreover, if these models are asked to imitate the writing style of prompts in non-”standard” varieties, they produce text that exhibits lower comprehension of the input and is especially prone to stereotyping. GPT-4 improves on GPT-3.5 in terms of comprehension, warmth, and friendliness, but also exhibits a marked increase in stereotyping (+18%). The results indicate that GPT-3.5 Turbo and GPT-4 can perpetuate linguistic discrimination toward speakers of non-”standard” varieties.

## introduction popular tools powered by language models, such as chatgpt, can exhibit harms towards marginalized groups, including stereotyping and worse performance. a growing area of research has examined harms on the basis of dialect bias-difficulties faced by speakers of dialects, or language varieties, that have fewer speakers or are stigmatized as nonstandard. given the vast numbers of people who speak varieties of english other than standard american english (sae), the variety typically produced by chatgpt, we examine how chatgpt performs for speakers of minoritized (or non-"standard") varieties of english. our work addresses two central questions. first, how does the behavior of chatgpt differ in response to different varieties of english? second, what harms (if any) do chatgpt responses exhibit toward speakers of minoritized varieties of english, such as perpetuating stereotypes? because standard varieties of english, particularly sae, dominate available training data and are prioritized in research and industry contexts, we hypothesized that "standard" varieties of english would be treated as the default and receive innocuous responses. by contrast, we hypothesized that models would produce potentially harmful outputs in response to minoritized varieties. we prompted both gpt-3.5 turbo and gpt-4 with text in ten varieties of english: two standard varieties, sae and standard british english (sbe); and eight minoritized varieties: african american english (aae), indian english, irish english, jamaican english, kenyan english, nigerian english, scottish english, and singaporean english. 1 first, to understand whether chatgpt imitates features of input varieties, we annotated the responses to each variety for a set of paradigmatic linguistic features of that variety (section 4). then, to understand whether speakers of minoritized varieties experience differences in response quality or potential harms when using language models, we surveyed native speakers of each variety for multiple qualities of the generated outputs (section 5). in our first study, we find that distinctive linguistic features are reduced in responses to all minoritized language varieties, while responses to sae and sbe retain the most features by a considerable margin. for minoritized varieties, feature retention appears to correlate with speaker population size. in our second study, we find that model responses to minoritized varieties are perceived as more stereotyping, demeaning, unnatural, and condescending; and less able to comprehend the input. we also find that when gpt-3.5 is prompted to imitate the input dialect, its responses exacerbate stereotyping content and lack of comprehension. gpt-4 responses imitating the input improve on gpt-3.5 in terms of warmth, comprehension, and friendliness, but further exacerbate stereotyping. given chatgpt's presumed excellent performance on english, understanding performance discrepancies for varieties of english that already face stigma is critical. discrepancies that limit language models' ease of use for minoritized populations could exacerbate existing global inequities. meanwhile, advancement of limiting stereotypes and other harms could discourage speakers of minoritized varieties from using language models and reinforce discriminatory perspectives. ## related work languages typically exhibit wide variation associated with speakers from different regions, social groups, or identities @xcite ; eckert and rick-1 "standard" language is an "abstracted, idealized, homogeneous spoken language...imposed from above" and modeled on "the written language" (lippi- @xcite . "standard" language is not actively spoken by any real community; moreover, all language varieties have more and less "standard" versions. we use "standard varieties" to refer to standard american english and standard british english because they have by far the most global prestige and influence. we use "minoritized varieties" for the other eight varieties tested. @xcite . speakers of language varieties that do not enjoy status as a "standard" dialect face discrimination across settings including housing, employment, education, and criminal justice @xcite @xcite . dialect discrimination often serves as a proxy for other forms of discrimination, such as racism, classism, and xenophobia @xcite . issues of linguistic discrimination in natural language processing (nlp) have raised increasing concern. research on english is the status quo @xcite ; even within english, prioritizing standard varieties could result in differential performance and opportunity allocation, as well as linguistic profiling @xcite . previous work has explored some dialect biases in language models. this research has largely focused on aae, for which studies have found evidence of bias in hate speech detection @xcite , language identification (blodgett et al., 2018), speech recognition @xcite @xcite @xcite , and text generation @xcite . hofmann et al. (2024) also find that language models exhibit harmful stereotypes about aae speakers in hypothetical decisions, such as employment and criminal conviction. on synthetic data for several varieties, ziems et al. (2023) find disparities on common nlp tasks such as semantic parsing. on other varieties of english, yong et al. (2023) find mixed results for generation of code-mixed southeast asian dialects and ryan et al. (2024) find disparities on a dialog intent prediction task for indian and nigerian english speakers. our research aimed to address several gaps in the existing literature. to address that most research has focused on aae or synthetic data, we studied responses to native speaker-authored text in a largescale study of ten widely spoken varieties of english globally. in addition, we aimed to understand how harms affect native speakers in the increasingly common setting of casual interaction with a language model such as chatgpt. to do so, we had native speakers evaluate open-ended gpt-3.5 turbo and gpt-4 responses to text in the varieties they speak. to complement previous work based on automatic evaluation metrics, we recruited native speakers for our evaluation. these annotators rated the responses along multiple axes and gave free-text feedback on their experiences to provide a richer understanding of native speaker perspectives. ## approach we selected ten varieties of english (aae, indian english, irish english, jamaican english, kenyan english, nigerian english, sbe, scottish english, singaporean english, and sae) based on factors including first and second language speaker population counts, availability of linguistic literature on the varieties, geographic spread, and sociohistorical context. we aimed to include varieties with larger speaker populations and enough linguistic description to determine distinctive features for each variety. finally, we ensured that the varieties chosen have a sufficient geographic spread and reflect different socio-historical contexts by which english came to be spoken in a particular area. english language data was collected from several sources (appendix d). nigerian, jamaican, indian, irish, and kenyan english data was drawn from the international corpus of english (ice) @xcite . for each of these varieties, we chose to only analyze social letters in order to mimic the informal tone and style of text that users would use in dialogue with language models. sae and sbe were sourced from reddit posts on us and uk cities' subreddits, respectively @xcite ). foot_0 aae was sourced from @xcite . scottish english data was drawn from the correspondence and letters subset of the scots corpus @xcite . singaporean english data was sourced from the text messages in the cosem corpus @xcite . ## study 1: assessing linguistic features of default responses for our first study, we conducted evaluations to test the following hypotheses: (1) that chatgpt responses will exhibit reductions in the features of all varieties tested except sae; and (2) that chatgpt responses will have increased american orthography. we selected ten of the most prominent features of each variety based on existing linguistic descriptions of each variety, focusing on the morphosyntactic features (word-and sentencelevel features that can be observed in written data) that existing documentation deems particularly distinctive (annotation guides in appendix b). these features range from distinctive lexical items (e.g. @xmath0.g. lack of subject-verb inversion in yes-no questions in indian english). we also identified the orthography of the output: american, british, or either (no distinctive features found). the focus on american and british orthography stems from the socio-historical context of english colonization in the british isles, africa, americas, and asia, and the united states' expanding sphere of influence and colonization efforts in the pacific, which has led to most english language communities adopting the orthography of britain or the us (e.g., analyse/analyze, favour/favor). as with the linguistic features discussed above, distinctive orthographic features were determined based on existing linguistic description. for each variety, we sampled approximately 50 messages foot_1 to prompt gpt-3.5 turbo via the ope-nai api. the inputs provided to the model focus on benign topics related to daily life (e.g. updates about how the author is doing, travel recommendations for particular areas, etc.). the system prompt (appendix c) encouraged the model to respond directly to the letter. two reviewers from our research team independently assessed each (input, output) pair for the ten selected distinctive features of the variety, in addition to the orthography (krippendorff's α = 0.97). we averaged results for the two reviewers per variety to conduct our evaluations. ## study 2: native speaker evaluation of output disparities our second study explored to what extent chat-gpt outputs might perpetuate harms in response to speakers of minoritized language varieties. our analyses aimed to answer three questions: for this study, after completing the annotation process, we selected all of the input letters for each language variety for which the input or output contained at least one feature. then, these selected input letters were fed into gpt-3.5 and gpt-4 with a new system prompt that instructs the model to attempt to match the style and tone of the letter in its response. native speakers of each variety were then recruited to evaluate the responses via prolific (see appendix d for details on survey format, filtering, consent, and compensation). each annotator completed a survey consisting of twelve input-output pairs in random order, with six outputs from gpt-3.5 (prompted simply to respond); three from gpt-3.5 (prompted to imitate the input); and three from gpt-4 (prompted to imitate the input). 6 this resulted in 879 total examples annotated (mean of 87.9 per variety). for each (input, output) pair, annotators assessed the output on 5-point likert scales for nine qualities: stereotyping, demeaning content, condescension, formality, comprehension, naturalness, warmth, friendliness, and respect. we included a short reflection at the end asking speakers to give open-ended feedback as well. we also asked annotators to optionally provide demographic and background information to ensure we incorporated diversity among our participants, account for common linguistic variation along demographic dimensions, and ensure participant familiarity with the english variety being examined. ## discussion & conclusion our research illustrates differences in gpt-3.5 and gpt-4 responses across varieties of english. gpt-3.5 retains features of sae and sbe more than features of minoritized varieties of english. the distinctive features that are retained tend to be lexical items or features that are grammatical in sae. for minoritized varieties, feature retention rate correlates with estimated maximum speaker population, which may reflect available training data. model responses can also fail to adequately serve speakers of minoritized varieties through increased stereotyping, demeaning content, condescension, and lack of comprehension. gpt-3.5 responses that attempt to imitate the input further exacerbate stereotyping and lack of comprehension. exceptions to this trend highlight places where model quality has improved: gpt-4 responses imitating the input tend to improve on imitative gpt-3.5 responses in terms of comprehension, warmth, and friendliness. however, gpt-4 responses exhibit even higher levels of stereotyping, suggesting that reducing stereotyping content in response to minoritized varieties is of particular concern. disparities in output quality for speakers of minoritized varieties may hamper their ability to use language models; furthermore, harmful responses can perpetuate discriminatory ideologies. as language model usage increases globally, these tools risk reinforcing power dynamics that harm minoritized language communities.