My LLM might Mimic AAE - But When Should It? (2025)

Abstract: We examine the representation of African American English (AAE) in large language models (LLMs), exploring (a) the perceptions Black Americans have of how effective these technologies are at producing authentic AAE, and (b) in what contexts Black Americans find this desirable. Through both a survey of Black Americans ( n= 104) and annotation of LLM-produced AAE by Black Americans ( n= 228), we find that Black Americans favor choice and autonomy in determining when AAE is appropriate in LLM output. They tend to prefer that LLMs default to communicating in Mainstream U.S. English in formal settings, with greater interest in AAE production in less formal settings. When LLMs were appropriately prompted and provided in context examples, our participants found their outputs to have a level of AAE authenticity on par with transcripts of Black American speech. Select code and data for our project can be found here: https://github.com/smelliecat/AAEMime.git

## introduction in our study, we explore how an underserved population, black americans in the united states, regards increasingly ubiquitous text-based ai tools in terms of their preferred functionalities and with respect to the authenticity of the language produced by these systems, given their unique needs. we specifically investigate the research questions: the black american population makes up approximately 13.6% of the united states total population in 2022 (u.s. census bureau, 2022; @xcite and represents a major stakeholder population of text-based ai technologies. aae, while predominantly a spoken language variety, is seeing increased representation in speech-like media such as texting and social media @xcite . its use has become synonymous with the cultural identity of some black americans @xcite with the language evolving over an extended period of time dating as far back as the period of black enslavement in the united states. in spite of the cultural importance of aae, black americans have had good reason to be hesitant to use the language outside of personal contexts due to widespread linguistic discrimination: racial identification and discrimination based on speech or writing in the work place and in schools @xcite . in addition, the use of aae may be associated with poverty or lower socioeconomic class @xcite , which could influence black americans to be cautious about the circumstances under which it should be used. aae is predominantly identified by grammatical patterns such as the use of double negatives, variable subject-verb agreement, and omission of verbal copulas. these patterns distinguish it sharply from mainstream u.s. english (muse) 3 . despite its distinct linguistic characteristics and the large proportion (80-90%) of the black american society that speaks aae in the united states @xcite @xcite , prior studies have shown that ai technologies often fail to accommodate its nuances, especially in the context of speech recognition ( §2). in this paper, we investigate the research questions listed above in the context of large language models. we conduct an online study ( §3) among black americans (@xmath0. in the survey ( §3.1), we aim to understand in what social contexts black americans would want (the option of) having an llm use aae, including both professional and personal settings, and including both continuation behavior (e.g., email autocomplete) and reply behavior (e.g., ai assistants). in the data annotation task ( §3.2), we ask black americans (@xmath1,654 judgments for 1,357 examples) to judge text generated by three different llms-gpt 4o-mini, llama 3, and mixtral-along axes like coherence, the explicit presence of aae features, and offensiveness. our main contributions ( §4) include: attitudes and perceptions of aae speakers towards technology @xcite examine the invisible labor aae speakers undertake to be understood by language technologies. their findings show that aae speakers often have to proactively adapt their speech, which leads to significant frustration and alienation when interacting with these systems. @xcite further discuss how black older adults anticipate and experience substantial challenges with voice assistants, exacerbating their reluctance to use these devices for tasks like health information seeking. our study explores the idea that black americans may prefer aae options in their interactions with speech or language-based ai technologies. we build on these lines of research by exploring black americans' preferences for aae representation in specific contextual settings. ## methodology we investigate two primary research questions: 1) whether black americans want generative ai technologies to communicate and understand african american english and in which contexts, and 2) how effective llms are at generating aae when prompted to do so. our institutional review board (irb)-approved 4 online study gathers participant feedback on these questions, via a survey on participants' desires ( §3.1), and data annotation to understand the effectiveness of current llms ( §3.2). for both, we recruit participants through prolific (prolific.co), who were required to be black american over 18 years old, familiar with aae, and reside in the u.s. in total, of our @xmath2. most other irb details redacted for anonymity. ## results we outline the results from our study below. we review our findings from our scenarios-based survey, which explored black american perspectives and expectations for interactions with language-based generative ai tools, looking at a range of personal and professional settings. we then review the results from the data annotation effort, whereby we gathered black americans' judgments of the effectiveness of large language models at production of aae and muse relative to our human baselines of black american transcribed interviews -coraal, tweets, and npr interviews. ## conclusion in this study, we explore how black americans perceive the appropriate use of aae pertaining to language-model-based technologies, especially in terms of their ability to authentically represent aae. we consider both the expectations of the community and the actual output of current systems, exploring the idea that aae should not merely be an option but a well-integrated feature. overall, the llms we reviewed were surprisingly capable and comparable in how they were perceived in terms of their ability to produce authentic aae in comparison to the transcribed speech of black americans from the coraal corpus. we found that the text completions by some llms were often perceived as more aae-heavy, or sounding more like something a black american would say, than our the language in our human aae cor-pus. if our human aae baseline is assumed to have the "right amount" of aae, then having more linguistic features of aae than the baseline could be considered to be excessive by some aae speakers, whereas less may be an insufficient amount of aae. at the same time, because the human aae is a transcript, it may not be fully reflective of all ways that aae is used in practice. llms are either slightly under-doing or over-doing aae, but on average, participants generally disagreed that the machine-generated text by the llms we studied was offensive to or mocking of black americans. for the scenarios-based questions of our survey, our findings reveal intricate preferences for ai applications across diverse environments, indicative of broader societal shifts toward technologies that are both personalized and context-sensitive. these insights are pivotal for developers and policymakers tasked with refining ai tech to align more closely with user expectations, thereby facilitating smoother integration of ai into everyday life. future work our study highlights that black americans prefer, at a minimum, the option for communication in aae with popular languagebased generative ai tools and generally deem llm-generated aae as credible and similar to spoken aae. in light of this finding, we encourage the technology community to expand the linguistic diversity in language-based generative ai tools; in particular, they should consider functionality that provides the autonomy to black americans to switch to aae on demand in circumstances of their choosing, as well as have an array of multimodal options for aae communications, including but not limited to generation and understanding of aae text as well as audio communications, including speech recognition and production. more generally, technological support for alternative dialects or sociolects in generative ai systems will make these systems more broadly accepted and equitable for a range of important stakeholder populations. finally, we encourage future research on the relationship between 1) the diverse attributes that characterize black americans (whether they be regional, cultural, socioeconomic or demographic) and 2) whether and how they express aae personally, or their preferences for its production by generative ai tools across different contexts.