Understanding the Impacts of Language Technologies’ Performance Disparities on A frican A merican Language Speakers (2024)

Abstract: This paper examines the experiences of African American Language (AAL) speakers when using language technologies. Previous work has used quantitative methods to uncover performance disparities between AAL speakers and White Mainstream English speakers when using language technologies, but has not sought to understand the impacts of these performance disparities on AAL speakers. Through interviews with 19 AAL speakers, we focus on understanding such impacts in a contextualized and human-centered manner. We find that AAL speakers often undertake invisible labor of adapting their speech patterns to successfully use language technologies, and they make connections between failures of language technologies for AAL speakers and a lack of inclusion of AAL speakers in language technology design processes and datasets. Our findings suggest that NLP researchers and practitioners should invest in developing contextualized and human-centered evaluations of language technologies that seek to understand the impacts of performance disparities on speakers of underrepresented languages and language varieties.

## introduction prompted by the widespread availability of language technologies, researchers have questioned whether these technologies are sufficiently inclusive of speakers of underrepresented languages and language varieties (e.g., @xcite @xcite @xcite . most notably, previous work has found that language technologies often exhibit performance disparities between african american language (aal) speakers and white mainstream english (wme) speak-ers @xcite . 1 much of this work-as well as work on performance disparities between other social groups when using language technologies-has used quantitative methods to uncover the existence and extent of performance disparities (e.g., @xcite @xcite . however, for the most part, this work has not sought to understand the impacts of these performance disparities on aal speakers. for example, although @xcite wrote that "the performance gaps we have documented suggest it is considerably harder for african americans to benefit from the increasingly widespread use of speech recognition technology... these disparities may also actively harm african american communities...," they did not examine the experiences of aal speakers when using language technologies. in contrast, our paper focuses on understanding these experiences in a contextualized and human-centered manner, illuminating the relationships between linguistic discrimination and failures of language technologies. we take a qualitative research approach to demonstrate the need for contextualized and human-centered evaluations of language technologies that seek to understand the impacts of performance disparities on speakers of underrepresented languages and language varieties. specifically, we conduct interviews with 19 aal speakers to examine their experiences when using language technologies, asking, what fairness-related harms might aal speakers encounter when using language technologies? we find that aal speakers often undertake invisible labor of adapting their speech patterns to successfully use language technologies (section 4.1), and they make connections between failures of language technologies for aal speakers and a lack of inclusion of aal speakers in language technology design processes and datasets (section 4.2). we emphasize that although quantitative methods can uncover the existence and extent of performance disparities, they do not help us understand the impacts of these performance disparities. to understand such impacts, qualitative methods are a better fit because they can be used to probe deeper into issues around social inequalities and illuminate the relationships between linguistic discrimination and failures of language technologies. ## findings our findings shed light on the experiences of aal speakers when using language technologies, in turn helping us understand the impacts of performance disparities on speakers of underrepresented languages and language varieties. first, we find that aal speakers often undertake invisible labor of adapting their speech patterns to successfully use language technologies. second, we find that aal speakers make connections between failures of language technologies for aal speakers and a lack of inclusion of aal speakers in language technology design processes and datasets. in section 5, we unpack the implications of these findings for nlp researchers and practitioners. ## discussion although previous work has focused on identifying, measuring, and mitigating a variety of fairnessrelated harms caused by language technologies, there remains ample room for further examination of the experiences of aal speakers when using language technologies. for example, much of the work to date on performance disparities between aal speakers and wme speakers when using language technologies has used quantitative methods to uncover the existence and extent of such performance disparities (e.g., @xcite , but has not sought to understand their impacts on aal speakers. this narrow focus limits what nlp researchers and practitioners are able to understand about the experiences of speakers of underrepresented languages and language varieties. in contrast, qualitative methods can be used to probe deeper into issues around social inequalities and illuminate the relationships between linguistic discrimination and failures of language technologies. our findings offer insights into the impacts of performance disparities on aal speakers, including invisible labor they undertake to adapt their speech patterns to successfully use language technologies, as well as connections they make between failures of language technologies for aal speakers and a lack of inclusion of aal speakers in language technology design processes and datasets. these impacts are not made visible by quantitative methods for uncovering the existence and extent of performance disparities. we therefore urge nlp researchers and practitioners to invest in developing contextualized and human-centered evaluations of language technologies that seek to understand the impacts of performance disparities on speakers of underrepresented languages and language varieties. these evaluations could use qualitative methods, but they could also also use quantitative methods, albeit in new ways. for example, an evaluation could measure the number of times users correct language technologies' outputs or manually enter inputs instead of using asr technologies. alternatively, an evaluation could measure how often users code-switch to wme. we also urge nlp researchers and practitioners to prioritize improving language technologies for speakers of underrepresented languages and language varieties, so they do not feel excluded and need to undertake invisible labor. at a minimum, language technology developers should seek ways to involve such speakers in their design processes.