Modeling Gender and Dialect Bias in Automatic Speech Recognition (2024)

Abstract: Dialect and gender-based biases have become an area of concern in language-dependent AI systemsincluding around automatic speech recognition (ASR) which processes speech audio into text. These potential biases raise concern for discriminatory outcomes with AI systems depending on demographic- particularly gender discrimination against women, and racial discrimination against minorities with ethnic or cultural English dialects.As such we aim to evaluate the performance of ASR systems across different genders and across dialects of English. Concretely, we take a deep dive of the performance of ASR systems on men and women across four US-based English dialects: Standard American English (SAE), African American Vernacular English (AAVE), Chicano English, and Spanglish. To do this, we construct a labeled dataset of 13 hours of podcast audio, transcribed by speakers of the represented dialects. We then evaluate zero-shot performance of different automatic speech recognition models on our dataset, and further finetune models to better understand how finetuning can impact performance. Our work fills the gap of investigating possible gender disparities within underrepresented dialects.

## introduction multiple fields within computing such as human-computer interaction, natural language processing (nlp), speech processing, and algorithmic fairness have identified the ways in which algorithmic systems produce disparate outcomes for minority groups, including on the basis of gender, race, and ethnicity. in language driven applications, such biases have been identified with respect to gender and with respect to dialects associated with minority groups @xcite . automatic speech recognition (asr), the machine learning application in which speech audio is transcribed to text, is among these applications which such biases have become an area for concern and further analysis @xcite @xcite . as a result of such biases in asr, minority dialect speakers and women may be more likely to struggle with accurate downstream applications using asr models, such as captioning @xcite and voice assistants @xcite . mitigating this discrepancy is an important step towards developing equitable technologies that work well regardless of a user's racial, ethic or gender background. in this work, we specifically focus on four english variants with specific significance within a us context: spanglish, chicano english, african american vernacular english, and standard american english. spanglish, also called engañol @xcite , is a language variety that broadly includes any combination english and spanish features such as grammar structure, innotation and words in real world conversational contexts. as english and spanish are the first and second most spoken languages in the us respectively this language variety has particular significance, especially as a community among the latinos within the country @xcite . chicano english, also known as mexican american english, is a specific english dialect originating from the mexican-american population in the us, most widely spoken in the south-west region of the country. importantly, while distinct from one-another, there is often overlap between spanglish and chicano english, and many speakers may use both, which is reflected in our data. african american vernacular english (aave), also called ebonics, african american language, african american english, or black english, is an english variety that derives from the us black population, originating from the southern region of the country. these three languages varieties, which we refer to as minority dialects throughout this work, are widely spoken among marginalized oppressed populations in the united states. to offer a better understanding of these minority dialects, example sentences and explanation of some of the unique dialect features is offered in we examine the overlap of gender and dialect by building upon prior research in algorithmic fairness that focuses on the intersection of gender and race or ethnicity @xcite , examining ethnicity using related dialects @xcite . examining the overlap of multiple protected attributes can give a richer analysis on the impact of systems of oppression and the severity of potential downstream harms of intelligent systems @xcite . prior works have identified serious concerns about how intelligent systems in other domains impact women of color @xcite finding that in many cases, gender and race based biases compound to most negatively impact women of color @xcite , but few works focusing on asr specifically analyze the impact of both race and gender on performance. in this paper, we investigate whether these issues are also applicable to the modality of speech. we investigate gender and dialect bias by sampling the existing spotify podcast dataset @xcite , and use a dialect-centered annotation process to transcribe with dialect and gender data. in this dialect-centered annotation process, we recruit minority dialect speakers to produce transcriptions capturing unique spellings, grammar patterns, and words that may not be accurately captured by those with less understanding of the specific english dialect. we then conduct zero shot evaluation of several state of the art automatic speech recognition models on the datasetwith respect to gender and dialect. we find that with respect to dialect, minority dialects consistently perform worse on speech models than standard american english, while with respect to gender, women typically perform better than men, except in the case of sae speakers, for which men perform better. our findings indicate that there is a significant discrepancy for minority dialect speakers. further, consistent with prior research showing gender discrepancies vary by language @xcite @xcite , we find that these discrepancies also differ across english dialects. we also find that finetuning improves overall performance, and we conclude our study with qualitative analysis out the model outputs, giving insight to the areas where asr models produce errors. to summarize our contributions are four-fold: (1) we study the gender and dialect bias in asr from an intersectional perspective. we (2) introduce a dataset annotated for dialect, gender, and other metadata. (3) we conduct extensive experiments to show such disparity with respect to dialect and gender, and (4) we highlight the challenges that asr systems suffer with respect to these issues. ## related works gender and dialect biases have been identified in a variety of nlp applications @xcite including machine translation @xcite , sentiment analysis (thelwall, 2018), content moderation @xcite and word embeddings @xcite . this phenomenon has also been observed with speech models and speech technologies @xcite . here, we outline prior work exploring gender bias, dialect bias, and their implications in real world systems. ## conclusion in our work we create a small dataset with diverse gender and dialect representations by using the spotify podcast dataset @xcite and our dialect-centered annotation process. we determine the word error rate of state of the art asr models on our dataset. we find that across all models, standard american english outperforms minority dialects. further we find that within minority dialects women speakers perform better, but within sae men speakers perform better, suggesting men of color, particularly minority dialect users, may be at the highest risk for inaccurate transcription within english.