Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech (2024)

Abstract: Automatic speech recognition (ASR) systems, increasingly prevalent in education, healthcare, employment, and mobile technology, face significant challenges in inclusivity, particularly for the 80 million-strong global community of people who stutter. These systems often fail to accurately interpret speech patterns deviating from typical fluency, leading to critical usability issues and misinterpretations. This study evaluates six leading ASRs, analyzing their performance on both a real-world dataset of speech samples from individuals who stutter and a synthetic dataset derived from the widely-used LibriSpeech benchmark. The synthetic dataset, uniquely designed to incorporate various stuttering events, enables an in-depth analysis of each ASR’s handling of disfluent speech. Our comprehensive assessment includes metrics such as word error rate (WER), character error rate (CER), and semantic accuracy of the transcripts. The results reveal a consistent and statistically significant accuracy bias across all ASRs against disfluent speech, manifesting in significant syntactical and semantic inaccuracies in transcriptions. These findings highlight a critical gap in current ASR technologies, underscoring the need for effective bias mitigation strategies. Addressing this bias is imperative not only to improve the technology’s usability for people who stutter but also to ensure their equitable and inclusive participation in the rapidly evolving digital landscape.

## introduction 1.background and motivation stuttering is a complex neurodevelopmental condition affecting over 80 million people worldwide @xcite . this condition is characterized by various involuntary speech disruptions or disfluencies @xcite , including repetitions of parts of words ("li-li-like this"), prolongations of sounds ("lllllike this"), and blocks (silent periods, "l-ike this"). while everyone may experience disfluencies, in people who stutter, these are more pronounced and are accompanied by a loss of control over speech @xcite . society's negative perceptions of stuttered speech often leads to discrimination and lower job satisfaction, as fluent speakers tend to be preferred in employment settings @xcite @xcite . in the post-covid-19 era, characterized by the widespread adoption of voice-activated artificial intelligence (voice ai), individuals who stutter are confronted with heightened challenges, especially in employment contexts. automatic speech recognition (asr) systems, the fundamental technology underpinning voice ai and designed to transcribe spoken language into text @xcite , have emerged as notable barriers in various aspects of daily and professional life for this demographic. the broad integration of asr technology across sectors such as education, healthcare, and employment, as well as in mobile devices and other applications @xcite , has led to the inadvertent marginalization of people who stutter. these systems frequently struggle to accurately decode speech that differs from typical patterns @xcite @xcite , a fact exemplified in fig. 1 . in stark contrast, asrs demonstrate up to 95% word accuracy when processing standard speech @xcite . the ramifications of this disparity are profound, ranging from difficulties in executing simple tasks, such as using automated phone systems for prescription refills, to experiencing bias in ai-driven job interviews @xcite @xcite . as the global voice recognition market is projected to nearly quintuple by 2030, reaching a valuation of us$59.62 billion @xcite , the prevalence of ai-powered voice technologies, bolstered by the recent success of large language models, is set to increase dramatically @xcite . this growth suggests an impending escalation in bias against individuals who stutter. leading technology companies like apple and google have begun addressing variations in speech, introducing alternative solutions such as typing commands to voice assistants @xcite . however, these measures, while helpful, fall short of addressing the inherent bias in asr accuracy. in particular, initiatives like google's project euphonia focus on collecting data for differing speech patterns, aiming to enhance asr capabilities for understanding people with speech impairments like dysarthria @xcite . however, these efforts often neglect the unique aspects of stuttered speech. given the significant variability of stuttering across different individuals and situations @xcite , this oversight reveals a critical gap. unless asrs are specifically tailored and refined to accommodate the diverse expressions of stuttering, they are likely to continue underperforming for individuals who stutter, exacerbating existing societal biases. ## related work this section provides a brief literature review of three key areas to set the stage for subsequent sections examining bias in asrs: the evolution and current state of asr models, their adaptation for people with speech differences, and the measurement of bias in ai models. ## methodology to investigate accuracy biases in contemporary asr systems against disfluent speech, we have designed two experiments, each focusing on a different speech dataset. the first dataset is well-known in the speech-language pathology (slp) community, and the second is widely used in asr mod-eling and evaluation. our objective is to compare asr performance on two versions of these datasets: one with disfluencies and another identical in content but without disfluencies. experiment 1 utilizes the fluencybank dataset (ratner and macwhinney, 2018), known for its collection of real stuttered speech samples with accompanying transcripts (detailed in sec. 3.2). we refer to this original dataset as fluencybank-y (fb-y), where "y" indicates the presence of disfluencies. these samples are transcribed using six different asr models (sec. 3.1), and the accuracy of these transcriptions is measured using wer, cer, and bertscore, a metric for semantic similarity analysis (sec. 3.4). for comparative analysis, we generate a version of this dataset with apparent instances of stuttering or disfluencies removed, named fluencybank-n (fb-n), where "n" signifies the absence of disfluencies, using textto-speech (tts) techniques (sec. 3.3). we then employ the spearman correlation coefficient test to systematically compare the performance of the asrs on both the original fb-y and the modified fb-n datasets. to complement experiment 1, experiment 2 utilizes the librispeech dataset @xcite , frequently used in training and evaluating asrs and characterized by standard, nondisfluent speech samples. we refer to this dataset as librispeech-n or ls-n to convey the absence of disfluencies. to assess the impact of disfluencies, we create a synthetic dataset by introducing controlled disfluency events into the librispeech dataset. this is achieved through a combination of audio processing and tts methods. this newly modified dataset, denoted librispeech-y or ls-y, where "y" indicates the presence of disfluencies, is then subjected to the same testing process as the fluencybank dataset in experiment 1, using the same set of six asr models. this approach ensures a systematic analysis of bias within these asr systems across diverse speaking scenarios. ## results and discussion our analysis, divided into two parts corresponding to each experiment in sec. 3, reveals significant insights into asr performance with real and synthetic disfluent speech. the evaluation was conducted using asr services as of march. 14, 2024, utilizing default api settings. for wav2vec 2.0, the analysis was performed locally using an nvidia t4 gpu. transcripts returned blank by models due to audio processing challenges (e.g., due to noise or blocks) were retained in our dataset. prior to evaluation, all transcripts underwent normalization using whisper's "basictextnormalizer" and "en-glishnumbernormalizer" functions from the transformers library (huggingface), ensuring consistent casing and punctuation removal. any speech atmospherics in output transcripts, such as laughing or indications of silence, were also removed. ## conclusion this study represents a significant step in identifying and quantifying the biases in asr systems against individuals who stutter. by analyzing the performance of six popular asr models using two distinct datasets-fluencybank and a customgenerated synthetic dataset incorporating disfluency events from librispeech-we have uncovered substantial biases. our findings reveal that all six models demonstrate notably worse wer, cer, and bertscore when processing disfluent speech. this bias is consistent across both real and synthetic datasets and is statistically significant. our research also delves into disfluency-eventspecific biases, revealing that the majority of disfluencies-six out of eight types-exhibit a statistically significant wer bias across all models. this discovery is crucial in understanding the nuances of asr performance and the challenges faced by people who stutter in real-world scenarios. to address these challenges, there are a few approaches that can be considered. first, incorporating speech from people who stutter in asr training; this is challenging due to the limited availability of relevant data. additionally challenging is the task of acquiring sufficient data that encompasses samples from a diverse population to capture the variability in speech associated with stuttering. however, with a strategic data collection approach, this can address a primary limitation in asr accessibility. secondly, obtaining feedback from people who stutter and benchmarking asrs before deployment are crucial steps towards accessibility. rigorous testing on disfluent speech can identify asr deficiencies, allowing for targeted training or post-processing to address them. looking forward, we also identify two promising directions for future research. firstly, there is a need to explore the impact of longer-form audio on asr biases. current models often struggle with correctly segmenting and transcribing longer audio samples containing disfluencies, leading to signifi-