Language-specific Effects on Automatic Speech Recognition Errors for World Englishes (2022)

Abstract: Despite recent advancements in automated speech recognition (ASR) technologies, reports of unequal performance across speakers of different demographic groups abound. At the same time, the focus on performance metrics such as the Word Error Rate (WER) in prior studies limit the specificity and scope of recommendations that can be offered for system engineering to overcome these challenges. The current study bridges this gap by investigating the performance of Otter’s automatic captioning system on native and non-native English speakers of different language background through a linguistic analysis of segment-level errors. By examining language-specific error profiles for vowels and consonants motivated by linguistic theory, we find that certain categories of errors can be predicted from the phonological structure of a speaker’s native language.

## introduction a central concern in the ethics of building natural language processing (nlp) tools is ensuring equity in service and representation through a commitment to linguistic justice (e.g., @xcite . this issue is especially pertinent to automatic speech recognition (asr) systems used to transcribe natural spoken language into text @xcite . as asr systems are increasingly being adopted into many aspects of social life (e.g., virtual assistants and automatic captioning), various concerns about the equity of asr systems have been raised. ideally, these systems should serve speakers of different demographic backgrounds equally well; however, the existing technologies are not entirely satisfactory. asr performance has been found to vary by users' dialect @xcite , gender @xcite @xcite @xcite and ethnicity @xcite . for instance, asr systems designed for american english typically perform worse for non-white speakers than for white speakers @xcite . such inequalities result in certain groups of speakers being better represented than others, and may even further exacerbate existing inequalities in society. at the same time, less is known about how the performance of asr systems can vary for secondlanguage (l2) english speakers, a particularly vulnerable population of english speakers with diverse backgrounds. a recent work by @xcite examined how otter, a popular automatic transcription system, performs on l1 (native) and l2 (second-language) speakers of 24 english varieties. not only do the english varieties supported by otter have lower word error rates (wer) compared to the unsupported varieties, gaps in performance is also driven by an independent effect of the structure of a speaker's first language -otter performs worse on english spoken by l1 speakers of a tonal language (e.g., mandarin). while the word error rate has been widely adopted in these studies due to the ease of quantifying system bias, this one-dimensional measure of performance is inadequate if the aim is to disentangle different types of errors that give rise to discrepancies in performance between speakers. for example, word-level errors can be driven by an error in one or multiple sound segments, and certain errors for consonants and vowels may be more common for speakers of one variety than speakers of another. studying these details is useful because these linguistic categories are wellstudied theoretical constructs and empirical phenomena in the research on sociolinguistic variation (e.g., @xcite @xcite @xcite . moreover, such linguistically-motivated features like conso-nant voicing and vowel height have acoustic correlates, which means that insights from studying system errors in terms of phonological variation and processes can be translated to applied system engineering. therefore, the current study aims to systematically investigate whether variations in language structure among native and non-native language speakers of english are tied to different types and degrees of transcriptions errors. first, we introduce and motivate a segment-level error analysis built on traditional error-rate algorithms, which allows an analysis of errors beyond the single dimension of performance. we then investigate the error profile for consonants and vowels across various english varieties, to determine whether the phonological structure of a speaker's native language/variety is predictably tied to certain types of errors that can be captured in terms of phonological processes. ## consonant analysis in this section, we zoom in on the consonant errors in the otter transcription of the same pas-sage read by l2 english speakers with different l1s. it has been established that phonological features of l1 are likely transferred into speakers' l2 @xcite @xcite . for example, given that japanese only allows cv syllables, native speakers of japanese have more difficulty with complex consonant cluster pronunciation (e.g. "sixth") when speaking english. therefore, it is not surprising if we find traces of different l1s in our current data set of l2 englishes. however, what remains under-explored is whether these phonological differences of l1s will be directly reflected in the kinds of consonant errors asr algorithms make on these nonnative englishes, especially one that claimed to have trained on non-native accents of english @xcite . we specifically test this in the following section. ## vowel analysis in the following section, we turn to the distribution of vowel errors in the otter transcriptions. we analyze the distribution of vowel errors in the acoustic space in context of the typology of vowel systems among the l1 languages represented in the data. specifically, we focus on vowel substitution errors (as opposed to insertions and deletions) as they allow us to explore otter's errors in terms of wellstudied phonological processes. ## discussion this study investigated the performance of otter's automatic captioning system on native and nonnative english speakers of different language backgrounds through a linguistic analysis of segmentlevel substitution errors. we proposed that understanding language-specific error profiles is crucial to preempting predictable system errors. in our analysis of consonant and vowel errors motivated by phonological theory, we report the following findings. results from our consonant analysis show that the phonological structure of a non-native english speaker's first language predicts the types of consonant errors that are dominant in the automatic transcription of their production of english. specifically, we find higher rates of consonant cluster errors in the transcription of speakers whose native language does not allow consonant clusters. similarly, we find higher rates of stop voicing errors for speakers whose first language has aspiration contrasts, as opposed to voicing contrasts like english. results from our vowel analysis show that the distribution of vowel substitution errors patterns with the structure of the vowel inventory of a nonnative english speaker's first language. specifically, we find three main patterns. first, the transcriptions of non-native speakers whose first language make fewer vowel distinctions than english show predictable regions of error in the vowel space. second, when languages have similar phonological structures compared to english, the frequency of vowel substitution errors tend to be lower. third, known sociolinguistic variation even among native speakers of english also predict vowel errors in a similar manner. understanding how the sound patterns of a speaker's native language and/or variety of english affect the performance of asr systems on specific categories of sounds is an important first step towards designing robust yet flexible acoustic models which can detect and adapt to varieties without reducing structural differences to correlated demographic information like race and ethnicity @xcite . recent successes in the design of accent adaptation algorithms support this view, such as the automatic accent identification model by najafian and russell (2020) trained on regional varieties of british english. furthermore, the focus on adapting to differences in linguistic structure grounded in phonological theory is crucial to the generalizability of asr systems, which is an important consideration for providing transcription services to speakers of minority languages. for example, an asr model designed to learn and leverage language-specific phonemic contrasts by @xcite vastly improved phone-level accuracy on very small corpora ( 1k utterances) of two indigenous low-resource languages, inuktitut and tusom. our study contributes to this momentum by offering insights into the phonological and acoustic nature of transcription errors for speakers of different language backgrounds. such research on language-specific error profiles can motivate the design of adaptation algorithms for supporting non-native english speakers of various language backgrounds. lastly, improving the performance of asr systems for non-native speakers of english is an important task not simply for the sake of catering to a large user-base given the status of english as the de facto lingua franca with far more l2 than l1 english speakers in the world (sil international, 2022), but also because l2 speakers of english are an especially vulnerable population facing specific, and often invisible, harms from the prevailing stereotypes of being uninterpretable in speech and lacking education and proficiency in english @xcite . many existing applications of asr systems are ill-equipped to work with speech input from non-standard varieties, as evidenced by accumulating cases of discrimination against nonnative speakers of english across all levels of harm @xcite . for example, allocational harms have been reported for even life-or-death situations such as in voice command systems for roadside vehicle assistance @xcite and for medical diagnoses and records management in healthcare systems @xcite . moreover, representational harms from stereotypes of unintelligibility are perpetuated by systems that claim to work on a language while neglecting how the system might perform differently among sociolects @xcite . as an extreme example of this ideology, some recruiting firms have claimed to screen and rank job applications by passing their voice data through off-the-shelf asr systems and using the interpretability of the transcription output itself as proxies for friendliness and communication skills, putting non-native english speakers at a disadvantage @xcite . in these ways, unchecked bias against non-native speakers of english in asr systems reinforce social inequalities and simultaneously dismiss real cases of need for accommodation. ## conclusion in this study, we examined the language-specific error profiles of native and non-native english speakers of diverse language backgrounds. a segmentlevel analysis of consonant and vowel errors made by otter's transcription system reveals that certain categories of errors are predictable from the phonological structure of a speaker's native language. thus, we demonstrate the fruitfulness of applying a linguistic analysis to transcription errors, informed by general phonological theory as well as specific literature from relevant domains such as sociolinguistic variation and second language acquisition. findings inform the design and maintenance of new and existing asr systems for adapting to nonnative speakers of english and speakers of nonstandard english varieties.