Context-sensitive evaluation of automatic speech recognition: considering user experience & language variation (2021)

Abstract: Commercial Automatic Speech Recognition (ASR) systems tend to show systemic predictive bias for marginalised speaker/user groups. We highlight the need for an interdisciplinary and context-sensitive approach to documenting this bias incorporating perspectives and methods from sociolinguistics, speech & language technology and human-computer interaction in the context of a case study. We argue evaluation of ASR systems should be disaggregated by speaker group, include qualitative error analysis, and consider user experience in a broader sociolinguistic and social context.

## introduction automatic speech recognition (asr) has become a common tool in human-computer interaction, enabling, for example, voice user interfaces and (imperfect) automatic captioning of multimedia content. as with other language technologies (e.g. @xcite , rapid improvements in performance have not been equal for different user groups. as @xcite show, discussions of this "bias" are often poorly defined, not grounded in explicit normative judgments and divorced from socio-historical contexts, origins and harms of the system behaviours. in this paper, we argue that researchers at the intersection of speech and language technologies (slt), humancomputer interaction (hci), and sociolinguistics are well-placed to consider the experiences and social context of different speaker/user groups in critical quantitative and qualitative evaluations of asr systems. knowledge about language variation and its relation to society coupled with expertise from hci allows us to understand how predictive biases reflect larger social structures and ideologies about language, and how they affect users. after presenting prior work on language variation and asr, a case study of self-recorded au-dio diaries collected for the lothian diary project foot_0 highlights the need for a context-sensitive approach to asr evaluation which we outline. recent work describes stark racial bias in commercial american english asr systems (including google's cloud speech) @xcite , with much higher word error rates (wer) foot_2 for speakers of african american english (aae) than white speakers of (californian) american english. notably, these types of error disparities appear to be driven by under-representation of aae training data both for the acoustic modelling @xcite and the language model used to decode sequences of phones into utterances @xcite ). "regional" variation has also been reported as a source of unequal performance, with particularly high error rates reported on youtube's captions for speakers from scotland and (the us state) georgia @xcite . similar to more recent work, youtube captions have been found to perform worse for african american speakers @xcite . these problems are not limited to proprietary systems, as mozilla's open source system deepspeech performs significantly worse for speakers of indian english than "american english" foot_3 @xcite , and also fails to transcribe aae morpho-syntactic variation correctly @xcite . while some early research has suggested asr performance differences based on (binary) speaker gender @xcite @xcite , it is unclear that gender by itself is a significant factor in recent systems @xcite . @xcite suggest that the interaction of gender and race is significant, with differences between black men and black women being more significant than between white men and white women or men and women across race foot_4 . these results appear to be linked to speaker's speech styles (e.g. in @xcite and use of dialect features @xcite , both of which have long been documented to pattern with gender (see @xcite , for a classic paper) and could be correlated with gender in training and test sets. other work in this space has focused on the potential of asr to improve accessibility of audio media and digital technologies, looking at experiences of deaf and hard of hearing users @xcite and dysarthric speakers @xcite . for both groups commercial asr systems perform quite poorly, though the severity and amount of errors varies by speaker. research on predictive bias in commercial asr for regional varieties of english beyond the united states and in the context of systems not exclusively trained on american english, as well as experiences of second language learners of english, and other groups who are potentially particularly reliant on asr to access computing technologies such as elderly people, is sparse. from a linguistic perspective, no language variety or speech style is inherently more difficult, incorrect, or inappropriate than any other. there are, however, powerful ideologies regarding the relative status of different varieties and styles which are rooted in broader socio-historical contexts and reflect the social status of the groups who speak them @xcite . in addition to being stigmatised in "traditional" contexts of power in society, varieties spoken by marginalised communities appear to be (not coincidentally) underrepresented in the data we use to build and evaluate speech technologies, leading to substantial predictive biases making speech technologies less accessible to already marginalised groups. ## lothian diaries: a case study the lothian diary project is an ongoing interdisciplinary research project inviting residents of the lothians region of scotland to contribute selfrecorded audio and video diaries about their experiences of the covid-19 pandemic. the more than 120 diaries collected so far are highly variable in recording quality, number of speakers and topics discussed, and participants are diverse foot_5 in terms of age, gender, linguistic background, ethnicity, socio-economic class and level of education. edinburgh and the surrounding lothians region are of particular interest for sociolinguistic research because of the capital region's status as a centre for higher education, finance and tourism. in addition to the variation within scottish english foot_6 between different areas and different socio-economic groups within the city, there is also a wide range of other first and second language varieties of english, as well as other languages. the lothian diary project also includes many of these other varieties of english, rather than focusing on speakers with long residential histories in a particular area (as is often the case in sociolinguistic work) or first language speakers (as is usually the case in slt evaluation). the recordings form a highly naturalistic and exceptionally varied data set. asr is used here to facilitate social science research which requires accurate and complete transcriptions (achieved through manual correction). so far, 13 diaries submitted by participants who agreed to have them made public, have been processed with the google cloud speech-to-text api 8 (gc stt). diaries (16 khz flac files) were processed in their entirety using the model used for long audio files which uses asynchronous speech recognition. wer was computed separately for id g variety wer rf f scottish english 46.4 la f scottish english 35.7 ce f scottish english 20.9 aa m scottish english 29.8 mr m scottish english 55.3 dl m scottish english/scots 88.9 li f southern british english 29.5 jw m canadian english * 25.3 l f l2 english (l1: lithuanian) 31.5 s f l2 english (l1: cantonese) 27.7 mg f l2 english (l1: italian) 35.3 jl m l2 english (l1: filipino) 40.8 a m l2 english (l1: chinese) 70. 2 all of these errors are particularly challenging for the accurate and complete transcription of spontaneous and conversational speech, especially for social science research where researchers (users) might consider hesitations, false starts and filled pauses important as they convey pragmatic information. considering impacts of this predictive bias, transcripts of speakers who produce more "fluent" speech are much more easily interpretable. retrieving speech content and speech style of less fluent speakers as well as some second language speakers, on the other hand, requires more labour and time, potentially negating any benefits of asr. ## proposed methods to document predictive bias in asr in a way that is mindful of 1) user experience, 2) socio-historical and (socio)linguistic context, 3) (potential) harms (re)produced by the system, and 4) technical aspects of asr, we need to draw on methodologies and knowledge from hci, sociolinguistics, research on fairness in ai, and slt. ## conclusion we have proposed an approach to asr evaluation which considers the experiences of different user/speaker groups, sociolinguistic context and potential impacts of predictive bias. we argue that this interdisciplinary approach is necessary to significantly advance our understanding of asr usability. we particularly invite perspectives from the fields of human-computer interaction in order evaluate speech and language technologies as systems situated in specific sociolinguistic and socio-technical contexts which perform specific tasks for specific (language) users.