Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps (2024)

Abstract: Current automatic speech recognition (ASR) models are designed to be used across many languages and tasks without substantial changes. However, this broad language coverage hides performance gaps within languages, for example, across genders. Our study systematically evaluates the performance of two widely used multilingual ASR models on three datasets, encompassing 19 languages from eight language families and two speaking conditions. Our findings reveal clear gender disparities, with the advantaged group varying across languages and models. Surprisingly, those gaps are not explained by acoustic or lexical properties. However, probing internal model states reveals a correlation with gendered performance gap. That is, the easier it is to distinguish speaker gender in a language using probes, the more the gap reduces, favoring female speakers. Our results show that gender disparities persist even in state-of-the-art models. Our findings have implications for the improvement of multilingual ASR systems, underscoring the importance of accessibility to training data and nuanced evaluation to predict and mitigate gender gaps. We release all code and artifacts at https://github.com/g8a9/multilingual-asr-gender-gap.

## introduction a new class of multi-task, multilingual neural networks @xcite @xcite has recently pushed the boundaries of several speech-related tasks, including automatic speech recognition (asr). as these models offer support to an increasing number of languages at no cost, they have found widespread adoption and have been integrated into applications for the general public, such as real-time voice tranwe transcribe three speech datasets across 19 languages and compute gender performance gaps. next, we investigate the models for possible causes of these gaps. despite a large literature on gender, voice, and asr, to our knowledge, no studies have tested whether multi-task, multilingual asr models serve genders to a comparable level across languages. arguably, overlooking potential disparities can result in unequal service quality for already socially disadvantaged individuals @xcite . to fill this gap, we systematically study gendered performance gaps in massively multilingual asr models. we set out to answer two research questions. (q1) do multi-task, multilingual asr models perform equally well across speakers identifying as women, men, or neither of the two? if so, (q2), can we relate gender gaps to acoustic and lexical variation in data or model internal states? to answer q1, we evaluate two state-of-the-art multilingual open-weight asr models on three datasets, covering 19 languages and two speech conditions (i.e. read and spontaneous). the results of our extensive evaluation show that models systematically exhibit gender performance gaps. however, whether models favor male or female speakers depends on the dataset and language. results on one dataset also highlight subpar performance for speakers who do not identify with either gender compared to males. while studying acoustic and lexical phenomena in test data (q2), we found no significant correlations of specific features with performance gaps between male and female speakers. this objective finding underscores the complexity of the issue and the need for further research. instead, interpretability analyses suggest that when presented with speech from men and women, models build different internal representations that can serve as a proxy for gender disparities. contributions. we conduct the first extensive evaluation of two widely used multilingual asr models for gender performance differences. we document significant gender gaps by inspecting model internal states. we release code, data, and all artifacts we produce for future research. ## methodology this section describes the experimental design to answer q1, i.e., whether there are gendered performance differences in multi-task multilingual asr models. our results ( §4) confirm it. ## gender performance gaps in the following, we report the results of evaluating whisper and seamless, focusing on gender performance disparities. we first investigate whether models are equally able to recognize female and male speakers ( §4.1). then, limited to cv, we repeat the analysis for speakers identifying with neither of the two against males ( §4.2). ## acoustic and lexical analysis gendered performance gaps vary across languages and datasets. here, we dive into a focused analysis on acoustic aspects or lexical phenomena present in the test data (q2). due to data availability, we conduct this analysis on the "female" -"male" setup. ## probing gender in asr models the field of natural language processing (nlp) has now established that transformer language models encode syntactic @xcite , semantic @xcite , and factual @xcite information in hidden representations. as such, recent work has focused on extracting this information through probes, i.e., supervised classifiers trained on the model's embeddings @xcite . compellingly, some have related extractability of sensitive attributes, e.g., gender, to bias in downstream application @xcite . probing is primarily motivated by the risk that models entangle protected attributes and predictions in sensitive use cases @xcite . motivated by this line of research, we measure gender extractability in asr models and ask (q2) whether and to what extent it explains gender-based performance gaps. to our knowledge, ours is the first study of this kind. 12 experimental design. we focus on one datasetmodel configuration, namely whisper and cv. we attach our probes to the model encoder's last layer embeddings and train one distinct probe for every position. we use logistic regression and minimum description length (mdl) @xcite to probe gender in the female-male binary setup (see appendix b.3 for details). findings. figure 3 shows gender extractability on the english cv split for the f-m setup. trends are similar across languages (full results in figure 9a ). first, f1 scores suggest that gender extractability is relatively easy. whisper produces representations of female and male speech that can be easily separated. second, extractability is not constant over time. it starts high in the first milliseconds, drops during the actual signal (i.e., the speaker is talking), and finally plateaus around the initial value. this finding recalls the "attention sink" theory, implicating that transformers prioritize initial positions for specific tasks @xcite . third, the trends observed in mdl probeswhere lower codelength indicates an easier task and higher extractability-suggest that simple logistic regression is a robust method, countering the issues identified in nlp @xcite ). 13 correlation with error rate. do gender probing scores tell us something about asr quality and gender bias (q2)? to answer this question, we compare f1 scores from logistic regression scores and asr error rates. we measure f1 s , i.e., the average f1 score achieved by probes at the posi13 to confirm this hypothesis, we conduct additional experiments by training logistic probes with randomly shifted labels. we report random guess performance on all languages (see figure 9b ) in appendix b.3. tions corresponding to actual speech, foot_10 ϕ(r m ) and ϕ(r f ) (error rates on male and female segments), and the disparity score e(r f , r m ). f1 s has a weak linear correlation with ϕ(r m ) and ϕ(r f ) (pearson's ρ is 0.13 and -0.003, respectively), hinting that f1 s cannot explain per-group quality. however, ρ between f1 s and e(r f , r m ) is -0.65 (@xmath0, see figure 4 ). in other words, linear correlation suggests that the better probes can extract gender from whisper's hidden states, the lower the f-m error rate gap is. note that, as per its definition, e(r f , r f ) can be negative (see points below the zero line in figure 4 ). negative values indicate performance favoring the minority group (here, women). this finding suggests that whisper does encode recordings from speakers that identify as men and women differently. this aspect can serve as a proxy for measuring and mitigating gender disparities, e.g. by reducing gender extractability from hidden representations @xcite . however, we caution the reader from attributing high extractability to gendered voice and discourage using probes on people's voice to predict gender (see ethical considerations, §8). ## discussion twists, humps, and pebbles-the way to equitable multilingual speech recognition models is no straight line. we have discovered that gender disparities vary across spoken setups and languages and that consistency is also weak between models. perhaps of greater interest, the advantaged group is sometimes women, sometimes men, and never anyone who does not identify with either. such a high variability prevents us from foreseeing gender disparities and, in turn, the actual impact on system users. we discuss here how to i) reduce such variability and ii) better use the metrics and test data at our disposal. free! the training data. while our acoustic and lexical inquiries on test data were inconsequential, experiments with probes on the model's internals showed that models learned and encoded properties to differentiate genders. intuitively, a follow-up study would focus on what shaped those model's properties in the first place: training data. models encode facts and information from training data, and some studies looked into how different sociodemographic groups are represented @xcite . studying gender distributions in pretraining multilingual speech data would be extremely valuable, as we could establish how gendered gaps and underrepresentation correlate. however, this analysis was impossible, as neither whisper @xcite nor @xcite released such information. we call for more transparency. go beyond group-wise metrics. a crucial choice in our design was measuring differences between two groups of speakers. we opted for e(•), measured as the relative difference between the group means. while the metric was primarily meant to make results across datasets, languages, and models comparable, it does not provide a complete picture of gender disparity's concrete impact. for instance, it hides absolute gaps: a e(•) = 10 gap corresponds to a 0.2 wer gap if the model averages a wer of 2 on male speakers and to a 2 wer gap if the average is 20. these gaps may or may not be significant given many contextual factors (e.g., what the model is being used for, who is using it, and which ramifications such gaps can have). this thinking echoes that of recent critiques on measuring social bias in nlp, advocating for explicit statements about what system behavior must be considered good or bad and under which social values @xcite , and how to build contextualized metrics @xcite . we would like a metric to indicate a more precise estimation of asr model failures and their severity for users. a step in this direction would be going beyond group-wise metrics and studying sentence-level errors. drawing on the concepts of wealth distribution and inequality @xcite , we can inspect error distributions between speakers and groups and address the question "assuming x is an acceptable error rate, what fraction of records from each group will be served to a satisfactory level?" following koenecke et al. ( 2020 ), we report in figure 5 the proportion of records having an error rate at least that large. due to space constraints, we focus on whisper's transcriptions of fleurs' arabic and dutch data. this visualization lets us identify where performance between genders starts to diverge. if, given our use case, we hypothesize that transcriptions become unusable for @xmath1, men and women would be equally affected in dutch. however, it would not be the case in arabic where 40% of records from female speakers, but half of those from males, would not be acceptable. parity is reached around @xmath2. this concrete analysis highlighted gender disparities that group-wise metrics could not capture otherwise. it also served as an example of a broader argumentation: future inquiries can benefit from using metrics and observations more closely related to real-world scenarios. ## conclusion we conducted the first extensive evaluation on gender-based performance gaps of whisper @xcite and @xcite for asr. these models consistently exhibit gender bias across 19 languages from eight language families. depending on the language, disparities can favor men or women but rarely favor speakers who identify as neither. we locate a potential source of these gaps using gender probes from interpretability approaches. our results show that probes can be a proxy for gender gaps and that group fairness in multi-task multilingual models remains unsolved.