On Mitigating Performance Disparities in Multilingual Speech Recognition (2024)

Abstract: How far have we come in mitigating performance disparities across genders in multilingual speech recognition? We compare the impact on gender disparity of different fine-tuning algorithms for automated speech recognition across model sizes, languages and gender. We look at both performance-focused and fairness-promoting algorithms. Across languages, we see slightly better performance for female speakers for larger models regardless of the fine-tuning algorithm. The best trade-off between performance and parity is found using adapter fusion. Fairness-promoting fine-tuning algorithms (Group-DRO and Spectral Decoupling) hurt performance compared to adapter fusion with only slightly better performance parity. LoRA increases disparities slightly. Fairness-mitigating fine-tuning techniques led to slightly higher variance in performance across languages, with the exception of adapter fusion.

## introduction automatic speech recognition (asr) systems have become ubiquitous in our daily lives, powering virtual assistants, dictation software, customer service interactions, and more. however, these systems are not always equally effective for all users, and gender disparity in their performance is a significant concern @xcite . one key factor contributing to gender disparity in asr performance is the way in which these systems are trained. if the training data predominantly consist of male voices, for example, asr systems may exhibit higher error rates when transcribing female speech. or vice versa. the downstream societal impact of gender disparity in asr systems is multifaceted. it can exacerbate existing inequalities by hindering access to information and services for certain social groups. in professional settings, inaccurate transcription can impede communication figure 1 : augmenting whisper with adapter fusion for better performance and gender parity. adapter fusion is over three adapters -one trained with a vanilla loss (erm), one trained with group-dro, and one trained with spectral decoupling (sd). and productivity, potentially affecting career advancement opportunities. finally, the perpetuation of performance disparities in technology can reinforce harmful stereotypes and norms, contributing to broader social inequalities. mitigating gender disparities is non-trivial. several algorithms have been proposed @xcite , but they often introduce significant performance costs @xcite , and their impact on gender disparity is often small @xcite . in this paper, we evaluate their impact on performance and gender disparity in multilingual asr, but we also present a simple technique to improve the robustness of fairness-improving finetuning methods. specifically, we augment multilingual whisper using adapter fusion @xcite over several fairness-promoting strategies; see figure 1 . contributions we evaluate the performance and gender disparity of four different fine-tuning strategies for multilingual asr models, two of which are fairness-promoting. our baselines are standard empirical risk minimization (erm) and so-called low-rank adaptation (lora) @xcite . the two fairness-promoting algorithms are group-dro @xcite and spectral decoupling @xcite . finally, we evaluate a novel combination of existing techniques: adapter fusion over erm, group-dro, and spectral decoupling. we evaluate all models across 16 languages with binary gender as our demographic variable, and find that only adapter fusion reduces gender disparity without incurring a significant performance drop. in fact, performance improves, and disparity is reduced. cross-lingual performance gaps widen a bit, however. ## fairness-promoting finetuning standard expected risk minimization (erm) finetuning in automatic speech recognition (asr) and fairness-promoting fine-tuning techniques like group-dro @xcite , spectral decoupling @xcite , and adapter fusion @xcite serve different purposes and address different aspects of model performance. erm fine-tuning, our baseline technique, minimizes the expected risk or the expected value of a loss function over the data distribution. in asr, this typically involves reducing the word error rate (wer). erm fine-tuning involves iteratively updating the parameters of the asr model using a gradient descent-based optimization algorithm, such as stochastic gradient descent (sgd) or adam, to minimize the loss function. fairness-promoting fine-tuning techniques aim to increase performance parity in asr systems by addressing disparities in model predictions across different demographic variables (e.g., gender, race, age). group-dro @xcite ) modifies the standard fine-tuning objective to optimize for performance parity by minimizing the worst-case error across multiple demographic groups, ensuring that no particular group is disproportionately affected by the model's predictions. spectral decoupling @xcite involves modifying the training process to remove unwanted biases from the learned representations, by decoupling sensitive features from the predictive features in the model. the fairness-promoting techniques differ from erm fine-tuning in their explicit attention to performance parity, often introducing additional constraints or modifications to the training process to achieve more equitable model outcomes. while erm fine-tuning seeks to optimize the overall performance of the asr system, fairness-promoting techniques specifically target performance dispari-ties across social groups. adapter fusion fine-tunes a pre-trained model by adding small neural adapters to the model architecture, which are specifically trained to mitigate performance disparities and improve empirical fairness in the model's predictions while retaining the knowledge learned from the pre-training phase. the adapter fusion model explored here combines erm, group-dro and spectral decoupling adapters by adding an extra adapter layer. ## discussion fairness and (two flavors of) regularization we saw that lora exhibits higher performance disparities than vanilla empirical risk minimization. since low-rank adaptation is a form of regularization, this seems at odds with previous work suggesting that regularization mitigates performance disparities @xcite . the incongruency is only apparent: it is well-known that sparsity-promoting regularization -like lora -hurts robustness @xcite @xcite , while ℓ 2 -regularization, ℓ ∞ -regularization, noise injection @xcite , and drop-out @xcite often improve robustness. the work cited above @xcite both use ℓ 2 -regularization. rawlsian fairness whether fairness is best measured by the absolute performance of the worst-off group (rawlsian fairness) or by the relative performance differences across groups (egalitarian fairness) is a philosophical question (jørgensen and søgaard, 2023), but we note that in our case, this has direct consequences for what approach to recommend: group-dro or adapter fusion? adapter fusion has superior performance, also for the worst off group, so it is preferable on rawls' account. the egalitarian would prefer something like group-dro, however, since cross-group differences (∆) are smaller. note that minimizing cross-group differences is preferable even to leveling down @xcite from an egalitarian perspective. linguistic fairness fairness is usually measured across social groups defined by the cartesian product of a set of protected attributes. however, global technologies can serve some language communities better than others @xcite @xcite . this goes for both error rates and disparities @xcite ; ramesh et al., 2023). if whisper or other multilingual asr models consistently exhibit much higher error rates or disparities on data in specific languages, their speakers will be disadvantaged. the standard deviations of performances across 16 languages for whisper-large are given in figure 3. fine-tuning cuts more than half of the standard deviation across languages. the fact that adapter fusion leads to the lowest disparity across languages, is promising. cross-lingual performance gaps such as the ones observed here are still worrying and call for more research; see also rust and søgaard (2023) for discussion. the pros and cons of stacked architectures adapter fusion adds additional layers of param-eters to adjudicate between the three adapter layers trained with different objectives. if we add parameters, we slow down inference time, but on the upside, fine-tuning adapter layers is inexpensive in comparison to fine-tuning encoder or decoder layers of the original whisper model. stacking is cheaper than voting in this respect, since we do not have to fine-tune with three different objectives. finally, the slow inference time can be improved by adapter pruning @xcite , without compromising fairness. ## conclusions in this paper, we evaluated the impact on performance and gender disparity of different fine-tuning algorithms in automated speech recognition, including fairness-promoting ones. our analysis considered model size, language, gender. in general, we saw better performance for female speakers for larger models and significant performance gaps across languages. fairness-promoting finetuning algorithms (group-dro and spectral decoupling) hurt performance with some improvements in performance parity. lora increases disparities slightly. we find the highest performance with acceptable parity in a novel, fairness-promoting variant of adapter fusion, which had positive ef-fects on performance, group fairness, as well as parity in performance between languages.