Fairness in Automatic Speech Recognition Isn’t a One-Size-Fits-All (2025)

Abstract: Modern Automatic Speech Recognition (ASR) systems are increasingly deployed in high-stakes settings, including clinical interviews, public services, and educational tools, where equitable performance across speaker groups is essential. While pre-trained speech models like Whisper achieve strong overall accuracy, they often exhibit inconsistent group-level performance that varies across domains. These disparities are not fixed properties of the model, but emerge from the interaction between model, data, and task—posing challenges for fairness interventions designed in-domain.We frame fairness in ASR as a generalisation problem. We fine-tune a Whisper model on the Fair-Speech corpus using four strategies: basic fine-tuning, demographic rebalancing, gender-swapped data augmentation, and a novel contrastive learning objective that encourages gender-invariant representations. We evaluate performance across multiple aspects of fairness and utility, both in-domain and on three out-of-domain test sets: LibriSpeech, EdAcc, and CognoSpeak.Our findings show that the method with the best in-domain fairness performed worst out-of-domain, illustrating that fairness gains do not always generalise. Demographic balancing generalises more consistently, while our contrastive method offers a practical alternative: it achieves stable, cross-domain fairness improvements without requiring changes to the training data distribution, and with minimal accuracy trade-offs.

## introduction modern automatic speech recognition (asr) models are no longer trained from scratch for each task; instead, foundation models like whisper @xcite are deployed across domains with minimal fine-tuning. while this shift has led to impressive gains in average accuracy, it poses a new challenge for fairness. interventions that previously relied on extensive task-specific training must now operate within a narrow fine-tuning window, and it remains unclear whether fairness improvements observed in one domain will hold under distributional shift. in this work, we explicitly frame fairness as a generalisation problem. we fine-tune whispersmall on fair-speech, a diverse dataset of 593 speakers and 26k utterances @xcite , using four adaptation strategies: (i) basic finetuning (ft), (ii) demographic rebalancing (ft-balanced), (iii) gender-swapped voice conversion augmentation (ft-augmented), and (iv) a novel contrastive learning objective (cl) that we propose to explicitly encourage gender-invariant representations while preserving linguistic content. to our knowledge, this is the first use of contrastive learning for gender fairness in asr. we evaluate both overall performance and fairness indomain and across three out-of-domain test sets: librispeech (read speech), edacc @xcite (accented conversational english), and cog-nospeak @xcite (task-oriented clinical interviews). across the four adaptation strategies we evaluate, spanning both data-level and representation-level interventions, we find that fairness must be treated as both a multi-metric and multi-domain property. reducing demographic gaps is not meaningful if it comes at the cost of degraded overall accuracy, or if improvements fail to persist under a distribution shift. this trade-off is most clearly illustrated by basic fine-tuning, which achieves the lowest gender gap in-domain but results in the worst outcomes in out-of-domain. this contrast demonstrates a central point of this work: methods that appear most fair in-domain can, under distribution shift, become the least fair overall. our results highlight the risk of treating fairness as a local optimisation problem, rather than a generalisation challenge. other approaches show more promise. demographic balancing generalises more consistently across test sets, though its effectiveness may depend on the demographic diversity of the training data. our proposed contrastive learning objective offers a complementary perspective: rather than training directly on synthetic data, we use genderswapped utterances to construct contrastive pairs that encourage gender-invariant representations. this representation-level regularisation achieves competitive fairness improvements with minimal impact on performance, while avoiding modifications to the supervised training distribution. to summarise, this work makes three main contributions: (1) we frame asr fairness as a generalisation problem and provide the first evaluation across multiple domains; (2) we compare data-level and representation-level fairness interventions under a unified setup, including a novel contrastive objective that promotes gender-invariant representations; and (3) we demonstrate that in-domain fairness gains often fail to generalise, and argue for evaluating fairness interventions under distribution shift, using both performance and disparity metrics across domains. ## results and analysis ## conclusion this work introduced fairness as a generalisation challenge in asr, showing that interventions effective in one domain may not persist across others. we evaluated four strategies on the whisper small model, comparing data-level and representationlevel interventions across in-domain and out-ofdomain benchmarks. our findings show that indomain fairness is not a reliable proxy for robustness, and that fairness must be assessed as a property of both model behaviour across groups and performance under distribution shift. among the methods, demographic balancing showed the strongest cross-domain fairness but depends on access to balanced training data. our proposed contrastive learning objective offers a complementary perspective: by encouraging gender-invariant representations using augmented data, it improves fairness without requiring changes to the supervised training distribution. unlike data augmentation strategies that inject synthetic samples directly into the main loss, our contrastive objective isolates fairness regularisation, minimising the impact of potentially lowerquality generated data. looking ahead, contrastive learning presents a promising direction for scalable, generalisable fairness in asr. while this work focused on gender, the objective naturally extends to other protected attributes such as age, accent, or gender identity. by treating same-content utterances across these attributes as positive contrastive pairs, future models could be trained to align representations across multiple demographic axes simultaneously. in contrast, expanding data-level augmentation to cover all such dimensions would significantly increase training set size and compound quality concerns. contrastive methods thus offer a principled path forward for multi-attribute fairness, especially in settings where demographic labels are available but rebalancing or augmenting each domain is infeasible.