Vers l’entraînement de modèles de reconnaissance automatique de la parole auto-supervisés équitables sans étiquettes démographiques (2025)

Abstract: Malgré des avancées importantes dans le domaine de la Reconnaissance Automatique de la Parole (RAP), les performances de reconnaissance restent inégales selon les groupes de locuteurs, ce qui pose des problèmes d’équité. Bien qu’il existe des méthodes pour réduire ces inégalités, elles dépendent de ressources externes au signal vocal, telles que des modèles de locuteur (speaker embeddings) ou des étiquettes démographiques textuelles, qui peuvent être indisponibles ou peu fiables. Dans ce travail, nous proposons une méthode pour améliorer l’équité dans la RAP qui ne dépend d’aucune de ces ressources. Notre approche utilise une méthode de clustering non supervisé à partir de représentations acoustiques classiques, auto-supervisées et hybrides. Nos expériences avec CommonV oice 16.1 démontrent que les modèles entraînés sur les clusters découverts améliorent les performances des groupes démographiques désavantagés tout en conservant des performances compétitives et en utilisant deux fois moins de données d’entraînement.

## introduction much like it did for text-based and image-based applications powered by artificial intelligence, recent years have seen speech-based applications become ubiquitous : from the comfort of our own homes, we live in continuous interaction with smart devices that grant us immediate access to speech-based services. like in text and image processing too, speech processing has benefited from the rise of self-supervised learning (ssl), a machine learning paradigm that seeks to extract general and robust representations of data without the need for labels. speech systems that rely on this paradigm have systematically seen performance gains across different tasks and domains largely due to an evergrowing variety of network sizes, network architectures, and the open release of very large data resources. however, although representations learned by these networks have indeed been shown to be meaningful, they have also been shown to be harmful @xcite @xcite , exhibiting a wide range of socioeconomic and demographic biases learned from data when applied to end-user applications. training and releasing language models without care or accountability for the biases they contain does not only enable the perpetuation of such biases but also directly hampers the development of fair and inclusive technologies. for instance, research in automatic speech recognition (asr) has shown that users from demographics known to be disadvantaged by a system might feel the need to adjust their speech to resemble that of a demographic they believe is best recognized in order to obtain better performance @xcite . scenarios such as this one, which occur as a direct consequence of deploying heavily biased systems into production environments, can compromise the feedback received by these systems ultimately leading to the development of technologies that are even more biased and less robust to out-of-distribution inputs. yet, in the context of asr, only a handful of works have targeted the study of system biases @xcite @xcite @xcite and even fewer attempts have been made to try to alleviate them @xcite @xcite . moreover, these works rely heavily on the usage of textual demographic labels to either balance out training corpora across demographic groups or over-represent demographic groups in disadvantage, both of which have been shown to be insufficient to achieve fairness @xcite . in our view, several sensible and perhaps obvious reasons exist as for why textual demographic labels are an unsuitable resource to train fair self-supervised speech processing models. demographic label categories are often too coarse to be used on their own (gender categories alone disregard the differences between speakers of different age or different accent), too granular to be applied together (data groups built out of couples of gender and age categories are often of insufficient size for training), or simply too complex to be defined (accent categories are often let to the user to describe, resulting in no consensus across different samples and even different datasets). furthermore, demographic label categories do not necessarily correlate to voice features and often overlap each other (the voices of young males are more similar to that of young females than to adult males). aside from the inherent challenges of obtaining manual annotations for these labels-such as their high cost, time-consuming nature and privacy-related issues-it is hard to imagine that these labels will ever be descriptive enough to account for speech variations and subtleties among speakers, which is essential to build representative demographic groups. finally, manually annotated demographic labels also introduce unnecessary biases into the developing pipelines @xcite , which further hinders the creation of fairer models. in this work, we present an approach for improving the demographic fairness of a self-supervised asr model that does not rely on external speaker embedding models or textual demographic labels. instead, our approach leverages classical, self-supervised and hybrid classical-embedded representations, which can be computed from the speech signal directly or extracted from the same self-supervised model. our proposed approach also leverages unsupervised clustering of these features, which further supports our claim that training fair asr models can be attained without the use of demographic labels. the contributions of this work towards reaching fairness in asr are the following : ## related work recent research in asr has highlighted significant performance disparities between demographic groups. for instance, with respect to their ethnicity, authors in @xcite found that native african-american english speakers experience word error rates up to twice as high as white standard american english speakers. with respect to the spoken dialect, authors in @xcite found asr systems are two to four times less accurate at correctly inferring habitual "be"-a key morpho-syntactic feature of african american english-compared to non-habitual "be" in standard english. in terms of both dialect and gender, authors in (tatman, 2017) found youtube's automatic captifor asr system performed worse overall for female speakers and speakers of highly accented variants, such as scottish english, when evaluating the system on five different regional english dialects. apart from these findings, the presence of socio-economic and socio-demographic biases in speech processing can be tracked down to much more than just end-system performance. as shown by @xcite in their study of the voxceleb speaker recognition challenge, biases exist and can stem from every stage of a speech processing pipeline. works in the field seeking to improve asr performance across demographic groups have approached the task by either trying to build relevant data groups for training using demographic labels as guidance @xcite or trying to learn demographic group differences directly within the model @xcite . our work is similar to those in the first category in the sense that we aim to develop an effective data-driven approach to reduce the disparities, and related to those in the second category by the application unsupervised clustering in our respective pipelines. our work, however, differs from previous efforts in that we do not rely on demographic labels to build training groups but we leverage classical, as well as self-supervised representations, as features to find emerging data groups in an unsupervised fashion. to the best of our knowledge, we have found no evidence of any other work in asr that explores and compares the effectiveness of these representations for building training clusters with the aim of reducing demographic disparities. ## method our pipeline for training fairer asr models builds on three consecutive steps : computing representations from speech data, applying unsupervised clustering over these representations and fine-tuning a self-supervised model for asr on cluster-derived training subsets. we further describe the details of each of these steps below. ## conclusion in this work, we introduce a three step pipeline-approach for mitigating demographic group performance disparities in asr by leveraging different types of representations, including ones defined by classical acoustic features and self-supervised models. our findings demonstrate that classical features such as mfccs and lps provide a meaningful representation that can be exploited through unsupervised clustering to form emerging training groups that improve asr fairness. compared to self-supervised embeddings, these representations resulted in more balanced wer performance across demographic groups while maintaining competitive performance. the results of our research suggest that fairness in asr can be enhanced through feature-driven clustering, reducing the reliance on potentially biased or insufficient demographic annotations. our study contributes to the broader effort of developing more equitable speech technologies and encourages further exploration of feature-informed learning paradigms in self-supervised speech processing.