Investigating the Impact of Gender Representation in ASR Training Data: a Case Study on Librispeech (2021)

Abstract: In this paper we question the impact of gender representation in training data on the performance of an end-to-end ASR system. We create an experiment based on the Librispeech corpus and build 3 different training corpora varying only the proportion of data produced by each gender category. We observe that if our system is overall robust to the gender balance or imbalance in training data, it is nonetheless dependant of the adequacy between the individuals present in the training and testing sets.

## introduction as pointed out by @xcite in their positional paper on the social impact of nlp, discriminatory performance could be the result of several types of biases. the roots of socio-technical biases in new technology could be situated in its very design, the selection of the data used for training @xcite , the annotation process @xcite , the intermediary representations such as word embeddings @xcite or in the model itself. gender bias in asr systems, defined as a systematically and statistically worse recognition for a gender category is still a working topic @xcite . pioneer work from @xcite found better performance on women's voices, while a preliminary research on youtube automatic caption system found better recognition rate of male speech @xcite but no gender-difference in a follow-up study @xcite . recent work on hybrid asr systems observed that gender imbalance in data could lead to decreased asr performance on the gender category least represented @xcite . this last study was conducted on french broadcast data in which women account for only 35% of the speakers. if systematic, this performance difference could lead to less indexing of media resources featuring female speech and contribute to the invisibilisation of women and women speech in public debate foot_0 and history @xcite . such results would also fall into the category of allocational harms, following the typology proposed by @xcite and @xcite , because women are more likely to be less represented in corpora @xcite , making all technologies relying on speech recognition less accessible for them. it could also result in representational harm such as the maintenance of the stereotype of inadequacy between women and technology. 2 but as other characteristics such as the speaker role, (i.e. his or her ability to produce professional speech) could explain some performance variations, we propose in this paper to address the question of asr systems' robustness to gender imbalance in training data. as data is now the starting point of every system, we know that the quality of a system depends on the quality of its data @xcite . to tackle this question, we work with the librispeech corpus, widely used in the community and based on audio books recordings. to evaluate the impact of gender imbalance in training data on our system performance, we proceed as follows: we first test the robustness of our model against the randomness introduced at training by the weight initialization stage. we then evaluate the impact of speakers selection in training data on model performance. we compare the obtained results to the impact observed when changing overall gender representation in training data. finally we observe the behavior of our model when trained on mono-gender corpora. we validate our model robustness against the impact of model seed and observe that overall system is quite robust to gender balance variation. we note that the random factor introduced in the selection process of speakers for the training set seems to have a statistically significant impact on performance. we argue that our model, whereas robust to gender representation variability, is strongly dependent on the individuals present in the training set, which questions the pertinence of gender as a category of analysis in asr and advocate for a return to a more incorporated conception of language. ## end-to-end model of automatic speech recognition for the last decade, the traditional asr models, based on hmm-gmms have been coexisting with hybrid models (hmm-dnns) @xcite and for the latest couples of years with end-to-end systems. the former acoustic, pronunciation and language models, made explicit by different modules in the final system, are now collapsed into one big architecture mapping directly the audio signal to its transcription. since speaker adaptation has been integrated into the entire training process of end-to-end models, we are expecting the gender imbalance within training data to be extrapolated by this kind of systems, resulting in gender-differentiated performance. ## impact of the model seed our hypothesis that systems might be impacted by a gender imbalance in training data is based on the fact that systems are deeply dependent on the data they are trained on @xcite . in order to control that the behaviors we observe are only due to the data variation, we conduct a first experiment where we test the robustness of our model to the seed variability at training. to do so, we train three models with the wper50 (gender-balanced) training set, changing only the model seed. obtained wer distributions are represented in figure 1 . when performing the kruskall-wallis test, no statistical significant difference is observed between the 3 distributions (p-@xmath0.17). the same observation is made when comparing the models two by two. we conclude that our model is robust to the randomness introduced at the initialisation stage. ## impact of the training data (data seed) we believe that gender is an attribute of the speaker and that speaker's gender variability goes beyond gender statistics. following judith butler's theory on the performativity of gender @xcite , we assume that gender is not expressed in the same way amongst speakers. the intrinsic variability of gender indexing @xcite leads us to consider that two people sharing the same gender "label" will not be interchangeable in a data set. in order to test this hypothesis, we created two other training sets with a 50-50 men/women balance but with a different random seed for the shuffle and selection process for these training corpora. we refer to this random element as the "data seed". we call the two models d2 and d3 (data seeds values were chosen arbitrarily). we obtained the distributions presented in figure 2 . wilcoxon rank sum test is statistically significant between the two distributions (@xmath1.5; p-@xmath2.003). model d2 obtains a mean wer of 9.31% and model d3 a mean wer of 7.65%. we argue that the data exhaustiveness is thus a strong factor of performance variation. when looking at the performance obtained by gender categories (see figure 3 ), we also observe distinct behaviors between the two models. wer for books read by male (8.14%) and female (10.2%) speakers are statistically different in our model d2 (@xmath3; p-@xmath4.008). this effect is not found in model d3 (@xmath5; p-@xmath6.038), although a difference of almost 2 points is also observable between the mean wer for men (6.80%) and women (8.31%). there is trend to obtain slightly better wer results for male speakers, with an average difference of 1.7 percentage point. the performance difference between the gender categories is thus of the same order as the difference between our models d2 and d3. ## gender balance and performance in this experiment we try to evaluate the impact of gender representation on the performance. to do so, we trained 3 asr systems, with our 3 different training sets presented in section 2.2. results are reported in a quick look at our wer distributions by gender category shows that the performance obtained for women are generally worse than the one obtained for men (see figure 4 ). this difference is statistically significant (p-@xmath8.003) when our training set contains only 30% of books read by women and p-value increases until it exceeds our alpha risk (p-@xmath9.04 for wper50 and p-@xmath10.10 for wper70). we can argue that an under-representation of a gender category leads to a higher error rate, but the same trend is not observable for male speakers. surprisingly, when training set contains 70% of books read by women, there is no significant difference between wer obtained for male and female speakers. even if it is not statistically significant, the trend observed in section 4 holds because with 70% of female speech in training data, we still observe better wer results for men. our overall system performance seems to be robust to the variation in gender representation in training data. in wper30 model we observe a statistically significant gender difference in wer. the better wer results for male speakers are expected as they are more represented in training data. this is not the case for wper70 model, where we expected better results for women. therefore we trained male-only and female-only models to analyse extreme behaviors. we maximized the size of our training set, reaching a book count in these monogender systems of 2671. hence, it is worth noting that the size of training data in these systems is smaller than the size of training data for our wper models. overall wer for the male-only model is of 12.3% and 11.7% for the female-only one without any statistically significant difference. in the maleonly model, wer distributions are statistically different by gender category (p-@xmath11, with an average wer of 9.11% on books read by men and of and of 14.7% on books read by model gender test-clean wper30 f 10.9% m 8.3% all 9.7% wper50 f 11.0% m 9.1% all 10.2% wper70 f 9.6% m 8.3% all 9.0% ## discussion & conclusion it is a common-sense claim to state that all gender categories need to be represented in a training corpus of an asr system in order to be able to transcribe speech regardless of the user's gender. we expected to find that the performance obtained on each gender category was dependent of their representation in training data. however, if we select individuals while maintaining a balanced gender distribution (see section 4), we obtain a significant difference in performance of around 1.7 percentage point. it is possible that these differences in performance, between systems and between genders, will not be found for other test corpora, because more than the selection of individuals present at training, it is the "proximity" between voices in the training and test sets that may explain these observed differences. when varying the percentage of female-read books in training sets, we find that the global performance keeps the same range of accuracy, without any statistical significance. as individuals also change when varying the proportion of men and women in training data, we expected our wer distribution to vary accordingly. however, it is worth noting that our three data sets always include the 30% of women of the wper30 set and the 30% of men of the wper70 set. surprisingly, for the three systems studied, both the gendered proportion and the individuals change without inducing significant differences in overall performance. we observe that the expected impact is not in terms of overall performance but in terms of performance by gender. there is a higher error rate for female speakers when the system is mostly trained on male speakers but the significance of this difference between men and women decreases slowly as we raise the quantity of female-read books in the training set. however, we do not observe the inverse trend: only with the mono-gender system trained with women voices only, do we achieve better wer results for women than for men, even if this trend is not significant. all in all, we cannot conclude that the gender distributions in the training data have a strong influence on the wer results. while it appears that men's voices are generally better recognised, it seems that increasing the proportion of women's voices in the training corpus helps to reduce gender-differentiated performance, while ensuring the same level of overall performance. from this study performed on librispeech, it appears that i) the selection of individuals in the training corpus, ii) the gender distribution with extreme variations and iii) the train/test corpus match have a significant impact on system performance. in this very controlled context of speech production, the gender variation seems to be negligible compared to the individual variation. we believe gender demographics are not enough to ensure the same level of performance on both gender groups. according to our results, it seems that an overrepresentation of female voices improves recognition of women voices without decreasing overall performance. further research is needed to disentangle the effects of gender representation in voice and data and the performance of asr systems. if considering the gender balance in training data is a starting point for fairer systems, trying to quantify the intra-variability of our training sets to estimate a measure of adequacy with our test data appears as a strong lead for future work. we plan on working on acoustic measures such as fundamental frequency and speech rate to assess something that could be named "voice variability cover" and try to finally get out of the binary sex-matrix.