Do self-supervised speech models develop human-like perception biases? (2022)

Abstract: Self-supervised models for speech processing form representational spaces without using any external labels. Increasingly, they appear to be a feasible way of at least partially eliminating costly manual annotations, a problem of particular concern for low-resource languages. But what kind of representational spaces do these models construct?Human perception specializes to the sounds of listeners’ native languages. Does the same thing happen in self-supervised models? We examine the representational spaces of three kinds of state of the art self-supervised models: wav2vec, HuBERT and contrastive predictive coding (CPC), and compare them with the perceptual spaces of French-speaking and English-speaking human listeners, both globally and taking account of the behavioural differences between the two language groups. We show that the CPC model shows a small native language effect, but that wav2vec and HuBERT seem to develop a universal speech perception space which is not language specific. A comparison against the predictions of supervised phone recognisers suggests that all three self-supervised models capture relatively fine-grained perceptual phenomena, while supervised models are better at capturing coarser, phone-level effects, and effects of listeners’ native language, on perception.

## introduction recent advances in speech recognition and representation learning show that self-supervised pretraining is an excellent way of improving performance while reducing the amount of labelled data needed for training. for example, for the lib-rispeech dataset @xcite , the current best word error rates @xcite are obtained by systems based on the self-supervised wav2vec 2.0 model @xcite . systems using self-supervised pre-training, both using wav2vec 2.0 and using hubert @xcite , show excellent word error rates after having been fine-tuned on only ten minutes of labelled data. what is the effect of this self-supervised pretraining? what type of representational spaces are learned by these models? @xcite compared wav2vec 2.0, hubert, and contrastive predictive coding @xcite using an abx discriminability metric @xcite , demonstrating that all three models preserve and enhance linguistically relevant speech sound contrasts in the language they are trained on. we build on this work, asking how these representational spaces compare to the perceptual spaces of human listeners, as inferred from behaviour on phone discrimination experiments. human listeners develop speech perception biases under the influence of their native languages. for example, japanese native speakers tend to confuse the english sounds /r/ and /l/ @xcite ) (right and light in english will be perceived as the same or very similar), and english native speakers struggle with the french contrast /y/-/u/ @xcite , having difficulty perceiving the difference between words such as rue (/y/: "street") and roue (/u/: "wheel"). these misperceptions start to show early on in the native language acquisition process: infants older than 6 months exhibit a facilitating effect at discriminating sounds from their native language, but a decline at doing so for some non-native sounds @xcite . as the importance of this improvement for native sounds and this decline for non-native sounds seems to have a positive impact on infants' future language ability @xcite , having a perceptual space with native language biases is probably essential to perceive and understand correctly native speech in all situations (with environmental noises, speaker change, etc). if our goal is to have speech models that are as resilient and as adaptable as humans, it is thus interesting to see if they present the same native language specific biases. by measuring human listeners' ability to discriminate a variety of familiar and unfamiliar speech sounds, we can create a detailed profile of listeners' perceptual biases in the form of a set of sounds' discriminabilities. we then ask whether the training language influences self-supervised speech models in the same way that human listeners' native languages do. in order to study speech models' perception biases and compare them with humans', we use the perceptimatic benchmark datasets, foot_0 a collection of experimental speech perception data intended to facilitate comparison with machine representations of speech. as of this writing, perceptimatic contains french-and english-speaking participants' behaviour on discrimination tasks for phones in six different languages, for a total of 662 phone contrasts, along with the sound stimuli used during the experiments. as in @xcite , we test state-of-theart self-supervised models: wav2vec 2.0 @xcite , hubert @xcite ) and a cpc model @xcite . we train these models on english and french speech recordings (the native languages of the participants in perceptimatic). we compare the performance of these self-supervised models with a supervised asr model, deepspeech @xcite , trained on the same data but using phonemic labels. to study the degree to which the models' representational space is impacted by properties of speech per se, we also train the same models on recordings of acoustic scenes not including human vocalisations (environmental noises, animal sounds, music, and so on). we use mel-frequency cepstrum coefficients (mfccs) as an acoustic baseline. we show that: (1) self-supervised models trained on speech recordings are better than models trained on acoustic scenes (non-speech) to discriminate speech sounds and to predict human discrimination behaviour (2) they are good at predicting human discrimination behaviour at the stimuli level, but they are worse than neutral acoustic features when we average human results per contrast (3) they show very few native (training) language effect. all our code and data are freely available. foot_1 ## related work we are not the first to compare speech models' representational spaces with humans. @xcite used metamers as a tool to compare deep neural networks with humans. in a comparison between three speech recognition models, including a fine-tuned wav2vec 2.0 model, @xcite showed that wav2vec 2.0 was the best at matching human low-level psycho-acoustic behaviour. however, the model exhibited clear differences with respect to humans-showing, for example, heightened sensitivity to band-pass filtering and an under-reliance on temporal fine structure. to perform a comparison at a slightly higher level of speech perception, @xcite visualised a supervised asr model's internal representations of different speech sounds to investigate its adaptation to new ambiguous phone categories and compare it to humans' behaviour. multiple datasets containing human behavioural data have been collected and openly released to encourage comparison of models with humans. it is for this reason that the interspeech 2008 consonant challenge @xcite and the ollo database @xcite , containing humans' phone identification behaviour in different paradigms, were created. this is also the case for the datasets making up the perceptimatic database @xcite @xcite that we employ in this article, which were individually used to study less wellperforming models than the ones we use here. more than just informing us on the kind of information speech models learn, comparing them with humans can have a broader impact on our knowledge of how human perceive speech, and how they learn to do so. @xcite showed, for example, that a simple self-supervised speech model reproduces the reduced sensitivity to the english [r]/[l] contrast when trained on japanese speech recordings. pointing to the fact that the model used lacks abstract phone categories, the authors proposed an alternative to standard explanations of early phonetic learning in infants, as theories about this phenomenon rely heavily on the notion of phone categories. with a similar method, matusevych et al. (2020) tested the ability of various self-supervised speech models to reproduce infants' discrimination behaviour in multiple languages for a small set of supervised_models_perception_biases pairs of sounds. however, no quantitative comparison with behavioural data was made. within the same test framework, @xcite showed that a neural network trained to perform phone recognition was better at qualitatively reproducing japanese and english native speakers' discrimination behaviour than an hmm-gmm model, focusing once again on the [r]/[l] pair of sound and also on vowel length differences. in this paper, we decide to: (i) evaluate different self-supervised speech models on more contrasts than these previous works (ii) directly compare their results with human behaviour (iii) measure models' similarity to humans at the stimuli level on top of doing it at the contrast level. ## results in all graphs, statistical significance of comparisons is evaluated by bootstrapping over participants' results (@xmath0; redundant statistical comparisons are omitted for clarity (i.e. @xmath1. confidence intervals shown are 95% bootstrap intervals. ## discussion we showed that the self-supervised models we tested seem to learn representational spaces relevant for predicting human phone discrimination at the stimuli level. however, while humans show consistent discrimination behaviour for certain contrasts, whatever the stimuli, the self-supervised models we test do not capture systematic effects of contrasts between specific pairs of phones. unlike our supervised reference, their similarity to human perceptual spaces is limited to capturing the discriminability of specific individual stimuli. the models tested were similar, but wav2vec 2.0 showed a slight advantage for predicting this kind of behaviour. we have also shown that training on speech data is essential to obtaining a human-like perceptual space: for all of our metrics (abx accuracy or similarity to humans), training on speech leads to better results than training on acoustic scenes (nonspeech). this strongly suggests that the benefits of self-supervised speech models comes from learning characteristics of human speech, not simply the fact that they are better general audio features. we speculate that this is not just important to their ability to predict human speech perception and to discriminate phones, but also of their (related) utility for doing downstream tasks such as asr. what these models learn about speech, however, is not typically language-specific-at least, not in the same way that human perception is. wav2vec 2.0 and hubert do not model language-specific differences in human speech perception, and can be seen as modelling a language-neutral or universal speech perception space. indeed, they exhibit very few native language effect (see figure 4 and 5 ). we note that the idea of self-supervised models learning universal speech features is consistent with the fact that models trained on one language, or multilingually, have proven useful for representing speech in unseen languages @xcite . cpc does capture effects of native language on perception at the contrast level, but to a far lesser extent than our supervised reference when we focus on a subset of perceptimatic designed to capture important differences in discrimination behaviour for our two groups of participants (worldvowels). our cpc model differs from the other models tested in its small size, its causal architecture (wav2vec and hubert use transformers), and in that it does not use masking during its training. its architecture is probably the most biologically plausible of the three self-supervised models we tested. we should note, however, that it does not make it the best predictor of human discrimination behaviour among the three models (see figure 1 and 2 ). one possible explanation for the self-supervised models' limitations we observe is insufficiency of training data: the models in question have generally shown good performance on downstream tasks when pre-trained on large amounts of data. we tested this using available pretrained wav2vec and hubert models trained on much larger amounts of data. the detailed results can be found in appendix e. the models show a slight improvement, but, when looking at the ρ statistic at the phone contrast level, they are still worse than mfccs. contrary to previous results @xcite , our supervised reference system is quite good at predicting human discrimination behaviour (in particular at the contrast level), and clearly predicts a native language effect. the main differences in our experiment with @xcite are the type of model (deepspeech instead of hmm-gmm), and with @xcite ) the type of training objective (phone recognition rather than prediction of orthographic text), and the size of the training corpora (we use fewer data). predicting phones rather than orthography seems to be critical (as we demonstrate in appendix f), and using a neural network instead of a bayesian model (hmm-gmm) leads to a more human-like representational space, as already highlighted by @xcite . given the advantage supervised phone recognizers show, a different approach to developing more human-like representational spaces in selfsupervised models might be the inclusion of tasks or constraints that push them to take into account longer time scales in order to encourage them to construct longer, more phone-like units.