Self-supervised speech representations display some human-like cross-linguistic perceptual abilities (2024)

Abstract: State of the art models in automatic speech recognition have shown remarkable improvements due to modern self-supervised (SSL) transformer-based architectures such as wav2vec 2.0 (Baevski et al., 2020). However, how these models encode phonetic information is still not well understood. We explore whether SSL speech models display a linguistic property that characterizes human speech perception: language specificity. We show that while wav2vec 2.0 displays an overall language specificity effect when tested on Hindi vs. English, it does not resemble human speech perception when tested on finer-grained differences in Hindi speech contrasts.

## introduction human listeners become attuned to the speech sounds of their native language already in their first year of life @xcite @xcite . by adulthood, nonnative contrasts which they were once able to discriminate are no longer discriminable @xcite @xcite . acquiring a second language as an adult can thus be marred by difficulty acquiring certain phonetic contrasts. this language specificity effect is a core property of human speech perception. for human listeners, the difficulty of acquiring a particular non-native contrast depends both on whether the acoustic-phonetic dimension used to distinguish the contrasts is used in the native language and on the perceptual similarity of the nonnative categories to native categories (see @xcite . for example, while english only has two categories for the coronal stop series (/t/ and /d/), hindi has eight (/ã/,/ã h /,/ú/, /ú h /,/t "/,/t " h /,/d "/,/d " h /). because of the relationship between the acoustic dimensions used and the perceptual similarity to existing english categories, hindi contrasts that differ along aspiration (/t/ vs. /t h /) are easier for english listeners to acquire than along place of articulation (dental vs. retroflex; /t/ vs. /ú/) or voicing (/t/ vs. /d/) @xcite @xcite . whether computational models of speech perception share certain properties with humans has been a subject of recent interest. previous work has suggested that like humans, some speech models with non-transformer architectures display language specificity effects @xcite @xcite . however, less work has examined this effect in transformer architectures. @xcite suggest that self-supervised speech transformers do not display a cross-linguistic difference in predicting human performance, but their measures aggregate across all contrasts, and given the complex relationship between native and nonnative contrasts, this makes interpretation of the results difficult. in fact, previous work with nontransformer speech models found that for vowels, while the model displayed an overall language specificity effect, the direction of the effect was in the opposite direction than expected @xcite : a non-native model better predicted native speakers' discrimination. it is not known to what extent the specific perceptual similarity space in transformers is similar to humans. in this paper, we test whether self-supervised transformer speech models (wav2vec 2.0) display an effect of language specificity. we do so by examining specific patterns of cross-linguistic differences, using hindi contrasts as a case study. we explore a series of contrast that are known be be difficult for native english listeners. for these listeners, as noted above, place (/t/ vs. /ú/) and voicing (/t/ vs. /d/) are more difficult dimensions to discriminate along than aspiration (/t/ vs. /t h /) @xcite @xcite . these behavioral results provide a test case to explore targeted fine-grained categorization patterns of speech models to determine whether the models' representations are structured similarly to human listeners. in experiment 1, we find that wav2vec 2.0 displays an overall language specificity effect: a native-trained model performs better on native categorization task than a non-native model. in experiment 2, we examine specific contrasts where second language learners are attested to struggle, and find that both the native and non-native model show high accuracy in categorization across the most difficult dimensions for humans -place and voicing. we additionally find that where human listeners have been shown to have the least difficulty overall, the models show the largest cross-linguistic difference in accuracy. this suggests that wav2vec 2.0 is encoding language specific information, but structured in ways that differ from human listeners. ## conclusion in this work we explored both global and finegrained cross-linguistic patterns of categorization in wav2vec 2.0. we found that models perform better overall at test on a language they have been trained on, displaying a global language specificity effect similar to humans. however, when we examined specific contrasts differing along certain phonetic features, the models pattern differently than humans. this result provides evidence of fundamental differences in the structure of representations learned by wav2vec 2.0 and human listeners.