On the Nature of Discrete Speech Representations in Multilingual Self-supervised Models (2023)

Abstract: Self-supervision has emerged as an effective paradigm for learning representations of spoken language from raw audio without explicit labels or transcriptions. Self-supervised speech models, such as wav2vec 2.0 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021), have shown significant promise in improving the performance across different speech processing tasks. One of the main advantages of self-supervised speech models is that they can be pre-trained on a large sample of languages (Conneau et al., 2020; Babu et al.,2022), which facilitates cross-lingual transfer for low-resource languages (San et al., 2021). State-of-the-art self-supervised speech models include a quantization module that transforms the continuous acoustic input into a sequence of discrete units. One of the key questions in this area is whether the discrete representations learned via self-supervision are language-specific or language-universal. In other words, we ask: do the discrete units learned by a multilingual speech model represent the same speech sounds across languages or do they differ based on the specific language being spoken? From the practical perspective, this question has important implications for the development of speech models that can generalize across languages, particularly for low-resource languages. Furthermore, examining the level of linguistic abstraction in speech models that lack symbolic supervision is also relevant to the field of human language acquisition (Dupoux, 2018).

## overview and research question self-supervision has emerged as an effective paradigm for learning representations of spoken language from raw audio without explicit labels or transcriptions. self-supervised speech models, such as wav2vec 2.0 @xcite and hu-bert @xcite , have shown significant promise in improving the performance across different speech processing tasks. one of the main advantages of self-supervised speech models is that they can be pre-trained on a large sample of languages @xcite , which facilitates cross-lingual transfer for low-resource languages @xcite . state-of-the-art self-supervised speech models include a quantization module that transforms the continuous acoustic input into a sequence of discrete units. one of the key questions in this area is whether the discrete representations learned via self-supervision are language-specific or languageuniversal. in other words, we ask: do the discrete units learned by a multilingual speech model represent the same speech sounds across languages or do they differ based on the specific language being spoken? from the practical perspective, this question has important implications for the development of speech models that can generalize across languages, particularly for low-resource languages. furthermore, examining the level of linguistic abstraction in speech models that lack symbolic supervision is also relevant to the field of human language acquisition @xcite . ## approach to answer our research question, we conduct a series of experiments with spoken language identification (slid) as a probing task. our intuition is that if we can accurately predict the language of a short speech sample (∼10 sec) from its discretized representation, this would suggest that the model has learned language-specific discrete units that are unique to each language. on the other hand, a difficulty in predicting the language would suggest that the model has learned a common set of discrete units that are shared across multiple languages. experimental data. we use a balanced subset of the common voice speech corpus @xcite consisting of 16 languages that span diverse sub-groups within the indo-european language family, namely: romance (catalan, portuguese, french, spanish, italian), germanic (german, dutch, swedish, frisian), slavic (ukrainian, russian, polish), celtic (welsh, breton), hellenic (greek), and indo-iranian (persian). our language sample exhibits a considerable degree of typological diversity with respect to various phonological features, including the consonant-vowel ratio, which is high in russian but low in german, french, and swedish @xcite . in addition, stress location patterns are highly variable in russian and spanish, but fixed in languages such as greek, persian, and welsh @xcite . we use ∼6.75, ∼3.75, ∼4.25 hours per each language for training, validation, and evaluation sets, respectively. a speech sample in our study is an utterance of a few seconds of read speech. ## preliminary results activated discrete units. first, we find that the set of activated units are nearly identical across the languages in our study, which implies that the models do not learn units that are predictable features of the identity of the spoken language. ## conclusion we summarized the findings of our experiments whereby we investigate the nature of the discrete units in multilingual, self-supervised speech models. we employed language identification as a probing task and demonstrated the difficulty of predicting the language of an utterance from its discretized representation. our findings support the hypothesis that latent, discretized speech representations in self-supervised models correspond to sub-phonetic events that are shared across the world's languages, rather than language-specific, abstract phonemic categories.