Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach (2024)

Abstract: Recent progress in Spoken Language Modeling has shown that learning language directly from speech is feasible. Generating speech through a pipeline that operates at the text level typically loses nuances, intonations, and non-verbal vocalizations. Modeling directly from speech opens up the path to more natural and expressive systems. On the other hand, speech-only systems require up to three orders of magnitude more data to catch up to their text-based counterparts in terms of their semantic abilities. We show that fine-tuning speech representation models on phoneme classification leads to more context-invariant representations, and language models trained on these units achieve comparable lexical comprehension to ones trained on hundred times more data.

## introduction and related work recent advances in self-supervised speech representation learning (ssl) @xcite @xcite have enabled the development of label-free representations that are valuable for various downstream tasks @xcite . these representations can be discretized and treated as pseudotext, allowing for the training of language models directly from raw audio @xcite , which capture both prosody and linguistic content @xcite . applications of these audio-based language models include dialogue modeling @xcite , emotion conversion @xcite , and direct speechto-speech translation @xcite . they can be trained not only on discretized ssl representations but also on continuous word-size tokens @xcite or on a combination of acoustic and semantic tokens @xcite . however, these models still lag behind their text-based counterparts in terms of capturing semantics when trained with similar data quantity (nguyen et al., 2020), with scaling laws up to three orders of magnitude slower @xcite . recent approaches tackled this issue by jointly training speech and text language models (lms) @xcite @xcite or by using existing lms as a warm initialization @xcite . one hypothesis for the data inefficiency of spoken language models is that they must at the same time perform language modeling and process irrelevant acoustic variations.. recent works have addressed this issue for background noise @xcite , speech rate change @xcite , and speaker change @xcite @xcite . however, contextual variations due to coarticulation remain a challenge @xcite : ssl units align more closely with contextual phone states @xcite than with linguistic units @xcite , which may affect the lm's capacity to learn higher-order representations of language. we release the code and models at https:// github.com/bootphon/spokenlm-phoneme . ## conclusion we showed that fine-tuning ssl representations with a phoneme classification task is an effective and simple procedure to improve context independence. lms trained on these units achieve comparable lexical comprehension to models trained on hundred times more data. and we also found that initializing the embeddings of the discrete tokens of the lms with the centroids of the units further helps with lm scores. this shows that the units found are meaningfully placed relative to one another in this representation space. our work also highlights the trade-off between language modeling (which works best with abstract units), and speech generation (which works best with specific units). fine-tuning on phoneme classification can adjust this trade-off.