Scaling Properties of Speech Language Models (2024)

Abstract: Speech Language Models (SLMs) aim to learn language from raw audio, without textual resources. Despite significant advances, our current models exhibit weak syntax and semantic abilities. However, if the scaling properties of neural language models hold for the speech modality, these abilities will improve as the amount of compute used for training increases. In this paper, we use models of this scaling behavior to estimate the scale at which our current methods will yield a SLM with the English proficiency of text-based Large Language Models (LLMs). We establish a strong correlation between pre-training loss and downstream syntactic and semantic performance in SLMs and LLMs, which results in predictable scaling of linguistic performance. We show that the linguistic performance of SLMs scales up to three orders of magnitude more slowly than that of text-based LLMs. Additionally, we study the benefits of synthetic data designed to boost semantic understanding and the effects of coarser speech tokenization.

## introduction inspired by the remarkable ability of preschool children to learn language from raw sensory inputs, @xcite introduced in their seminal paper the textless nlp (natural language processing) project. the project aimed to leverage advances in self-supervised speech representation learning for unsupervised unit discovery @xcite and generative neural language models @xcite to jointly learn the acoustic and linguistic characteristics of a language from audio alone, without access to textual supervision (e.g. lexicon or transcriptions). they formalized this goal in the task of generative spoken language modeling (gslm), in which a language model is trained on sequences of selfsupervised learned speech units. beyond bridging the gap between human and machine language acquisition, the textless nlp project hoped to democratize access to nlp technologies by extending them to the millions of users of languages with little or no textual resources (e.g. due to a lack of standardized orthography). these languages are unlikely to be supported by current technologies, which are heavily dependent on massive volumes of text data. in today's landscape, where nlp-based ai systems are becoming increasingly relevant and pervasive, it is all the more pressing to expand their inclusivity by building speech-based systems that can match the capabilities of their text-based counterparts. despite a significant body of research on these speech-based language models (slms) @xcite @xcite , they are still far from matching the syntactic and semantic abilities of text-based systems @xcite . therefore, the promise of textless nlp is yet to be realized. however, if the scaling behavior of text-based neu-ral language models @xcite holds for the speech modality, we can reasonably expect those abilities to improve as the amount of compute used for training increases. in this work, we apply recently proposed models of the scaling behavior of neural language models to slms, and use them to estimate the scale at which our current methods will match the linguistic performance of large language models (llms), generative text-based systems that have achieved remarkably strong performance across a wide range of nlp applications @xcite . the main contributions of this work are: the training source code, data, and models will be released at https://github.com/ tiagocuervo/slm_scaling . ## related work previous works have studied the scaling behavior of neural networks on speech applications. @xcite showed that acoustic models trained with an auto-predictive coding loss follow similar power laws to those observed in neural lms. @xcite used the scaling laws from @xcite to model the scaling behavior of the upstream loss of neural lms on multiple modalities, including speech. they used a speech tokenizer with higher framerate (50 hz) and vocabulary size (@xmath0.1). such fine-grained tokenizers capture a lot of the paralinguistic information in speech @xcite . therefore, their speech tokens can be considered almost a different modality due to the acoustic variance. furthermore, they do not study the behavior with scale of downstream performance. in this work, we focus on the linguistic content of the signal. as reported by hassid ## discussion our work showed that the upstream and downstream linguistic performance of our current methods for gslm scales predictably with compute. this suggests that, with sufficient computational resources, the goal of the textless nlp project-achieving neural lms trained exclusively on speech, and matching the linguistic proficiency of their text-based counterparts-is achievable. however, the cost of such models could be prohibitive, as we estimate that they will require up to three orders of magnitude more compute than a text-based llm to achieve equivalent performance. we believe this points to the need for leveraging the rich language representations already learned by text llms. this seems to be the current trend in the community, as several recent works have sought to improve slms through transfer learn-ing from text-based models @xcite @xcite . however, considering one of the grand goals of the textless nlp project-extending the benefits of large-scale language modeling to low-resource or non-written languages-we will have to address the question of how knowledge transfer from text llms performs when the speech data is in a different language than the one the text llm was trained on. if cross-lingual knowledge transfer between text and speech modalities proves to be unfeasible, then purely speech-based slms, such as the ones studied here, could still offer a compelling solution for low-resource languages. we explored the use of synthetic data and coarser tokenization to increase the semantic abilities of slms. our synthetic dataset improved semantic performance, but using a coarser tokenization led to overall degradation of downstream performance. we do not have yet an hypothesis for why coarser tokens degrade performance, as this seems counter-intuitive, and contradicts the findings on other speech applications @xcite . we leave this as an interesting issue to address in future work. moreover, we believe that working on methods that allow to increase the information density per context-window of slms holds promise to improve their scaling behavior. ## limitations any extrapolation from our models of the scaling behavior of slms should be considered optimistic for the following reasons: 1) our models for downstream performance ignore the fact that the metrics saturate. as observed in text llms, the improvements with scale slow down as performance approaches the saturation value. it is likely that, due to saturation, the compute required to yield a particular performance will be larger than predicted. moreover, due to the lower density of linguistic information per context window in slms relative to llms, the saturation values of the metrics may be lower for slms. 2) the llms from the pythia suite that we used in this study are likely overtrained (all models were trained with ∼300b tokens). optimally trained llms (according to equation 6 ) should show better performance with scale, and therefore widen the gap with the scaling efficiency of slms. 3) the envelope of minimal loss per flop (figure 1 ) might show a slight negative curvature at larger scale @xcite , reducing the scaling efficiency. @xcite note that the scaling law coefficients for text llms, and consequently the optimal compute allocation, can vary depending on the training datasets used in the scaling study. commonly used text datasets are significantly larger and more diverse than the academic speech datasets typically used for gslm, such as those in this study. as a result, these speech datasets represent a more biased sample of the overall distribution of speech data, making scaling laws derived from them less likely to generalize. therefore, we cannot guarantee that the scaling laws we have developed will be universally applicable to other datasets. however, we do not expect significant deviations that affect the conclusions here presented. future research could explore validating the predictions from this study on larger and more diverse datasets, such as the recently released yodas @xcite . ## conclusions we have trained a large set of slms with different compute budgets and studied the scaling properties of their upstream and downstream performance using recently proposed models of scaling laws for neural lms. the obtained models allow practitioners to optimally allocate compute to attain a specific loss, syntactic, and/or semantic performance. we showed that the pre-training loss and downstream linguistic performance of slms and llms is highly correlated, and both scale predictably according to power laws. this allowed us to compare the scaling properties of slms and llms, from which we established that the linguistic abilities of slms scale up to three orders of magnitude more slowly. additionally, we proposed a new speech dataset, stinystories, and showed that its use during pre-training improves downstream seman-tic performance. finally, we explored the use of coarser speech tokenization as a method to increase the amount of tokens per context window in slms, but obtained worse downstream performance.