Slamming: Training a Speech Language Model on One GPU in a Day (2025)

Abstract: We introduce *Slam*, a recipe for training high-quality Speech Language Models (SLMs) on a single academic GPU in 24 hours. We do so through empirical analysis of model initialisation and architecture, synthetic training data, preference optimisation with synthetic data and tweaking all other components. We empirically demonstrate that this training recipe also scales well with more compute getting results on par with leading SLMs in a fraction of the compute cost. We hope these insights will make SLM training and research more accessible. In the context of SLM scaling laws, our results far outperform predicted compute optimal performance, giving an optimistic view to SLM feasibility. See code, data, models, samples - https://pages.cs.huji.ac.il/adiyoss-lab/slamming .

## introduction speech language models (slms) have gained significant interest from researchers @xcite @xcite , demonstrating remarkable performance in traditional speech tasks @xcite , diverse generative applications @xcite , and reasoning over speech and audio signals @xcite . slms can generally be classified into two main categories: (i) generative speech language models (lms) (which can also incorporate text) and (ii) speech-aware lms. the first category follows a similar pre-training approach to text-based large language models (llms), directly maximising the likelihood of speech considering both input and output, typically by representing audio as a sequence of discrete tokens. the second category consists of pre-trained text lms adapted to process speech inputs. in this work, we focus on the first. training high-quality slms can be highly resource intensive @xcite @xcite @xcite . for example, @xcite trained their slm on approximately 570k hours of speech data, while @xcite utilised around 7m hours. additionally, @xcite proposed slm scaling laws, suggesting that training high-quality slms requires ∼ 3x more data compared to text-based counterparts. these computational demands restrict the required fundamental research aimed at enhancing slms, such as advancements in speech tokenisation, efficient acoustic modelling, etc. in the natural language processing (nlp) community, numerous studies have investigated efficient model training techniques, including masked language models such as cramming @xcite and modernbert @xcite , along with next-token prediction llms such as mobilellm @xcite . these methods include implementation efficiencies, architectural improvements, data selection strategies, and enhancements to the overall training pipeline. inspired by cramming @xcite ) in text, we investigate compute-limited slm training, which we term slamming. we pose the question: is it possible to train high-quality slms using a single gpu within 24 hours? for that, we conduct an extensive empirical analysis exploring how different training components influence performance. from this, we derive a training recipe that maximises model performance within a fixed compute budget. specifically, we investigate the impact of model initialisation and architecture, various optimisers and learning rate schedulers, data selection strategies -including the role of synthetic data, text-interleaving and preference optimisation. we believe that developing these training strategies and proving their feasibility will empower the speech and audio research community to advance slms beyond the scope of large, well-funded academic and industrial labs. figure 1 illustrates the performance of various slms relative to their training compute budget, with circle sizes representing the size of the models. furthermore, we compare our results with the scaling performance predicted from @xcite . although the authors present a somewhat pessimistic view of the computational resources needed to train highquality slms, we empirically show that reality is more promising, demonstrating that it is possible to significantly exceed the predicted performance per unit of compute. we encourage the community to refine and expand scaling laws specifically tailored for slm training across various settings. ## setup in this study, we explore decoder-only generative slms, which aim at maximising the likelihood of speech samples represented as discrete tokens. we examine both purely speech-based slms trained on speech tokens and joint speech-text slms using interleaving strategies @xcite . similarly to @xcite ; @xcite , we obtain speech tokens by quantising continuous latent representations of a self-supervised speech representation model using the k-means algorithm, often known as semantic tokens. specifically, we utilise a multilingual hubert @xcite model running at 25 hz, as employed in @xcite . we then train slms by minimising the negative log-likelihood of the input segments. unless mentioned otherwise, all slms are trained using a single a5000 gpu (24gb vram) along with 16 cpu cores for 24 hours. we deliberately focus on this constrained compute budget, assuming that most academic labs can access similar resources, thereby ensuring the accessibility of our research. the training data is pre-processed, i.e. extracting hubert units and dividing data into chunks, and stored prior to model training. as a result, this pre-processing time is excluded from the compute budget. this approach, aligned with @xcite , is practical since many research experiments utilise the same pre-processed data. we additionally do not count the time for running validation and visualisations as they are not used as part of the optimisation pipeline and only used for demonstration purposes. evaluation metrics. we assess all slms using five distinct evaluation metrics. the first three are based on likelihood evaluation, while the fourth and fifth are generative metrics. for likelihood based modelling we consider sblimp @xcite , spoken story-cloze (ssc)), and topic story-cloze (tsc) @xcite . for modellinglikelihood metrics, we evaluate the likelihood assigned by the slms to pairs of speech utterances, consisting of a positive example and a distractor. we calculate the percent of pairs in which the slm assigns higher likelihood to the positive sample. sblimp focuses on grammatical abilities thus the negative is ungrammatical version of the positive. ssc and tsc focus on semantic modelling abilities. in ssc, the distractor suffix is taken from the original textual storycloze dataset @xcite , allowing to assess fine-grained semantic speech understanding. in tsc, however, the distractor suffix is drawn from a different topic, enabling us to evaluate the model's ability to understand the overall semantic concept. to assess the generative abilities of slms, we compute generative perplexity (genppl). following the approach of @xcite ; @xcite , we provide the slm with a short speech prompt and generate speech tokens continuation. we use unit-vocoder with duration prediction to convert the tokens into speech @xcite . the generated speech is then transcribed, and its perplexity (ppl) is evaluated using a pre-trained text llm. to minimise the impact of token repetition on ppl measurements, we ground the generated text using diversity metrics derived from the auto-bleu score @xcite . similarly to @xcite we use bigram auto-bleu. in other words, we ensure that all models achieve similar auto-bleu scores, allowing for a fair comparison of ppl. specifically, we transcribe speech segments using whisperlarge-v3-turbo model @xcite and measure ppl using llama-3.2-1b model @xcite . we calculate genppl on correct samples from the spoken story-cloze dataset. finally, for our final models, we also compute gptscore. given a speech prompt and a generated continuation, we transcribe both and use gpt-4o to judge the quality of the continuation given the prompt, on a scale of 1 to 5. we follow the same setup and prompt as @xcite for the metric. we use this metric as the final form of evaluation, as it is the most costly to run. ## investigations with this setup, we systematically analyse and ablate each component of the training pipeline, ultimately refining an optimised cook-book for training slms. we specifically examine the influence of model family, initialisation, size, and architectural choices (e.g., dropout, positional embedding, etc.). we analyse optimisation parameters and data characteristics. lastly, we explore alternative training objectives beyond standard next-token prediction, including speech-text interleaving and direct preference optimisation using synthetic data. ## final recipe building on these empirical findings, we develop the final slam recipe. using it, we train slms based on qwen2.5-0.5b. we then compare slam to the twist model family across various sizes: 350m, 1.3b, 7b, and 13b. we also present results for twist-350m using our computational constraints but following twist's original training recipe, along with our synthetic data. finally, we report results for the top-performing model from @xcite , including their predicted optimal performance under our compute budget based on slm scaling laws. results are reported in transcribed generated examples by slam can be seen in appendix d.4. to show that the slam models do not overfit a single domain (audiobooks/stories), we provide results for genppl on a different domain. this can be seen in appendix d.3. we further evaluate the quality of the generated audio using mosnet @xcite , similarly to align-slm. results are presented in appendix d.2. as the quality of the generated audio is mainly affected by the vocoder, which is identical across evaluated methods, results are comparable. interestingly, twist 1.3b and twist 7b achieve slightly worse scores. ## increasing compute similarly to geiping and goldstein (2023), we analyse whether the proposed approach holds well also in increased compute budget. we opt for 48 hours on 2 a100 gpus as a reasonable academic budget for larger scale tests, and represents ∼ 10 times more compute than the slamming setting. we use exactly the same slam recipe for more steps, and increase the batch size times 2. we provide the full results in we also wish to assess whether our suggested recipe holds for larger models, thus we evaluate training a larger qwen2.5 text lm as the base model. we use qwen2.5 -1.5b for the same compute budget as above -i.e two a100 gpus for 48 hours. all training details are identical, but of course the larger model was trained for less steps (and tokens). we provide results from this model, denoted slam (large) in ## limitations while the slms trained under slamming compute budget performed notably well compared to other slms trained with much more compute they might perform less well in other areas. for instance, evaluating their abilities on acoustic or prosodic elements as in salmon @xcite could show further challenges of low resource settings. furthermore, we focus in this study on the well used hubert @xcite model as a tokeniser, and while we do not make any adjustments specifically for it, future work might wish to investigate our cramming approach with new tokenisers, such as mimi @xcite and sylboost @xcite . ## conclusion in this work we show that training high quality slms with a very modest compute budget, is feasible. we give these main guidelines: we hope that these insights, and open source resources will be of use to the community in furthering slm research.