Contrastive Learning for Task-Independent S peech LLM -Pretraining (2025)

Abstract: Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs. To address these challenges, we propose a scalable, two-stage training approach: (1) A task-independent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a task-specific fine-tuning stage requiring minimal data. This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.

## introduction large language models (llms) have demonstrated impressive capabilities in language understanding and translation tasks @xcite . however, their reliance on written text limits their application in real-world scenarios, where communication often occurs in the form of speech. extending llms to spoken language could enable them to reason about and process spoken content effectively. a straightforward approach is to use a cascaded pipeline @xcite where speech is first converted to text using an automatic speech recognition (asr) model, and the text is subsequently processed by an llm. while simple, this approach has limitations: it cannot be trained end-to-end, asr errors can propagate through the pipeline, and critical paralinguistic information, such as speaker features or noise, is lost during transcription. to address these issues, recent works have proposed speechllms @xcite https://github.com/maikezuefle/ contr-pretraining 2024; @xcite , which connect speech encoders to llms via a trainable projector in an end-to-end manner. however, training them effectively remains a key challenge due to the scarcity of labelled data for downstream tasks @xcite . pretraining speechllms before finetuning on task-specific data can help overcome this challenge, reducing the need for large downstream datasets. a straightforward strategy is to use asr data, training on the asr task itself @xcite . however, this might lead to overfitting to this pretraining task @xcite . our work introduces contrastive learning as a task-independent pretraining strategy for speech-llms using asr data. contrastive learning aligns the representations of paired speech and text inputs while encouraging representations of unrelated inputs to remain distinct. we demonstrate that applying a contrastive loss to the representations after every layer and pretraining on just 400 hours of parallel speech-text data serves as an effective, task-agnostic foundation for speechllms. to evaluate our approach, we finetune the pretrained model on three downstream tasks: asr, speech translation (st), and speech question answering (sqa). we compare its performance against that of traditional asr pretraining and nextword prediction (nwp) pretraining using mixed speech-text input. the contrastive strategy not only surpasses the other methods but also matches the performance of task-specific state-of-the-art (sota) models on nearly all metrics. we also analyse combinations of the pretraining methods to explore potential complementary interactions. scaling up contrastive pretraining to 1,400 hours yields further performance gains, with our models surpassing specialized models and current speech-llms. even in simulated low-resource scenarios, using only 10% of the finetuning data, our method consistently outperforms existing speechllms. our speechllm architecture is simple yet flexible, consisting of a speech encoder, a projector, and an llm. only the projector is trained, making the model parameter-efficient and enabling maximum flexibility in adapting to the ever-evolving landscape of llms and speech encoders. moreover, this approach preserves the llm's text-to-text capabilities, ensuring its functionality across both speech and text-based tasks. an overview of our approach is provided in fig. 1 . the main contributions of our paper are: in this work, we use the latter approach, using a frozen speech encoder and llm while finetuning only the projection layers. this preserves the text-to-text abilities of the llm, while also being parameter efficient. text and speech alignment. while 'alignment' has various meanings @xcite , in this work, we use it to refer to the process of creating meaningfully similar representations for text and speech. one approach to achieve this alignment is to directly finetune on the desired tasks without explicitly modelling alignment @xcite ). an alternative is to use a downstream task to pretrain the speechllm before finetuning on desired tasks, for example pretraining on asr @xcite or asr variants @xcite . other works achieve alignment by encouraging paired speech and text embeddings closer together using an appropriate loss term. for example, @xcite use an l2 loss on speech and text em-beddings in addition to a kl-divergence loss on the output distributions, and @xcite use a cosine distance loss between paired embeddings. optimal transport has also been used to align paired embeddings in encoder-decoder speech translation systems @xcite . contrastive training is another widely used approach, bringing parallel text and speech representations closer while pushing non-parallel representations apart. it has been used in knowledge distillation for spoken language understanding @xcite , as an auxiliary loss for encoder-decoder st systems @xcite , and for music @xcite . it has also been applied to classification tasks, such as emotion recognition @xcite . other approaches mix text and speech in the input to help models learn both modalities. for example, @xcite align the output distributions of speech and mixed text-speech inputs, while @xcite and @xcite mix speech and text sequences for next-word prediction and question-answering tasks. in this work, we leverage the previous success of contrastive learning on speech models and apply it to speechllms. we show how contrastive learning can be applied successfully in this new scenario and combined with other losses. in contrast to other works, we experiment with using optimal transport for the contrastive loss function and apply the loss to several hidden layers of the speechllm. ## methods we propose contrastive learning as a taskindependent pretraining method for aligning speech and text representations, leveraging abundantly available asr data. we compare with two asrbased pretraining baselines: traditional asr pretraining and next-word prediction with mixed textspeech spans, which we also introduce as a pretraining approach in this paper. notation. let @xmath0, where each token is embedded into a h-dimensional space. the speech sequence is embedded into m vectors, with @xmath1. note that usually @xmath2. ## experiments our experiments follow a two-stage training strategy. first, we perform text-speech alignment pretraining using asr data to adapt the model to handle speech. in the second stage, we finetune the model jointly on a mix of task-specific data. this approach leverages abundant asr data to reduce reliance on large task-specific datasets, enabling effective adaptation even in low-resource settings. ## results the goal of our experiments is to find the best setup for using contrastive learning for speechllms. we investigate various contrastive pretraining losses and where to apply them, combinations with other pretraining losses, and the impact of additional data. we then compare our best-performing model to recent speechllms. finally, we investigate how text-speech alignment affects the detection of paralinguistic features. for st, and you et al. (2022) for sqa. metrics with * contribute to the normalized average as in eq. (3). ## conclusion we introduce a contrastive learning-based pretraining strategy for speechllms, aligning text and speech embeddings in a parameter-efficient manner. this approach establishes an effective taskagnostic foundation for speechllms and demonstrates state-of-the-art performance across multiple downstream tasks, even when little finetuning data is available. importantly, this pretraining method preserves the model's ability to understand paralinguistic information, offering a promising path forward for advancing speechllms.