S par QL e: Speech Queries to Text Translation Through LLM s (2025)

Abstract: With the growing influence of Large Language Models (LLMs), there is increasing interest in integrating speech representations with them to enable more seamless multi-modal processing and speech understanding. This study introduces a novel approach that combines self-supervised speech representations with instruction-tuned LLMs for speech-to-text translation. The proposed approach leverages a modality adapter to align extracted speech features with instruction-tuned LLMs using English speech data. Our experiments demonstrate that this method effectively preserves the semantic content of the input speech and serves as an effective bridge between self-supervised speech models and instruction-tuned LLMs, offering a promising approach for various speech understanding applications.

## introduction progress in speech processing has been accelerated by the introduction of self-supervised learning (ssl) methods that utilize large amounts of unlabeled speech data, which established new benchmarks in the field @xcite @xcite . continuous representations and/or discrete units derived from self-supervised models have been used to extract relevant latent features from speech data and improve performance in downstream tasks, including speech recognition @xcite , speech synthesis @xcite , speech translation @xcite and general speech understanding @xcite . progress in text processing has also been accelerated by the emergence of pretrained large language models (llms), which enabled new applications such as few-shot/zeroshot language processing @xcite @xcite and multi-modal processing @xcite . recent efforts in speech understanding explored the possibility of incorporating speech representations directly into llms @xcite @xcite . multi-modal speechlanguage models signify a shift in both speech and natural language processing. by incorporating speech data, llms can enhance their contextual grasp, providing a deeper and more thorough representation of spoken language. in addition, the existing multilingual functionalities of llms can be leveraged to enhance speech processing applications, such as speech translation, without additional dedicated training. in this work, we describe an efficient method to query instruction-tuned llms using speech input. the model aligns speech features extracted through self-supervised learning (ssl) with llms using only a modality adapter trained with english data and a small portion of translated text. we demonstrate the generalization of translation performance across both seen and unseen target languages. we call our approach sparqle foot_0 . sparqle is inspired by querying transformer modules used in vision language models to bootstrap vision-language representations from frozen image encoders @xcite . we demonstrate through speech translation that sparqle enables the integration of existing pre-trained speech encoders and llms without the need for updating the parameters of either speech encoder or llm. in contrast to previously explored speech-llm integration approaches, our method is the first to utilize frozen ssl speech representations, without relying on large pre-trained asr models like whisper @xcite . we experimentally demonstrate the effectiveness of this relatively simple approach and release both the pre-trained and fine-tuned models foot_1 . method speech encoder (param.) language model (param.) adapter (param.) tasks (chen et al., 2024) nemo (0.6b -1.1b) megatronllm (40b-1t) lora (14m-94m) / conformer (115m) multitask (wang et al., 2023a) whisper (74m-1.5b) llama (6.7b-65.2b) conv.layers (4m) alignment (wang et al., 2023c) usm (2b) mt0-mt xxl (13b) adapter (156m) multitask (wang et al., 2023d) ctc encoder (220m) t5 xxl (11b) -rag speech2text, speech2entity retriever multitask (yu et al., 2024) whisper (1.5b) vicunallm (13b) fc 3 (24m) / mhsa 4 (133m) / seg-q-former (24m) asr (tang et al., 2024) whisper (1.5b) + beats (90m) vicunallm (13b) lora + seg-q-former (33m) asr/multitask (das et al., 2024) wavlm (316.62m) flan-t5-xl (2.85b) cnn + lora(14m-94m) multitask (chu et al., 2024) whisperlarge (1.5b) qwen (7b) -multitask sparqle hubert (316m) llama3 (8b) q-former (187m) ast/multitask the availability of instruction-tuned llms @xcite @xcite opened a new research direction for speech processing by connecting speech directly to these multi-task models. @xcite proposed multitask speech-language modeling with unified llm framework that shows in-context learning ability. @xcite utilized three approaches for adapting speech to text modality: fully connected linear layers following @xcite adapter method, multi-head cross attention mechanism described in @xcite , and query transformer @xcite . for processing speech input, they utilized two models: whisper large-v2 @xcite and beats @xcite . the speechverse @xcite ) framework used wavlm-based @xcite speech encoder interfaced with a flan-t5-xl (chung et al., 2024) language model. in a another study, @xcite demonstrated the sufficiency of a single linear layer for speech-llm integration in asr, albeit with limited exploration beyond this task. speech as language modeling was also studied in speechgpt @xcite which integrates both speech and text modalities. the model incorporates a speech tokenizer that converts raw audio waveforms into discrete speech tokens, enabling efficient processing within the transformer architecture. through multi-task fine-tuning on downstream tasks such as asr, translation, and generation, the model demonstrates remarkable versatility. qwen2-audio @xcite , designed as a general-purpose audio understanding model, exhibiting broad applicability across various audio-related tasks. the model employs self-supervised learning techniques, such as masked audio modeling and contrastive learning, to capture rich audio representations. lora rank is the number of linearly independent rows or columns in a parameter (weight) matrix; a lower rank means approximating a large weight matrix with fewer parameters to simplify and speed up training. the number of additional parameters can be roughly estimated as the initial hidden dimension multiplied by the rank and then multiplied by two to account for all added parameters. in ## model sparqle is a parameter efficient model designed to extract information from speech representation and route them to query pre-trained open-sourced llms, without modifications to the underlying speech encoder or llm. motivated by the success of multi-modal representations in vision language modeling @xcite , we propose the adoption of speech representations to llms for generative tasks, specifically automatic speech translation (ast). we pre-trained our model using english data first and fine-tuned with mix of english and french. we used hubert @xcite as the speech encoder. the output from its final hidden layer is fed into the query adapter. a query adapter incorporates query tokens, which are special tokens (placeholders) added to the input of a speechlanguage model. they do not correspond to specific llama speech encoder modality adapter speech regions, but are meant to extract information from the whole speech sequence in a flexible way. the final features from the query tokens are passed to a large language model to generate natural language responses. figure 1 shows the overall structure of the system, which consists of three main parts: a pre-trained speech encoder, a bridging mechanism (sparqle) and a text generator. in our experiments, we employed llama3 (ai@meta, 2024) as the text language model. ## discussion we introduced a framework for efficient routing of ssl speech features to query llms, and demonstrated its effectiveness in speech translation tasks. the results indicate that the proposed model and training paradigm result in generalized performance and avoid instruction over-fitting; the model was able to adhere to instructions for translating speech into multiple target languages (see figure 3 ). sparqle demonstrates ability to translate speech input into diverse languages not encountered during our fine-tuning stage, such as german, russian, arabic, etc. finally, with the appropriate prompts, the instruction-tuned model is capable of performing multiple tasks in a single run, (see figure 4 in section 5.1). the performance in speech translation shows promising results, where the proposed approach outperformed both weak and strong baselines from @xcite in both french and german. you are a speech-to-text conversion model. your tasks include accurately transcribing spoken language and translating audio samples as per user instructions. please ensure clarity and precision in both transcription and translation processes. back in new york, i am the head of development for a non-profit called robin hood. en nouvelle-york, je suis la tête du développement pour un organisme à but non lucratif appelé robin des bois. please let me know if you need any further assistance! <@xmath0 ## conclusion & future work in this short paper, we demonstrate the performance of the proposed sparqle model, an aligned speechto-text model based on ssl features, for speech translation applications. what we have demonstrated in this study is only a subset of potential applications of this method. the sparqle model can potentially handle both text and speech modalities, and can be applied for any speech-to-text applications. as demonstrated, our model outperforms existing speech translation baselines from iwslt 2022 challenge, which demonstrates the potential of transferring the inherent capacities of llms into speech tasks using a parameter-efficient approach. future work can explore the generalization of the model to other languages and speech understanding tasks and analyze the characteristics of the resulting queries.