W av2 P rompt: End-to-End Speech Prompt Learning and Task-based Fine-tuning for Text-based LLM s (2025)

Abstract: Wav2Prompt is proposed which allows integrating spoken input with a text-based large language model (LLM). Wav2Prompt uses a straightforward training process with only the same data used to train an automatic speech recognition (ASR) model. After training, Wav2Prompt learns continuous representations from speech and uses them as LLM prompts. To avoid task over-fitting issues found in prior work and preserve the emergent abilities of LLMs, Wav2Prompt takes LLM token embeddings as the training targets and utilises a continuous integrate-and-fire mechanism for explicit speech-text alignment. Therefore, a Wav2Prompt-LLM combination can be applied to zero-shot spoken language tasks such as speech translation (ST), speech understanding (SLU), and spoken-query-based question answering (SQQA). It is shown that for these zero-shot tasks, Wav2Prompt performs similarly to an ASR-LLM cascade and better than recent prior work. If relatively small amounts of task-specific paired data are available, the Wav2Prompt-LLM combination can be end-to-end (E2E) fine-tuned and then yields greatly improved results relative to an ASR-LLM cascade for the above tasks. For instance, for English-French ST, a Wav2Prompt-LLM combination gave a 5 BLEU point increase over an ASR-LLM cascade.

## introduction text-based large language models (llms) @xcite @xcite have achieved remarkable performance in a wide range of natural language processing (nlp) tasks @xcite . llms are trained on huge quantities of text and are highly flexible. they are able to be applied to a range of tasks for which they have not been explicitly trained, known as the emergent abilities of llm @xcite . to further expand the use-cases of llms, it is important to enable llms to handle other modalities including spoken input. the conventional approach is to use an automatic speech recognition (asr) model to transcribe speech into text, which is then used as the llm input. however, this cascaded system suffers from error accumulation and can not be end-to-end (e2e) fine-tuned. @xcite . many studies have explored connecting llms directly to the speech acoustic encoder (encoder-llm) for various speech tasks such as asr or speech translation (st) @xcite @xcite . however, these approaches restrict the system to a specific task, thereby losing the ability of llms to handle a wide range of zero-shot spoken language tasks. recent work has begun to explore ways to restore the zero-shot capabilities of llms. work in this area includes approaches that make use of audio or speech-based question-answering (qa) tasks @xcite , as well as the introduction of additional steps in the training pipeline, such as multi-task training @xcite , instruction tuning @xcite @xcite , and activation tuning @xcite . however, these approaches greatly complicate model training. unlike previous work, this paper enables textbased llms to understand speech and perform various untrained speech tasks using a straightforward single-task training process using only easily accessible asr data. wav2prompt allows spoken input to be integrated with an off-the-shelf text-based llm. after training using asr data (i.e. speech and associated text transcript), wav2prompt generates representations from speech and uses them as llm prompts for downstream tasks. it allows the wav2prompt-llm combination to work well in a range of zero-shot spoken language tasks. however, wav2prompt can also give much improved performance when limited task-specific spoken lan-guage data is available through e2e fine-tuning, without updating the llm. wav2prompt serves as an e2e alternative for the asr-llm cascade. wav2prompt takes llm token embeddings as training targets to naturally maintain the zero-shot capability of text-based llms. this is a key difference to the conventional asr task, which takes discrete text tokens as the only target and thus leads to the asr-llm cascade approach. however, learning llm token embeddings is challenging for speech models given the difference in input sequence length between speech and text. wav2prompt addresses this issue using a continuous integrate-and-fire (cif) @xcite mechanism to generate a label-level speech representation and a mean squared error (mse) loss can be used to enforce consistency with the llm token embeddings. wav2prompt can therefore be combined with a text-based llm not only for zeroshot speech tasks, but also for e2e fine-tuning with limited task-specific data, which is a key advantage compared to an asr-llm cascade. this paper evaluates wav2prompt on diverse spoken language tasks including speech translation (st), spoken language understanding (slu), and spoken-query-based question answering (sqqa), all of these are unseen during training as wav2prompt only uses asr training data. the results show that the wav2prompt-llm combination could achieve similar performance to the asr-llm cascade in zero-shot cases and greatly surpasses the existing encoder-llm method @xcite . in scenarios with limited task-specific available data, after e2e fine-tuning, wav2prompt shows improved performance over the asr-llm cascade for all of these tasks. the main contributions of this paper can be summarised in three main parts: ## related work text-based large language model the evolution of text-based llms, exemplified by a large increase in model parameters and training data seen in gpt-3 @xcite and palm (chowdhery et al., 2023), has revolutionised nlp tasks. this progress has facilitated the development of advanced models such as gpt-4 @xcite , showcasing the remarkable capabilities of llms in various domains. alongside these advancements, "smaller" llms like llama @xcite have been introduced, achieving a better balance between performance and computational resources. there are several variant models such as vicuna @xcite developed from conversation-based fine-tuning and multi-lingual llms, e.g. bloom @xcite . one key aspect of llms is that they can exhibit remarkable performance in a range of tasks on which they have never been explicitly trained. examples include zero-shot task transfer @xcite and few-shot learning @xcite . this is sometimes known as the emergent abilities of llms @xcite . speech-enabled large language model while discrete speech tokens have been explored to build spoken generative lms @xcite , this paper focuses on utilising off-theshelf text-based llms. recently, several studies have worked on building speech-enabled llms to support direct speech input @xcite @xcite @xcite . since the speech input sequence is much longer than the corresponding text, different strategies have been studied for downsampling. @xcite @xcite stacks the acoustic encoder output to achieve a fixed-rate length reduction. @xcite treats multiple modalities as foreign languages, in which cif is used to obtain the speech representation. @xcite explored the use of q-former @xcite which transforms input sequences of varying lengths into fixed-length outputs. however, the emergent abilities of llm are lost in this task-specific training. unlike work such as @xcite , which focus on enabling llms to handle multi-modal inputs including speech, this paper focuses on maintaining the zero-shot abilities of llms. some recent work explored regaining the zero-shot abilities of llms. in @xcite , an instruction tuning stage followed by an activation tuning stage was introduced after pre-training to alleviate task over-fitting. in addition, @xcite simulated a speech qa dataset from asr data, finding that a speech-enabled llm trained on it could handle spoken qa tasks and potentially other tasks like st. prior work extended speech capabilities for llm, but they come at the cost of increased complexity and extensive resources based on full-parameter or parameter-efficient training. since text-based llms have a fundamental connection with speech, wav2prompt focuses on training a model that can be combined and e2e fine-tuned with a text-based llm through a straightforward process while keeping the llm fixed. prompt tuning fine-tuning llms can be expensive. as an alternative, the prompting technique fixes all llm parameters and uses a prompt to query llms @xcite . early prompting uses simple keyword-based inputs or fill-inthe-blank style prompts @xcite . for generative llms, natural language prompts can be used @xcite . however, these discrete prompts can result in sub-optimal performance in numerous cases @xcite . instead, prompt tuning adds trainable continuous embeddings, i.e. continuous prompts, to the original input token embedding @xcite . during training, only the parameters of the continuous prompts are updated @xcite . this paper follows the prompt tuning approach, updating only the parameters of wav2prompt while keeping the llm fixed when e2e fine-tuned on limited task-specific data. ## analysis of task over-fitting task over-fitting @xcite occurs when a speech-enabled llm can only perform tasks that are seen during supervised training and shows limited performance on unseen tasks. this section provides a detailed analysis of this issue which leads to the proposed wav2prompt method. to connect a decoder-only llm with speech input, speech representations can be prepended to the original text token embedding sequence and the llm will be conditioned on these speech representations when predicting the next token in order to perform speech tasks. to be more specific, in the normal case with a text-based user-input prompt, the next token probabilities of an llm can be formulated as: it is counter-intuitive to let a fixed text-based llm attend to speech representation s as it has never seen speech input during pre-training. however, after e2e training on supervised data, prior work has shown connecting a fixed llm with the acoustic encoder (referred to as encoder-llm) can perform asr tasks @xcite . since the main building block of llms is the attention mechanism which is also the first module that interacts with inputs, the process is simplified to a single attention function att(q, k, v ) for theoretical analysis where the conclusions can be generalised to the entire llm. in the normal case with text-based input, if the token embedding of y n-1 after the llm embedding layer is denoted z n-1 and @xmath0, • • • , z n-1 ), eq. 1 can be expressed as: speech e 1 e 7 e 8 e 11 e t ... sum α 1 α 7,1 ... α 7,2 α 8 α 11 ... ... ... s 1 s 2 s m cif module [s 1 , s 2 , ..., s m ] [p 1 , p 2 , ..., p m ] mse loss llm embedding layer text-based large language model prefix text embedding [[sos], y 1 , y 2 , ..., y n ] [z 0 , z 1 , ..., z n ] ... cif output + + + [y 1 , y 2 , ..., y n , [eos]] postfix text embedding acoustic encoder transcript token embeddings use of s and p are consistent, correct predictions can be obtained. however, z n-1 and γ differ in different tasks, and s learned on a certain task cannot always guarantee that l @xmath1, leading to so-called task over-fitting. for example, preliminary experiments found that the encoder-llm-based asr system has trouble following new instructions to perform zero-shot st. prior work relies on task-specific e2e training to implicitly optimise s @xcite @xcite , and through complex training pipelines gradually enables s to learn a correct alignment that can preserve the llm zero-shot capability @xcite . however, to learn a correct s, there is already a feasible and clear target, which is the llm token embedding given that llm can always flexibly handle text inputs for zero-shot tasks. unlike prior work, this paper proposes using llm embeddings as the target to explicitly guide the learning of the speech representation s, which greatly simplifies the process and only requires asr data for training. ## wav2prompt this paper proposes wav2prompt, a model that naturally enables text-based llm to handle speech input while maintaining the zero-shot capabilities of the original llm. wav2prompt provides a straightforward process that can be combined with llm using only asr data for training and does not require subsequent multi-stage tuning, it can serve as an e2e alternative approach that is superior to the conventional asr-llm cascade. ## experimental setup after training on asr data, wav2prompt was evaluated on a range of unseen tasks, including speech translation (st), spoken language understanding (slu), and spoken-query-based question answering (sqqa) tasks. ## experimental results in zero-shot cases, wav2prompt aims to achieve results close to the asr-llm cascade while greatly surpassing the encoder-llm. with limited taskspecific data and e2e fine-tuning, it aims to greatly surpass asr-llm and match the encoder-llm. ## conclusions this paper describes wav2prompt, which proposes a method to connect spoken input with text-based llms using only asr training data while retaining the zero-shot capability for other spoken language tasks. wav2prompt extracts label-level speech representations using the cif mechanism and explicitly enforces the consistency between the speech representations and llm embeddings using the mse loss function, thus avoiding the issue of task over-fitting. experiments on a range of tasks, including st, slu, and sqqa, showed that wav2prompt can achieve results close to the asr-llm cascade system in zero-shot scenarios and greatly outperforms the existing speech-enabled llm method. it gives results that exceed those of the asr-llm cascade in cases with limited taskspecific data. wav2prompt is an e2e alternative to conventional asr when combined with text llms.