W av LLM : Towards Robust and Adaptive Speech Large Language Model (2024)

Abstract: Recent advancements in large language models (LLMs) have expanded their scope in natural language processing (NLP) to encompass multimodal functions. However, integrating listening capabilities effectively remains a significant challenge for generalization and complex auditory task execution. In this work, we introduce WavLLM, a robust and adaptive speech large language model featuring dual encoders—a Whisper encoder for semantics and a WavLM encoder for speaker characteristics. Within the two-stage curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks such as combinations of the elementary tasks. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks and also apply it to specialized speech-question-answer (SQA) dataset, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. The codes, models, audio samples, and SQA evaluation set can be accessed at https://github.com/microsoft/SpeechT5/tree/main/WavLLM .

## introduction large language models (llms) have witnessed a meteoric rise in advancement within the last couple of years, reaching or even exceeding the proficiency of humans in a myriad of natural language processing (nlp) tasks @xcite @xcite . with large language models attaining substantial breakthroughs, the focus is increasingly shifting towards the capabilities and advancements of multi-modal large language models (mllms), which possess the ability to listen @xcite , speak @xcite , see @xcite , and create content @xcite . amidst the broadening scope of abilities, speech stands out as a crucial form of human communication, prompting extensive research to equip large language models (llms) with speech perception capabilities @xcite @xcite @xcite . typically, llms are augmented with an auxiliary audio encoder designed to preprocess audio signals, transforming them into the same input space as that of the llms, enabling them to achieve various speech tasks, such as automatic speech recognition (asr), speech question answering (sqa), and so on. however, previous research has yet to overcome significant challenges in achieving effective generalization due to two main issues: (1) specialized tasks are highly sensitive to prompt design, resulting in performance degradation when confronted with unseen or complex instructions; (2) there is an absence of speech chain-of-thought (cot) @xcite capability, which is essential for addressing complex tasks. in this work, we propose a robust and adaptive speech large language model, wavllm, aiming at enhancing the generalization capabilies, following speech instruction effectively, and processing the given speech in accordance with provided textual prompts, as well as supporting multi-round dialog. specifically, to distinguish various types of speech information, we utilize a whisper @xcite encoder to encode the semantic content of the speech, and a wavlm @xcite encoder to capture the acoustic information, like unique characteristics of the speaker's identity. during the model training phase, we develop a curriculum learning method that progressively finetune llms to follow instructions for understanding and processing speech, starting from simple tasks and advancing towards more complex ones. in the initial mixed single-task training stage, we leverage a substantial dataset of synthesized spoken question-answering content generated by gpt-4 and tailored to various speech-centric tasks such as automatic speech recognition (asr), speechto-text translation (st), emotion recognition (er), speaker verification (sv), and so on, to fine-tune the wavllm with low rank adaptation (lora) techniques @xcite . to enhance the generalization on the unseen or complex instructions 1 , we introduce an advanced multi-task training stage, incorporating a specially constructed prompt-aware multi-task speech processing dataset combining the elementary tasks. furthermore, we design a novel prompt-aware lora weight adapter for this stage, capable of adaptively tuning the lora weights according to varying prompts, thereby improving the model's generalization capacity and robustness. we evaluate the proposed model on 1) single tasks, including a) universal speech benchmark, including asr, sv, er and st; b) spoken-querybased question answering and english listening comprehension test in chinese national college entrance examination, which presents a spoken dialogue, and requires to answer text-based choice questions related to the conversation, and 2) multiple tasks, consisting of c) instruction-independent multi-tasks dataset that combines multiple independent prompts in a single instruction; d) speech cot evaluation set that decomposes a complex task into multiple sub-tasks. extensive evaluations demonstrate that our proposed model exhibits robust generalization and cot capabilities, consistently surpassing strong baselines across a broad spectrum of speech-related tasks. in summary, the contributions of this paper can be categorized as follows: 1) equipped with the proposed prompt-aware lora weight adapter, a curriculum learning method is leveraged for model training, by incrementally fine-tuning large language models with robustness and generalization capabilities, beginning with simple tasks and progressing to complex 1 please find detailed motivations in section 2. ones. 2) our proposed model employs a decoupling strategy for speech information, utilizing the whisper encoder to capture semantic content and the wavlm encoder for acoustic features, thereby enriching speech representation and improving performance on downstream tasks. 3) wavllm demonstrates exceptional generalization capabilities when responding to diverse prompts and completing complex tasks. it exhibits impressive capabilities in zero-shot sqa such as english listening comprehension, and shows strong proficiency in cot-based tasks, delivering performance gains over non-cot tasks. ## related work the exploration of multi-modal large language models involves the integration of diverse data types including text, images, video, speech, audio, and more. this represents a natural progression from text-based large language models, designed to enable the perception of the world and the creation of content @xcite @xcite . for instance, kosmos-1 @xcite and gpt-4v (openai, 2023) are able to perceive general modalities beyond text, and follow instruction provided by users to process and analyze image inputs. another research direction focuses on improving the multi-modal generative abilities of language models, enabling them to produce visual content like images or videos, as exemplified by minigpt-5 @xcite and sora @xcite . related research to this work focuses on speech-enhanced large language models that aim to endow llms with the capability to perceive and process speech signal @xcite @xcite @xcite . among these studies, speechgpt @xcite empowers large language models with crossmodal conversational abilities by three-stage training stages, using hidden units as the discrete representation of speech. llasm @xcite builds a large chinese/english speech language model that can understand and follow instructions, through pre-training and cross-modal instruction fine-tuning stages. blsp @xcite bootstraps language-speech pre-training via behavior alignment of continuation writing. salmonn @xcite , named from a speech audio language music open neural network, boosts large language models with generic hearing abilities with a activation tuning stage by playing with the lora scaling factor. qwen-audio @xcite scales up audio-language pre-training to cover over 30 tasks and various audio types, including human speech, natural sounds, music, and songs. ## method the wavllm is optimized by maximizing the following probability: ## experiments please find implementation details in appendix g. ## conclusion in this paper, we propose wavllm, a robust and adaptive speech large language model, which uses llama-2-chat as foundational llm backbone, and extracts semantic and acoustic information from speech audio utilizing whisper and wavlm encoders. utilizing a curriculum learning approach, the proposed wavllm commences with singletask instruction training in the initial mixed training phase and subsequently expanding our training to encompass additional multi-task prompts in the second advanced phase with the integration of the proposed prompt adapter. massive experiments demonstrate that our wavllm model delivers stateof-the-art performance on various speech-related tasks and robust generalization abilities on singletask and multi-task instructions, enhanced by a cot processing ability that greatly improves its effectiveness in tackling intricate tasks.