BLSP -Emo: Towards Empathetic Large Speech-Language Models (2024)

Abstract: The recent release of GPT-4o showcased the potential of end-to-end multimodal models, not just in terms of low latency but also in their ability to understand and generate expressive speech with rich emotions. While the details are unknown to the open research community, it likely involves significant amounts of curated data and compute, neither of which is readily accessible. In this paper, we present BLSP-Emo (Bootstrapped Language-Speech Pretraining with Emotion support), a novel approach to developing an end-to-end speech-language model capable of understanding both semantics and emotions in speech and generate empathetic responses. BLSP-Emo utilizes existing speech recognition (ASR) and speech emotion recognition (SER) datasets through a two-stage process. The first stage focuses on semantic alignment, following recent work on pretraining speech-language models using ASR data. The second stage performs emotion alignment with the pretrained speech-language model on an emotion-aware continuation task constructed from SER data. Our experiments demonstrate that the BLSP-Emo model excels in comprehending speech and delivering empathetic responses, both in instruction-following tasks and conversations.

## introduction large language models (llms) have demonstrated remarkable capabilities in intent understanding @xcite , instruction following @xcite , and problem-solving @xcite , revolutionizing humanmachine interaction. speech, as the primary mode of human communication, conveys rich paralinguistic features related to emotions, tones, and in-figure 1 : illustrative example of an empathetic large language model responding to speeches with identical linguistic content but different emotional tones. recent work on end-to-end modeling of speech inputs with llms falls into two categories. the first category focuses on adapting llms for a wide range of speech and audio-related tasks, such as speech recognition, translation, and emotion recognition @xcite . however, these models lack the ability to retain the general instruction-following capabilities of llms and cannot engage in conversations with speech inputs. the second category aims to extend llms' instruction-following capability to speech inputs, enabling direct speech interaction with llms @xcite . nevertheless, these approaches primarily focus on the semantics in speech and fail to capture paralinguistic cues related to emotions. some studies have attempted to train models to understand emotions in speech and respond empathetically @xcite . however, these efforts rely on speech instruction data constructed with expressive text-to-speech synthesis tools, which limits their generalization capability with natural human speech. annotating large quantities of new emotion-sensitive instruction or conversation data for natural speech would be costly. in this paper, we present the blsp-emo approach, which aims to develop an end-to-end speech-language model capable of understanding semantics and emotions in speech and generating empathetic responses, using only existing speech recognition (asr) and speech emotion recognition (ser) datasets. blsp-emo builds upon recent work on speech-language models developed with the blsp method @xcite , which are bootstrapped from and aligned at the semantic level with an llm using asr data. these speech-language models exhibit generation behaviors consistent with the llm when presented with speech input containing the same linguistic content. we propose to perform emotion alignment to understand emotions, in addition to semantics, in speech and generate empathetic responses. specifically, we first prompt an llm to generate emotionaware continuations of transcripts in the ser data given the reference emotion label. we then adapt a speech-language model bootstrapped from the same llm to generate these continuations directly from speech. this adaptation step encourages the model to comprehend and react to both the linguistic content and paralinguistic emotion cues in speech, generating text continuations that are aligned with those the llm would produce if provided with the same linguistic content and emotion label. the contributions of our work are as follows: ## method our proposed approach, termed blsp-emo, aims to develop an end-to-end speech-language model that understands both linguistic content and paralinguistic emotion cues in speech and generates empathetic responses. blsp-emo builds upon bootstrapped speech-language models developed with the blsp method @xcite , which are adapted from a text-only llm using asr data. blsp-emo leverage ser data to enable these bootstrapped speech-language models to also comprehend and react to the paralinguistic emotion cues. in what follows, we will describe the model architecture and introduce how we achieve semantic alignment and emotion alignment. ## experiments although blsp-emo is trained only on continuation tasks, we have found that the resulting model has the ability to comprehend both linguistic content and paralinguistic emotion cues in speech and respond accordingly. this enables the model to not only follow task instructions but also demonstrate empathy toward the emotional tone conveyed in the speech. next, we will present results on speech emotion recognition, instruction-following with empathetic responses, multi-turn conversation, and generalization to other languages. ## conclusion in summary, this paper presents blsp-emo, a novel approach to build empathetic large speechlanguage models by utilizing existing speech recognition and speech emotion recognition datasets, through a two stage alignment process: semantic alignment and emotion alignment. through quantitative evaluations, we demonstrate that the blsp-emo approach extends instruction-following llms with competitive abilities to understand both semantics and emotions in speech and perform standalone speech emotion recognition, generate empathetic responses, and engage in multi-turn conversations.