Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations (2024)

Abstract: In spoken dialogue, even if two current turns are the same sentence, their responses might still differ when they are spoken in different styles. The spoken styles, containing paralinguistic and prosodic information, mark the most significant difference between text and speech modality. When using text-only LLMs to model spoken dialogue, text-only LLMs cannot give different responses based on the speaking style of the current turn. In this paper, we focus on enabling LLMs to listen to the speaking styles and respond properly. Our goal is to teach the LLM that “even if the sentences are identical if they are spoken in different styles, their corresponding responses might be different”. Since there is no suitable dataset for achieving this goal, we collect a speech-to-speech dataset, StyleTalk, with the following desired characteristics: when two current speeches have the same content but are spoken in different styles, their responses will be different. To teach LLMs to understand and respond properly to the speaking styles, we propose the Spoken-LLM framework that can model the linguistic content and the speaking styles. We train Spoken-LLM using the StyleTalk dataset and devise a two-stage training pipeline to help the Spoken-LLM better learn the speaking styles. Based on extensive experiments, we show that Spoken-LLM outperforms text-only baselines and prior speech LLMs methods.

## introduction large language models (llms) have demonstrated remarkable capabilities in dialogue generation, natural language understanding, and commonsense reasoning @xcite . while llms mostly focus on text modality, speech 1 demo of the styletalk dataset and output of spoken-llm are at https://sites.google.com/view/ spoken-llm/home . code and dataset are available at https: //github.com/daniellin94144/styletalk . represents the most natural form of human communication in our daily lives. in this work, we aim to inject speech modality for modeling spoken conversation with multi-modal llms (mm-llms). the main goal is to develop a humanizing agent capable of listening, understanding, and engaging in dialogue with humans, ultimately leading to higher user satisfaction. speech signals contain linguistic aspects (words, phonetics, syntax, and semantics), paralinguistic elements (emotions and speaker characteristics), and prosodic factors (speaking style, emphasis, and attitude). in human conversation, while the dialogue primarily relies on the lexical aspect, the speaking styles convey rich information beyond text, and can even alter the semantics of the spoken sentences @xcite . neglecting spoken styles can lead to misinterpretation of communica-tion or unnatural human interaction. for example, as shown in figure 1 , the current speech with the same current text (looks like it might rain later this week though.) but different speaking styles. the friendly speaking style leads to a cheerful response while speaking in a slow and neutral tone leans toward a sad and negative response. although there are recent studies on mm-llms for speech/audio and text, most of the existing studies focus on content-centric spoken language modeling (slm) @xcite , joint text and speech processing tasks @xcite @xcite @xcite or general audio perception and hearing ability @xcite @xcite . there is less attention on spoken dialogue with advanced methods and suitable datasets for modeling paralinguistics and speaking styles of spoken responses. to model spoken dialogue with a generative language model, dgslm @xcite proposes a dual-tower slm on discrete speech units to model two-channel spoken dialogue, but the generated spoken sentences lack semantic meaning. paralingpt @xcite organizes tasks in the sequence of current paralinguistic attribute prediction, response paralinguistic attribute prediction, and response text generation with autoregressive conditioning. however, it only uses the speech sentiment as speaking style, which might be primarily based on textual information, and how the speaking styles affect the spoken response is unclear. a concurrent work e-chat @xcite enhances llm to generate responses in different emotional contexts, but the training and evaluation data are entirely generated by gpt-3.5 without human supervision, equivalent to distillation and prompting of gpt-3.5. it can only generate response text, constraining its capacity to control response style or speech-to-speech modeling. to overcome the current limitation, we collect a novel speech-to-speech conversational dataset named styletalk. this dataset is the first spoken conversation benchmark with the same dialogue context and input sentence in different speaking styles, accompanied by corresponding expressive spoken responses for speech-to-speech modeling. the dataset will be released upon the paper's acceptance. the proposed model can predict response speaking style and text, enabling the subsequent expressive text-to-speech (tts) model to generate natural and diverse speech responses. we validate the performance through objective and subjective evaluations of spoken responses. with the same backbone model, the proposed method outperforms the text and speech llm baseline in lexical/semantic similarity and response style f1 score. the human evaluation also indicates that the proposed method yields more reasonable and proper response speech than the text-only llm baseline approach. ## experiments 4.1 baseline method all baseline methods are fine-tuned on the same amount of training data and warmup pre-training, with the identical llm backbone and speech style encoder for paralingpt. text-llm (text-only): the initial simple baseline is built by simply fine-tuning text-to-text llm on styletalk. this serves as a performance reference to evaluate the model's capability without knowing any explicit speaking style information. since the model cannot predict the response style, a randomly selected response style is assigned for this method to synthesize expressive speech. text-llm (cascaded): one can represent the style information in text to enable the model to better predict the response style and text. this approach, referred to as the cascaded pipeline method, involves cascading a style recognition model 3 with the text llm. the text-llm (upper bound) method is the cascaded text-llm with ground truth style labels. paralingpt @xcite : the serialized multitasking approach proposed by paralingpt is a sequential conditioning mechanism, unifying current style prediction, response style prediction, and response text generation within an autoregressive chain. the main difference between paralingpt and spoken-llm is that spoken-llm performs two-stage training (style alignment for current speech then focus on the response speech), but paralingpt directly models them in an autoregressive chain, which might be prone to error propagation if the incorrect current style prediction or focusing too much on the current style. ## related works speech-text multimodal llm: the progress in speech self-supervised learning (ssl) @xcite @xcite and neural audio codec @xcite @xcite @xcite enable extracting discrete speech units, drawing attention to generative spoken language modeling. specifically, the discrete speech units are treated as a special language for unit language modeling @xcite @xcite @xcite , further enabling multiple speech processing tasks in single multimodal llm @xcite @xcite @xcite @xcite dialogue context a : i've finally hit my goal of running five miles every day! b : that's amazing, hard work really does pay off, doesn't it? a : absolutely, and i've never felt more alive! current text b: i really need to get back into exercising too. not knowing current style text-llm (text-only): totally understand, let's start small together, maybe a walk or something? current style @xmath0, normal, @xmath1: <friendly, normal, @xmath2; i could use the company. spoken-llm: <cheerful, normal, @xmath3: <friendly, normal, @xmath4, we could be workout buddies if you want, i can help you get started! current style @xmath5, fast, @xmath6: <neutral, normal, @xmath7, no pressure though, take things at your own pace, you know? spoken-llm: <unfriendly, normal, @xmath8, no need to stress out about it, we can start with something small if you want. ground truth: <unfriendly, normal, @xmath9, no need to get upset, maybe we can find a way to ease you into a routine? the other line of work aims for the universal speech and audio understanding model to have general audio perception and hearing ability, either leveraging off-the-shelf expert models @xcite , or with a single mm-llm @xcite @xcite . those methods are mainly trained to perform comprehensive speech and audio understanding tasks and then fine-tuned for audio-based instructionfollowing data generated by off-the-shelf llm like gpt-3.5. however, they are limited to only generating text responses without considering responding styles, and the data quality from llm is unknown. in contrast, we focus on modeling speaking style in speech-to-speech conversation with a manual-filtered dataset. speaking style in spoken dialogue: speaking style is important for speech understanding and response generation in spoken dialogue. the understanding of speaking styles in spoken dialogue is crucial for extracting style attributes such as emotion, sentiment, sarcasm, and more. representative corpora for studying speaking styles in spoken conversations include iemocap @xcite , @xcite , mustard @xcite , switchboardsentiment @xcite , meld @xcite , meisd @xcite , and mspimprov @xcite . these datasets are primarily constructed based on label annotations from real speech conversations (e.g., tv series) or acted spoken conversations. another research direction involves spoken conversation in the form of spoken question answering. datasets in this category include nmsqa @xcite , scqa @xcite , open-saqa @xcite , and e-chat200 @xcite , where the data sample is presented as a tuple of (question, answer). specifically, for stylerelated questions and answers, opensaqa employs gpt-3.5 to generate textual questions based on the speech content and metadata style information, while e-chat considers text with emotion labels as the question for gpt-3.5 to generate the responding answer as gold answers. in all existing datasets, only one style is attached to the speech, and one corresponding response speech exists for each conversational context. thus, prohibiting researchers from investigating the impact of different styles given the same context and the same words. additionally, the sqa data in opensaqa and e-chat are fully generated by gpt-3.5 and not carefully checked by humans, resembling distillation and prompting of gpt-3.5, which is concerning whether the sample follows a human standard as spoken conversation. our work provides the spoken dialogue data with the same context, the same current text with different speaking styles, and the corresponding response speech with human annotator filtering. ## conclusion this paper focuses on enhancing llm by modeling how the same sentence spoken with different speaking styles causes different responses in speech, in the spoken conversation scenario. due to the absence of a suitable dataset, we first collect the speech-to-speech styletalk dataset that contains the same dialogue context the same sentence spoken in different styles, and the corresponding different response speech. next, we propose spoken-llm, a two-stage multi-modal training framework to capture different speaking styles and respond properly. the proposed method yields better performance than the text and speech baseline on objective metrics and performs better than text-only llm on subjective evaluation. we encourage the research community to use the released styletalk for joint speaking style and language modeling.