Behavior- SD : Behaviorally Aware Spoken Dialogue Generation with Large Language Models (2025)

Abstract: Spoken dialogue involves behaviors like turn-taking, interruptions, filler words, and backchannels, which make interactions more natural and engaging but are often overlooked in language models. These models struggle to explicitly model these behavioral traits, resulting in a less natural and personalized communication style that aligns with user needs. To address this challenge, we make two key contributions. First, we introduce Behavior-SD, a large-scale dataset containing over 100K spoken dialogues (2,164 hours) annotated with various conversational behaviors, synthesized via LLMs to model diverse full-duplex interactions. Second, we propose BeDLM, the first dialogue model capable of generating natural conversations conditioned on specific behavioral and narrative contexts, supporting simultaneous contributions from both speakers. Through human evaluations and behavior-adherence metrics, we demonstrate that BeDLM outperforms baseline models in generating natural, coherent, and behaviorally rich dialogues. Our work opens new possibilities for developing behaviorally-aware dialogue systems that more closely mimic human conversational dynamics, enhancing user engagement and communication effectiveness.

## introduction large language models (llms) have significantly improved their ability to generate coherent text; however, producing human-like spoken dialogues remains a considerable challenge. spoken conversations are inherently dynamic, involving elements like prosody, emotion @xcite , turn-taking, and interruptions @xcite . capturing these nuances is crucial for generating lifelike dialogue, yet current tts systems @xcite and spoken language mod-* equal contribution. els @xcite struggle to replicate this complexity fully. a particularly challenging aspect of real-world conversations is managing full-duplex communication, where speakers frequently overlap or interrupt each other. full-duplex conversations, unlike traditional systems that enforce strict turn-taking, enable more fluid interactions, which can be beneficial in applications such as customer service, education, or voice assistant applications @xcite @xcite . additionally, conversational dynamics are influenced by personal traits, social contexts, and factors like politeness @xcite , making this task even more complex. despite the prevalence of these dynamics in human conversations, they are often underexplored in dialogue generation. while existing spoken dialogue datasets @xcite @xcite include conversational behaviors such as interruptions and backchannels, these behaviors are not labeled at the speaker level, limiting the ability to model nuanced interaction patterns. recent datasets @xcite focus on sequential turn-taking, overlooking critical aspects of natural conversations like overlapping speech and interruptions. to address these gaps, we present behavior-sd, a large-scale dataset of over 100k spoken dialogues (sd), totaling 2,164 hours, and covering a wide range of conversational behaviors and social situations. unlike existing datasets constrained by narrow contexts @xcite @xcite , behavior-sd explicitly models key conversational traits such as verbosity, backchannels, and interruptions, allowing for more realistic modeling of human-like dialogues. this dataset serves as a foundation for improving both the scale and behavioral diversity of dialogue generation. specifically, it captures four key speaker behavior traits, as outlined in the length of utterances during a speaker's turn. (f) filler words the occurrence of filler words (e.g, "like," "um", "you know") during a speaker's turn. (b) backchannels the frequency of backchannel responses (e.g, "yeah," "uh-huh") during the other speaker's turn. (i) interruptions the frequency of interrupting the other speaker's turn to begin speaking. moreover, we introduce behavior-conditioned spoken dialogue language model (bedlm), the first model capable of generating full-duplex spoken dialogues by incorporating behavior traits. by integrating these traits through control tokens, bedlm produces more dynamic and lifelike conversations, adapting naturally to speaker personalities and narrative context. this novel approach enables customization of dialogue behaviors in a way that previous models, which typically focus on static turn-taking, cannot achieve. in our experiments, bedlm demonstrates improvements in dialogue naturalness and adherence to conditions compared to state-of-the-art models. the dataset, model, and code will be made publicly available to encourage further research in this area. ## related works automatic spoken dialogue generation. the automatic synthesis of dialogues using llms has made significant strides, but generating spoken dialogue remains challenging due to features like paralinguistic cues, interruptions, and unclear turn boundaries in speech @xcite @xcite . approaches such as dgslm @xcite have tackled overlapping speech generation using dual-tower transformers without text inputs, while chats @xcite builds on this by generating natural spoken dialogues from text transcriptions. however, these methods lack mechanisms to condition overall conversational behavior throughout a dialogue. ## behavior-driven data generation our framework for dataset generation operates in four stages: (1) generating dialogues with behaviorspecific traits, (2) inserting backchannels at contextually appropriate moments, (3) captioning the speech style of each utterance for tts integration, timing-driven utterance placement and ( 4 ) converting text dialogues into spoken form. figure 1 overviews the framework. all detailed prompts for llms can be found in appendix a. ## the behavior-sd dataset using the pipeline in the previous section, we contribute behavior-sd (behavior-driven spoken dialogues) as a large-scale, full-duplex dataset that captures a diverse range of social interactions and conversational behaviors. a sample from our dataset is shown in figure 3 . as shown in figure 2 presents the detailed statistics of our dataset, showing the distribution of key conversa-tional features across different behavior levels. it illustrates how varying behavior categories impact the length and structure of conversations. for example, the duration of utterances differs notably between behavior levels, indicating different conversational patterns. additionally, features such as filler words, backchannels, and interruptions show distinct distribution trends across behavior levels. ## experiments in this section, we provide the details of model training and baselines for spoken dialogue generation. ## conclusion we present a scalable framework for generating spoken dialogues with a diverse combination of conversational behaviors using llms and tts models. our contribution includes the behavior-sd dataset, comprising over 100k spoken dialogues with behavior annotations, and the bedlm, generating spoken dialogues that outperform baseline models in naturalness, diversity, and behavioral coherence. future work could explore expanding multi-speaker dialogues, improving behavior control granularity, and integrating sophisticated realtime interaction capabilities to enhance practical applications in various spoken dialogue systems.